How to Monitor DeepEval with Vigilmon

DeepEval brings software engineering rigor to LLM testing: 14+ research-backed metrics, pytest integration, red-teaming scans, and a dashboard for tracking quality over time. Teams use it to enforce quality gates in CI/CD, catch hallucinations before they reach production, and measure RAG pipeline fidelity.

The risk is the same one that affects any test suite: it can stop running without telling you. An expired API key, a dependency update, a CI configuration change — and your quality checks go silent while you keep shipping. This guide shows you how to make that silence impossible with Vigilmon heartbeat monitoring.

What can go wrong with a DeepEval pipeline

Eval runs silently stop. DeepEval jobs triggered by CI, cron, or scripts can fail without surfacing alerts — especially when the failure is infrastructure-level (API timeout, out of memory) rather than a test assertion.

The Confident AI dashboard goes stale. DeepEval optionally syncs results to a hosted dashboard. If your sync stops working, you're making model decisions based on data that stopped updating.

Red-teaming scans get skipped. Vulnerability scans are expensive and often run weekly or on a separate schedule. They're the first thing to silently fall off when CI flakiness triggers a skip rule.

Metric computation fails for a subset of test cases. G-Eval, faithfulness, and hallucination metrics each call your LLM judge. If the judge times out on certain inputs, those test cases fail silently rather than raising errors, making your pass rate look artificially high.

Step 1: Add heartbeat pings to your DeepEval test suite

DeepEval integrates with pytest. The cleanest place to add a heartbeat ping is in a pytest fixture that runs after the full suite:

# conftest.py
import os
import httpx
import pytest


@pytest.fixture(scope="session", autouse=True)
def ping_heartbeat_on_success(request):
    """Ping Vigilmon heartbeat after a successful eval run."""
    yield  # All tests run here

    # Only ping if no tests failed
    failed = request.session.testsfailed
    if failed == 0:
        heartbeat_url = os.environ.get("DEEPEVAL_HEARTBEAT_URL")
        if heartbeat_url:
            try:
                httpx.get(heartbeat_url, timeout=10)
                print(f"\nHeartbeat pinged (0 failures in {request.session.testscollected} tests)")
            except Exception as e:
                print(f"\nWarning: heartbeat ping failed: {e}")
    else:
        print(f"\nHeartbeat skipped — {failed} test(s) failed")

Now run your evals normally:

deepeval test run test_rag_pipeline.py

If any test fails or the process crashes, no ping is sent — the absence of a ping IS the alert.

Step 2: Set up heartbeat monitoring in Vigilmon

Create a heartbeat monitor for each DeepEval pipeline:

1. Sign up at vigilmon.online — free tier, no credit card
2. Click New Monitor → Heartbeat
3. Set the expected interval (e.g. 25 hours for nightly evals)
4. Copy the unique ping URL
5. Add it to your environment:

# .env
DEEPEVAL_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/your-unique-token

For multiple eval pipelines, create one heartbeat per pipeline:

DEEPEVAL_RAG_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/token-1
DEEPEVAL_SAFETY_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/token-2
DEEPEVAL_REDTEAM_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/token-3

Use a separate conftest or pytest mark to identify which suite you're in:

# conftest.py
import os

HEARTBEAT_ENV_VAR = os.environ.get("DEEPEVAL_SUITE", "DEEPEVAL_RAG")

@pytest.fixture(scope="session", autouse=True)
def ping_heartbeat_on_success(request):
    yield
    if request.session.testsfailed == 0:
        url = os.environ.get(f"{HEARTBEAT_ENV_VAR}_HEARTBEAT_URL")
        if url:
            httpx.get(url, timeout=10)

DEEPEVAL_SUITE=DEEPEVAL_SAFETY deepeval test run test_safety.py

Step 3: Write DeepEval tests with metric thresholds

For context, here's a complete DeepEval test file that covers the most important LLM quality checks:

# test_rag_pipeline.py
import pytest
from deepeval import assert_test
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    ContextualRecallMetric,
    HallucinationMetric,
)
from deepeval.test_case import LLMTestCase

# Your RAG pipeline function
from app.rag import query_pipeline


@pytest.mark.parametrize(
    "question,context,expected_answer",
    [
        (
            "What is our refund policy?",
            ["Refunds are available within 30 days of purchase."],
            "You can get a refund within 30 days.",
        ),
        (
            "How do I reset my password?",
            ["Click 'Forgot password' on the login page."],
            "Use the 'Forgot password' link.",
        ),
    ],
)
def test_rag_answer_quality(question, context, expected_answer):
    actual_output = query_pipeline(question, context)

    test_case = LLMTestCase(
        input=question,
        actual_output=actual_output,
        expected_output=expected_answer,
        retrieval_context=context,
    )

    assert_test(
        test_case,
        metrics=[
            AnswerRelevancyMetric(threshold=0.7),
            FaithfulnessMetric(threshold=0.8),
            ContextualRecallMetric(threshold=0.6),
        ],
    )


def test_no_hallucination():
    actual_output = query_pipeline(
        "What is the capital of France?",
        context=["France is a country in Western Europe."],
    )

    test_case = LLMTestCase(
        input="What is the capital of France?",
        actual_output=actual_output,
        context=["France is a country in Western Europe."],
    )

    assert_test(test_case, metrics=[HallucinationMetric(threshold=0.3)])

Run the suite:

deepeval test run test_rag_pipeline.py -v

Step 4: Monitor CI/CD eval gates

Add the heartbeat to your CI pipeline so Vigilmon knows the gate ran:

# .github/workflows/llm-quality.yml
name: LLM Quality Gate

on:
  push:
    branches: [main]

jobs:
  deepeval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install deepeval httpx

      - name: Run DeepEval test suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          DEEPEVAL_HEARTBEAT_URL: ${{ secrets.DEEPEVAL_HEARTBEAT_URL }}
        run: deepeval test run test_rag_pipeline.py

      # conftest.py handles the heartbeat ping on success
      # The step above fails (non-zero exit) if any eval fails,
      # which also prevents the heartbeat from being sent

If the CI job is skipped, cancelled, or its runner fails, no heartbeat arrives — Vigilmon alerts you that your quality gate has a gap.

Step 5: Monitor the Confident AI dashboard sync

If you use DeepEval's hosted dashboard (Confident AI), add a monitor to verify your data is syncing:

# scripts/verify_dashboard_sync.py
"""
Verify that recent DeepEval results have synced to the Confident AI dashboard.
Run this after every eval job as a sync health check.
"""
import os
import httpx
import sys
from datetime import datetime, timedelta, timezone


async def verify_recent_sync():
    api_key = os.environ.get("CONFIDENT_API_KEY")
    if not api_key:
        print("CONFIDENT_API_KEY not set — skipping sync verification")
        return True

    async with httpx.AsyncClient() as client:
        response = await client.get(
            "https://app.confident-ai.com/api/v1/test-runs",
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=10,
        )

    if response.status_code != 200:
        print(f"Dashboard API returned {response.status_code}")
        return False

    runs = response.json().get("data", [])
    if not runs:
        print("No test runs found in dashboard")
        return False

    latest = runs[0]
    created_at = datetime.fromisoformat(latest["created_at"].replace("Z", "+00:00"))
    age = datetime.now(timezone.utc) - created_at

    if age > timedelta(hours=26):
        print(f"Latest dashboard sync is {age} old — expected within 26 hours")
        return False

    print(f"Dashboard sync OK — last run {age} ago")
    return True


if __name__ == "__main__":
    import asyncio
    ok = asyncio.run(verify_recent_sync())
    sys.exit(0 if ok else 1)

Add a separate heartbeat for this sync check:

# Run after each eval job
python scripts/verify_dashboard_sync.py && curl -s "$DASHBOARD_SYNC_HEARTBEAT_URL"

Step 6: Set up alerts

Configure alert delivery in Vigilmon:

For Slack:

1. Create an incoming webhook in Slack
2. In Vigilmon go to Notifications → New Channel → Slack
3. Paste the URL and enable it on your heartbeat monitors

When an eval pipeline misses its window:

🔴 MISSED: deepeval-rag-pipeline heartbeat
Expected every: 25 hours
Last ping: 31 hours ago

For on-call teams:

🔴 MISSED: deepeval-safety-scan heartbeat
Expected every: 168 hours (weekly)
Last ping: 10 days ago

Create a public status page for stakeholders who care about LLM quality pipeline health:

1. Go to Status Pages → New Status Page
2. Add your DeepEval heartbeat monitors
3. Share the URL with your ML and product teams

What you've built

| What | How | |---|---| | Eval run monitoring | Heartbeat ping via pytest session fixture | | CI quality gate monitoring | GitHub Actions step + heartbeat on pipeline pass | | Per-pipeline visibility | Separate heartbeat per eval suite | | Dashboard sync monitoring | Heartbeat on Confident AI sync verification script | | Instant alerts | Slack webhook notifications | | Quality pipeline status page | Vigilmon status page for ML team visibility |

The real value of DeepEval is the data it produces over time. Gaps in that data are invisible without active monitoring — this setup makes them impossible to miss.

Get started free at vigilmon.online — your first monitor is running in under a minute.