Monitoring Confident AI-Evaluated LLM Applications with Vigilmon: External Uptime, API Health & Heartbeat Monitoring

Confident AI is an LLM evaluation and testing platform that helps teams measure the quality of their AI applications through automated test suites, regression checks, and production evaluation pipelines. It tracks metrics like answer correctness, hallucination rate, faithfulness, and contextual relevance — giving you an inside view of how well your LLM behaves on your specific use cases. What Confident AI doesn't provide is an independent external health check: a synthetic probe that measures whether your AI application's public endpoints are reachable and responding correctly from the outside. Vigilmon fills that gap. It sits entirely outside your infrastructure and probes your services from multiple external locations, alerting you independently of whether Confident AI's evaluation pipeline is healthy.

This tutorial covers adding external uptime monitoring to LLM applications already evaluated with Confident AI.

What You'll Build

• A /health endpoint that reflects the real state of your AI application's dependencies
• A Vigilmon HTTP monitor with response-body assertions
• A heartbeat monitor for asynchronous LLM evaluation and regression jobs
• An alert routing strategy that keeps Confident AI evaluation alerts and Vigilmon uptime alerts separate but correlated

Prerequisites

• A Confident AI account with your LLM application sending evaluation results via the DeepEval SDK
• At least one AI application or inference API endpoint reachable via a public HTTPS URL
• A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your AI Application

Confident AI evaluates your LLM outputs from inside your test suite and CI pipeline. Vigilmon needs an HTTP endpoint it can reach from the outside. Make the health check verify real dependencies — your LLM provider connectivity, vector store availability, and any backing database — not just a static 200 that would pass even when core dependencies are degraded.

Python (FastAPI with OpenAI)

# app/health.py
import os
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

@app.get("/health")
async def health():
    checks = {}
    ok = True

    # LLM provider reachability check
    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            resp = await client.get(
                "https://api.openai.com/v1/models",
                headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
            )
            checks["llm_provider"] = "ok" if resp.status_code == 200 else f"http_{resp.status_code}"
            if resp.status_code != 200:
                ok = False
    except Exception as exc:
        checks["llm_provider"] = f"error: {exc}"
        ok = False

    # Knowledge base / retrieval store check
    try:
        import chromadb
        client = chromadb.HttpClient(host=os.environ["CHROMA_HOST"])
        client.heartbeat()
        checks["retrieval_store"] = "ok"
    except Exception as exc:
        checks["retrieval_store"] = f"error: {exc}"
        ok = False

    status_code = 200 if ok else 503
    return JSONResponse(
        status_code=status_code,
        content={"status": "ok" if ok else "degraded", "checks": checks},
    )

Node.js (Express with OpenAI)

// src/health.js
const express = require('express');
const router = express.Router();

router.get('/health', async (req, res) => {
  const checks = {};
  let ok = true;

  // LLM provider connectivity check
  try {
    const response = await fetch('https://api.openai.com/v1/models', {
      headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
      signal: AbortSignal.timeout(5000),
    });
    checks.llm_provider = response.ok ? 'ok' : `http_${response.status}`;
    if (!response.ok) ok = false;
  } catch (err) {
    checks.llm_provider = `error: ${err.message}`;
    ok = false;
  }

  // Redis cache check (used for response caching)
  try {
    await redisClient.ping();
    checks.cache = 'ok';
  } catch (err) {
    checks.cache = `error: ${err.message}`;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    checks,
  });
});

module.exports = router;

Verify the endpoint before configuring Vigilmon:

curl -s https://your-ai-app.example.com/health | jq .
# {"status":"ok","checks":{"llm_provider":"ok","retrieval_store":"ok"}}

Step 2: Configure Vigilmon HTTP Monitor

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://your-ai-app.example.com/health
3. Check interval: 60 seconds — appropriate for production AI APIs.
4. Response timeout: 15 seconds — LLM health checks can be slower due to provider round-trips.
5. Expected status code: 200.
6. JSON body assertion:
   • Path: status
   • Expected value: ok
7. Click Save.

Vigilmon immediately begins probing from multiple external probe locations. A non-200 response or a failed JSON assertion triggers an alert independently of Confident AI's evaluation pipeline.

Multiple endpoints and environments

AI applications evaluated with Confident AI typically span multiple services. Mirror each in Vigilmon:

| Endpoint | Vigilmon monitor URL | Interval | |---|---|---| | Inference API | https://api.example.com/health | 60 s | | RAG retrieval service | https://rag.example.com/health | 60 s | | Embeddings endpoint | https://embeddings.example.com/health | 60 s | | Staging environment | https://api-staging.example.com/health | 120 s |

Group monitors under a single Monitor group in Vigilmon to keep your dashboard organized by environment.

Step 3: Heartbeat Monitor for LLM Evaluation and Regression Jobs

Confident AI evaluation suites often run on a schedule — nightly regression checks against golden datasets, weekly hallucination audits, or CI-triggered evaluation pipelines. These jobs run outside your request/response lifecycle. Use Vigilmon heartbeats to guarantee these evaluation jobs complete on schedule.

# jobs/nightly_eval.py
import os
import requests
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

def run_nightly_evaluation():
    """Run nightly regression evaluation suite against production model."""
    test_cases = load_golden_dataset()

    metrics = [
        AnswerRelevancyMetric(threshold=0.7),
        FaithfulnessMetric(threshold=0.8),
    ]

    evaluate(test_cases=test_cases, metrics=metrics)

if __name__ == "__main__":
    try:
        run_nightly_evaluation()
        requests.post(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)
        print("Nightly evaluation complete — heartbeat sent")
    except Exception as exc:
        # Deliberate silence — missed heartbeat fires Vigilmon alert
        print(f"Nightly evaluation failed: {exc}")
        raise

In Vigilmon, create a Heartbeat monitor and set the grace period slightly above your evaluation cadence. For a nightly job, use 25 hours. Store the heartbeat URL in your secrets manager and inject it as VIGILMON_HEARTBEAT_URL.

GitHub Actions integration

# .github/workflows/nightly-eval.yml
name: Nightly LLM Evaluation

on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install deepeval

      - name: Run evaluation suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          CONFIDENT_AI_API_KEY: ${{ secrets.CONFIDENT_AI_API_KEY }}
        run: python jobs/nightly_eval.py

      - name: Send heartbeat
        if: success()
        run: curl -X POST "${{ secrets.VIGILMON_HEARTBEAT_URL }}"

Step 4: Alert Routing Strategy

Confident AI alerts on LLM quality signals: evaluation metric regressions, rising hallucination rates, answer relevancy drops, faithfulness violations. Vigilmon alerts on infrastructure availability: the endpoint is down, the health check is failing, the evaluation job missed its window. These are distinct failure modes that require different responses.

| Failure mode | Source | Route to | |---|---|---| | Answer relevancy regression | Confident AI | Slack #ai-quality + ML team | | Hallucination rate spike | Confident AI | On-call ML engineer | | External API endpoint down | Vigilmon | PagerDuty on-call + Slack #prod-alerts | | Evaluation job missed schedule | Vigilmon | Slack #ai-ops-critical | | TLS certificate expiring | Vigilmon | DevOps team |

Configure Vigilmon alert channels under Alerts → Add channel:

• Email: your on-call distribution list.
• PagerDuty webhook: wire to your primary on-call rotation for endpoint outages.
• Slack webhook: a dedicated #vigilmon-alerts channel separate from Confident AI dashboards.

Step 5: Correlating Confident AI Evaluations and Vigilmon Alerts During Incidents

When Vigilmon fires a downtime alert for an AI application endpoint, use this checklist to determine root cause:

1. Check Vigilmon probe regions — did all probe locations fail or just one? All failing points to your application or LLM provider. One region failing suggests a network routing issue.
2. Open Confident AI → check recent evaluation run results for the same time window. A sudden increase in provider timeout errors or API failures during evaluations often precedes an availability failure.
3. Check token quota and rate limits — LLM provider rate-limit exhaustion can cause your health endpoint to return 503 even when your application code is healthy.
4. Check evaluation run timestamps — did Confident AI start reporting provider errors before Vigilmon fired? If so, the LLM provider issue likely caused the downtime.
5. Review model version history — a recent deployment that broke a Confident AI evaluation suite may also have broken the live serving endpoint.

Step 6: Public Status Page

Confident AI evaluation results are private quality metrics visible to your engineering team. Give customers and API consumers a public-facing status page using Vigilmon.

1. In Vigilmon, go to Status Pages → Create.
2. Add your production inference API monitor and any critical heartbeat monitors.
3. Configure a custom domain (e.g., status.example.com) or use the Vigilmon subdomain.
4. Embed the status badge in your developer documentation:

<img src="https://vigilmon.online/badge/<monitor-id>.svg" alt="API Status" />

Your API consumers get real-time uptime visibility without requiring access to your Confident AI workspace or internal evaluation infrastructure.

What Vigilmon Adds to a Confident AI Stack

| Capability | Confident AI | Vigilmon | |---|---|---| | LLM evaluation metrics (relevancy, faithfulness) | Yes | No | | Hallucination and regression detection | Yes | No | | A/B model comparison testing | Yes | No | | Golden dataset management | Yes | No | | External synthetic HTTP probing | No | Yes | | DNS resolution failure detection | No | Yes | | TLS/SSL certificate expiry alerts | No | Yes | | Scheduled evaluation job heartbeats | No | Yes | | Public status page | No | Yes | | Independent of LLM provider health | No | Yes |

Confident AI gives you deep visibility into whether your LLM application is producing high-quality, faithful, and relevant outputs. Vigilmon gives you the external, synthetic view that tells you what your users actually experience when they try to reach your AI application. Together they cover every dimension of AI service reliability: from evaluation metrics to customer-facing uptime.

Add external uptime monitoring to your Confident AI-evaluated application today — register free at vigilmon.online.