Monitoring Opik-Instrumented LLM Applications with Vigilmon: External Uptime, API Health & Heartbeat Checks

Opik is an open-source LLM evaluation and observability platform by Comet. It gives you inside-out visibility into your LLM applications: tracing every prompt and completion, running automated evaluations with LLM-as-judge, tracking evaluation scores over time, and monitoring production traffic for quality regressions. What Opik doesn't provide is an independent external health check — a synthetic probe that measures whether your LLM application's public endpoints are reachable and responding correctly from the outside world. Vigilmon fills that gap. It sits entirely outside your infrastructure and probes your services from multiple external locations, alerting you independently of whether your Opik instance or its tracing pipeline is healthy.

This tutorial covers adding external uptime monitoring to LLM applications already instrumented with Opik.

What You'll Build

• A /health endpoint that reflects the real state of your LLM application's dependencies
• A Vigilmon HTTP monitor with response-body assertions
• A heartbeat monitor for LLM evaluation pipelines and batch processing jobs
• An alert routing strategy that keeps Opik evaluation alerts and Vigilmon uptime alerts separate but correlated

Prerequisites

• An Opik account (Comet-hosted or self-hosted) with your LLM application sending traces via the Opik SDK
• At least one LLM application endpoint reachable via a public HTTPS URL
• A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your LLM Application

Opik traces your LLM calls and evaluates their quality from inside your application. Vigilmon needs an HTTP endpoint it can reach from the outside. Make the health check verify real dependencies — your LLM provider connectivity, vector store availability, and any backing database — not just a static 200 that would pass even when core dependencies are degraded.

Python (FastAPI with OpenAI)

# app/health.py
import os
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

@app.get("/health")
async def health():
    checks = {}
    ok = True

    # LLM provider reachability
    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            resp = await client.get(
                "https://api.openai.com/v1/models",
                headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
            )
            checks["llm_provider"] = "ok" if resp.status_code == 200 else f"http_{resp.status_code}"
            if resp.status_code != 200:
                ok = False
    except Exception as exc:
        checks["llm_provider"] = f"error: {exc}"
        ok = False

    # Database probe
    try:
        import sqlalchemy
        engine = sqlalchemy.create_engine(os.environ["DATABASE_URL"])
        with engine.connect() as conn:
            conn.execute(sqlalchemy.text("SELECT 1"))
        checks["database"] = "ok"
    except Exception as exc:
        checks["database"] = f"error: {exc}"
        ok = False

    # Redis / cache check
    try:
        import redis
        r = redis.Redis.from_url(os.environ.get("REDIS_URL", "redis://localhost:6379"), socket_timeout=2)
        r.ping()
        checks["cache"] = "ok"
    except Exception as exc:
        checks["cache"] = f"error: {exc}"
        ok = False

    return JSONResponse(
        status_code=200 if ok else 503,
        content={"status": "ok" if ok else "degraded", "checks": checks},
    )

Node.js (Express with Anthropic)

// src/health.js
const express = require('express');
const router = express.Router();

router.get('/health', async (req, res) => {
  const checks = {};
  let ok = true;

  // Anthropic API connectivity
  try {
    const response = await fetch('https://api.anthropic.com/v1/models', {
      headers: {
        'x-api-key': process.env.ANTHROPIC_API_KEY,
        'anthropic-version': '2023-06-01',
      },
      signal: AbortSignal.timeout(5000),
    });
    checks.llm_provider = response.ok ? 'ok' : `http_${response.status}`;
    if (!response.ok) ok = false;
  } catch (err) {
    checks.llm_provider = `error: ${err.message}`;
    ok = false;
  }

  // PostgreSQL probe
  try {
    await db.query('SELECT 1');
    checks.database = 'ok';
  } catch (err) {
    checks.database = `error: ${err.message}`;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    checks,
  });
});

module.exports = router;

Verify the endpoint before configuring Vigilmon:

curl -s https://your-llm-app.example.com/health | jq .
# {"status":"ok","checks":{"llm_provider":"ok","database":"ok","cache":"ok"}}

Step 2: Configure Vigilmon HTTP Monitor

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://your-llm-app.example.com/health
3. Check interval: 60 seconds — appropriate for production LLM APIs.
4. Response timeout: 15 seconds — LLM health checks can be slower due to provider round-trips.
5. Expected status code: 200.
6. JSON body assertion:
   • Path: status
   • Expected value: ok
7. Click Save.

Vigilmon immediately begins probing from multiple external probe locations. A non-200 response or a failed JSON assertion triggers an alert to your configured channels, independently of Opik's tracing pipeline.

Multiple endpoints

LLM applications commonly expose multiple endpoints. Create a Vigilmon monitor for each:

| Endpoint | Vigilmon monitor URL | Interval | |---|---|---| | Chat completion API | https://api.example.com/health | 60 s | | RAG retrieval API | https://rag.example.com/health | 60 s | | Agent orchestration endpoint | https://agents.example.com/health | 60 s | | Staging API | https://api-staging.example.com/health | 120 s |

Group all monitors under a Monitor group to see overall LLM platform availability at a glance.

Step 3: Heartbeat Monitor for Evaluation Pipelines and Batch Jobs

Opik tracks evaluation scores for prompts and responses in real time, but offline evaluation pipelines — nightly quality benchmarks, weekly regression tests against golden datasets, scheduled batch completions — run outside the request/response lifecycle. A silently failed evaluation pipeline means your Opik dashboards show stale scores. Use Vigilmon heartbeats to enforce execution guarantees.

# jobs/run_evaluations.py
import os
import requests
import opik
from opik.evaluation import evaluate
from opik.evaluation.metrics import Hallucination, AnswerRelevance

def run_nightly_evals():
    """Run LLM-as-judge evaluations on the past 24 hours of production traces."""
    client = opik.Opik()
    dataset = client.get_dataset("production-sample-v2")

    def evaluate_trace(item):
        response = call_llm(item["input"])
        return {"output": response, "context": item.get("context")}

    evaluate(
        dataset=dataset,
        task=evaluate_trace,
        scoring_metrics=[Hallucination(), AnswerRelevance()],
        experiment_name="nightly-eval",
    )

if __name__ == "__main__":
    try:
        run_nightly_evals()
        requests.post(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)
        print("Evaluation pipeline complete — heartbeat sent")
    except Exception as exc:
        # Deliberate silence — missed heartbeat fires Vigilmon alert
        print(f"Evaluation pipeline failed: {exc}")
        raise

In Vigilmon, create a Heartbeat monitor and set the grace period slightly above your evaluation cadence. For a nightly job, use 25 hours. Store the heartbeat URL in your secrets manager and inject it as VIGILMON_HEARTBEAT_URL.

Docker Compose scheduled evaluation

# docker-compose.yml
services:
  eval-runner:
    image: your-registry/eval-runner:latest
    command: python jobs/run_evaluations.py
    environment:
      - OPIK_API_KEY=${OPIK_API_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - VIGILMON_HEARTBEAT_URL=${VIGILMON_HEARTBEAT_URL}
    restart: "no"

  eval-scheduler:
    image: mcuadros/ofelia:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: daemon --docker
    labels:
      ofelia.job-run.eval-runner.schedule: "@daily"
      ofelia.job-run.eval-runner.container: eval-runner

Step 4: Alert Routing Strategy

Opik alerts on LLM quality signals: hallucination rate increases, answer relevance drops, evaluation score regressions, prompt template performance changes. Vigilmon alerts on infrastructure availability: the LLM API endpoint is unreachable, a health check failed, an evaluation pipeline missed its window. These are distinct failure modes.

| Failure mode | Source | Route to | |---|---|---| | Hallucination rate increase | Opik | ML team Slack #llm-quality | | Evaluation score regression | Opik | ML team + product lead | | LLM API endpoint down | Vigilmon | PagerDuty on-call + Slack #prod-alerts | | Evaluation pipeline missed | Vigilmon | ML team Slack #eval-ops-critical | | TLS certificate expiring | Vigilmon | DevOps team |

Configure Vigilmon alert channels under Alerts → Add channel:

• Email: your on-call distribution list.
• PagerDuty webhook: wire to your primary on-call rotation for endpoint outages.
• Slack webhook: a dedicated #vigilmon-alerts channel separate from Opik dashboards.

Keeping channels separate prevents alert fatigue and speeds incident triage — "our LLM quality is degrading" and "our LLM API is completely unreachable" have very different runbooks.

Step 5: Correlating Opik Evaluations and Vigilmon Alerts During Incidents

When Vigilmon fires a downtime alert on an LLM endpoint, use this checklist to determine root cause:

1. Check Vigilmon probe regions — all failing points to your application or LLM provider. One region failing suggests network routing.
2. Open Opik → check the error rate in production traces for the same time window. A spike in LLM provider errors often precedes an availability failure.
3. Check LLM provider rate limits — provider rate-limit exhaustion can cause your health endpoint to return 503 even when your application code is healthy.
4. Check context window exhaustion — some LLM endpoints fail when requests consistently exceed the model's context limit, which can cause cascading 400/500 errors.
5. Compare timestamps — did Opik show quality degradation before Vigilmon fired? Quality regressions (e.g., from a bad prompt template deployment) can cascade to availability failures if the model starts throwing exceptions.

Step 6: Public Status Page

Opik dashboards are private LLM quality views. Give your API consumers a public-facing status page using Vigilmon.

1. In Vigilmon, go to Status Pages → Create.
2. Add your production LLM API monitor and any critical heartbeat monitors.
3. Configure a custom domain (e.g., status.example.com) or use the Vigilmon subdomain.
4. Embed the status badge in your developer documentation:

<img src="https://vigilmon.online/badge/<monitor-id>.svg" alt="LLM API Status" />

Your API consumers get real-time uptime visibility without requiring access to Opik or your internal infrastructure.

What Vigilmon Adds to an Opik Stack

| Capability | Opik | Vigilmon | |---|---|---| | LLM call traces and evaluations | Yes | No | | LLM-as-judge scoring | Yes | No | | Prompt template version tracking | Yes | No | | Dataset-based regression testing | Yes | No | | External synthetic HTTP probing | No | Yes | | DNS resolution failure detection | No | Yes | | TLS/SSL certificate expiry alerts | No | Yes | | Evaluation pipeline heartbeat monitoring | No | Yes | | Public status page | No | Yes | | Independent of LLM provider health | No | Yes |

Opik gives you deep visibility into the quality and correctness of every LLM call your application makes. Vigilmon gives you the external, synthetic view that tells you what your users actually experience when they try to reach your LLM API. Together they cover every dimension of LLM service reliability: from evaluation quality metrics to customer-facing uptime.

Add external uptime monitoring to your Opik-instrumented LLM application today — register free at vigilmon.online.