tutorial

Monitoring Patronus AI-Tested LLM Applications with Vigilmon: External Uptime, API Health & Heartbeat Monitoring

Practical guide to complementing Patronus AI's automated LLM evaluation and red-teaming with Vigilmon's external uptime checks — health endpoints, synthetic HTTP monitors, and independent alerting for AI applications.

Patronus AI is an automated LLM evaluation and red-teaming platform. It tests your AI application's robustness against adversarial prompts, jailbreaks, and safety violations — and evaluates production outputs for hallucinations, toxicity, and accuracy regressions. Patronus gives you a continuous inside-out quality and safety view of your LLM application. What it doesn't provide is an independent external health check: a synthetic probe that measures whether your AI application's public endpoints are reachable and responding correctly from the outside. Vigilmon fills that gap. It sits entirely outside your infrastructure and probes your services from multiple external locations, alerting you independently of whether Patronus AI's evaluation pipeline is healthy.

This tutorial covers adding external uptime monitoring to LLM applications already tested with Patronus AI.

What You'll Build

  • A /health endpoint that reflects the real state of your AI application's dependencies
  • A Vigilmon HTTP monitor with response-body assertions
  • A heartbeat monitor for asynchronous red-teaming and evaluation jobs
  • An alert routing strategy that keeps Patronus AI quality alerts and Vigilmon uptime alerts separate but correlated

Prerequisites

  • A Patronus AI account with your LLM application integrated via the Patronus SDK or REST API
  • At least one AI application or inference API endpoint reachable via a public HTTPS URL
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your AI Application

Patronus AI evaluates your LLM outputs through automated test runners and red-team probes. Vigilmon needs an HTTP endpoint it can reach from the outside. Make the health check verify real dependencies — your LLM provider connectivity, guardrail service availability, and any backing database — not just a static 200 that would pass even when critical dependencies are degraded.

Python (FastAPI with OpenAI)

# app/health.py
import os
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

@app.get("/health")
async def health():
    checks = {}
    ok = True

    # LLM provider reachability check
    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            resp = await client.get(
                "https://api.openai.com/v1/models",
                headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
            )
            checks["llm_provider"] = "ok" if resp.status_code == 200 else f"http_{resp.status_code}"
            if resp.status_code != 200:
                ok = False
    except Exception as exc:
        checks["llm_provider"] = f"error: {exc}"
        ok = False

    # Guardrail / content moderation service check
    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            resp = await client.get(
                f"{os.environ['GUARDRAIL_SERVICE_URL']}/ping",
            )
            checks["guardrail_service"] = "ok" if resp.status_code == 200 else f"http_{resp.status_code}"
            if resp.status_code != 200:
                ok = False
    except Exception as exc:
        checks["guardrail_service"] = f"error: {exc}"
        ok = False

    # Database connectivity check
    try:
        import asyncpg
        conn = await asyncpg.connect(os.environ["DATABASE_URL"])
        await conn.fetchval("SELECT 1")
        await conn.close()
        checks["database"] = "ok"
    except Exception as exc:
        checks["database"] = f"error: {exc}"
        ok = False

    status_code = 200 if ok else 503
    return JSONResponse(
        status_code=status_code,
        content={"status": "ok" if ok else "degraded", "checks": checks},
    )

Node.js (Express)

// src/health.js
const express = require('express');
const router = express.Router();

router.get('/health', async (req, res) => {
  const checks = {};
  let ok = true;

  // LLM provider connectivity check
  try {
    const response = await fetch('https://api.openai.com/v1/models', {
      headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
      signal: AbortSignal.timeout(5000),
    });
    checks.llm_provider = response.ok ? 'ok' : `http_${response.status}`;
    if (!response.ok) ok = false;
  } catch (err) {
    checks.llm_provider = `error: ${err.message}`;
    ok = false;
  }

  // Content moderation service check
  try {
    const response = await fetch(`${process.env.GUARDRAIL_SERVICE_URL}/ping`, {
      signal: AbortSignal.timeout(3000),
    });
    checks.guardrail_service = response.ok ? 'ok' : `http_${response.status}`;
    if (!response.ok) ok = false;
  } catch (err) {
    checks.guardrail_service = `error: ${err.message}`;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    checks,
  });
});

module.exports = router;

Verify the endpoint before configuring Vigilmon:

curl -s https://your-ai-app.example.com/health | jq .
# {"status":"ok","checks":{"llm_provider":"ok","guardrail_service":"ok","database":"ok"}}

Step 2: Configure Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-ai-app.example.com/health
  3. Check interval: 60 seconds — appropriate for production AI APIs.
  4. Response timeout: 15 seconds — health checks involving LLM providers can be slower.
  5. Expected status code: 200.
  6. JSON body assertion:
    • Path: status
    • Expected value: ok
  7. Click Save.

Vigilmon immediately begins probing from multiple external probe locations. A non-200 response or a failed JSON assertion triggers an alert independently of Patronus AI's evaluation pipeline.

Multiple endpoints and environments

AI applications tested by Patronus AI often have separate serving endpoints by use case. Mirror each in Vigilmon:

| Endpoint | Vigilmon monitor URL | Interval | |---|---|---| | Chat inference API | https://api.example.com/health | 60 s | | Document analysis API | https://docs-api.example.com/health | 60 s | | Guardrail proxy | https://guardrail.example.com/health | 60 s | | Staging environment | https://api-staging.example.com/health | 120 s |

Group monitors under a single Monitor group in Vigilmon to see your full AI platform availability at a glance.

Step 3: Heartbeat Monitor for Red-Teaming and Evaluation Jobs

Patronus AI runs red-team evaluations and safety tests on a schedule — nightly adversarial probes, pre-deployment safety gates, or weekly hallucination audits. These jobs run outside your request/response lifecycle. Use Vigilmon heartbeats to guarantee these evaluation jobs complete on schedule.

# jobs/nightly_safety_eval.py
import os
import requests
import patronus

def run_safety_evaluation():
    """Run nightly red-team safety evaluation against production model."""
    client = patronus.Client(api_key=os.environ["PATRONUS_API_KEY"])

    # Evaluate a batch of outputs against safety criteria
    results = client.evaluate(
        evaluator="lynx",  # Patronus hallucination detector
        criteria="patronus:hallucination",
        task_input="What is the capital of France?",
        task_output=get_production_sample_outputs(),
    )

    # Fail job if safety threshold is exceeded
    failed = [r for r in results if not r.pass_]
    if len(failed) / len(results) > 0.05:
        raise RuntimeError(
            f"Safety threshold exceeded: {len(failed)}/{len(results)} outputs failed"
        )

if __name__ == "__main__":
    try:
        run_safety_evaluation()
        requests.post(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)
        print("Nightly safety evaluation complete — heartbeat sent")
    except Exception as exc:
        # Deliberate silence — missed heartbeat fires Vigilmon alert
        print(f"Safety evaluation failed: {exc}")
        raise

In Vigilmon, create a Heartbeat monitor with a grace period slightly above your evaluation cadence. For a nightly job, use 25 hours. Store the heartbeat URL in your secrets manager and inject it as VIGILMON_HEARTBEAT_URL.

Kubernetes CronJob for evaluation runs

# k8s/safety-eval.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-safety-eval
spec:
  schedule: "0 1 * * *"  # 1 AM daily
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: eval
            image: your-registry/safety-eval:latest
            env:
            - name: PATRONUS_API_KEY
              valueFrom:
                secretKeyRef:
                  name: patronus-secrets
                  key: api-key
            - name: VIGILMON_HEARTBEAT_URL
              valueFrom:
                secretKeyRef:
                  name: vigilmon-secrets
                  key: heartbeat-url
          restartPolicy: Never

Step 4: Alert Routing Strategy

Patronus AI alerts on LLM safety and quality signals: jailbreak vulnerabilities, rising hallucination rates, toxicity regressions, and failed safety criteria. Vigilmon alerts on infrastructure availability: the endpoint is down, the health check is failing, the evaluation job missed its window. These are distinct failure modes.

| Failure mode | Source | Route to | |---|---|---| | Jailbreak vulnerability detected | Patronus AI | Security team + Slack #ai-safety | | Hallucination rate regression | Patronus AI | ML team Slack #model-quality | | External API endpoint down | Vigilmon | PagerDuty on-call + Slack #prod-alerts | | Safety evaluation job missed | Vigilmon | Slack #ai-ops-critical | | TLS certificate expiring | Vigilmon | DevOps team |

Configure Vigilmon alert channels under Alerts → Add channel:

  • Email: your on-call distribution list.
  • PagerDuty webhook: wire to your primary on-call rotation for endpoint outages.
  • Slack webhook: a dedicated #vigilmon-alerts channel separate from Patronus AI dashboards.

Step 5: Correlating Patronus AI Evaluations and Vigilmon Alerts During Incidents

When Vigilmon fires a downtime alert for an AI application endpoint, use this checklist to determine root cause:

  1. Check Vigilmon probe regions — did all probe locations fail or just one? All failing points to your application or LLM provider. One region failing suggests a network routing issue.
  2. Open Patronus AI → check recent evaluation runs for the same time window. A spike in provider timeouts or API errors during evaluation jobs often foreshadows a serving outage.
  3. Check guardrail service health — a guardrail proxy that sits in the critical path of your inference endpoint can bring down availability even when the LLM provider is healthy.
  4. Check model deployment history — a deployment that introduced a new safety filter can cause the health endpoint to return 503 if the filter service fails to initialize.
  5. Compare timestamps — did Patronus AI evaluation jobs start timing out before Vigilmon fired? If so, the LLM provider issue likely caused the downtime.

Step 6: Public Status Page

Patronus AI evaluation results are private safety and quality metrics visible to your engineering and safety teams. Give customers and API consumers a public-facing status page using Vigilmon.

  1. In Vigilmon, go to Status Pages → Create.
  2. Add your production inference API monitor and any critical heartbeat monitors.
  3. Configure a custom domain (e.g., status.example.com) or use the Vigilmon subdomain.
  4. Embed the status badge in your developer documentation:
<img src="https://vigilmon.online/badge/<monitor-id>.svg" alt="API Status" />

What Vigilmon Adds to a Patronus AI Stack

| Capability | Patronus AI | Vigilmon | |---|---|---| | Automated red-teaming and jailbreak testing | Yes | No | | Hallucination and toxicity evaluation | Yes | No | | Safety criteria enforcement | Yes | No | | Production output sampling and evaluation | Yes | No | | External synthetic HTTP probing | No | Yes | | DNS resolution failure detection | No | Yes | | TLS/SSL certificate expiry alerts | No | Yes | | Scheduled evaluation job heartbeats | No | Yes | | Public status page | No | Yes | | Independent of LLM provider health | No | Yes |

Patronus AI gives you continuous safety and quality visibility into whether your LLM application is producing safe, accurate, and trustworthy outputs. Vigilmon gives you the external, synthetic view that tells you what your users actually experience when they try to reach your AI application. Together they cover every dimension of AI service reliability: from red-team safety testing to customer-facing uptime.

Add external uptime monitoring to your Patronus AI-tested application today — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles