tutorial

How to Monitor Agenta with Vigilmon

How to monitor Agenta with Vigilmon — API health checks, LLM playground availability, evaluation pipeline endpoint monitoring, and SSL certificate alerts for your open-source LLM development platform.

Agenta is the open-source LLM application development and evaluation platform that teams use to manage prompts, run A/B tests on prompt variants, build evaluation datasets, and deploy LLM-powered applications with production observability. When Agenta's backend is down, prompt engineers cannot iterate on variants, evaluation pipelines stop producing quality metrics, and the observability data that reveals production regressions stops flowing. Vigilmon gives you external visibility into Agenta's availability: the REST API, LLM playground endpoint, evaluation service, and SSL certificates — whether you self-host Agenta on your own infrastructure or use Agenta Cloud.

What You'll Build

  • An HTTP monitor on Agenta's backend health endpoint
  • A playground availability check to verify prompt iteration is accessible
  • An evaluation service monitor to confirm quality pipelines are running
  • SSL certificate monitoring for your Agenta deployment domain
  • An alerting setup that separates backend failures from evaluation pipeline failures

Prerequisites

  • Agenta self-hosted (Docker Compose or Kubernetes) or Agenta Cloud account
  • Your Agenta backend URL (e.g., https://agenta.yourcompany.com or https://cloud.agenta.ai)
  • Agenta API key from Settings → API Keys
  • A free account at vigilmon.online

Step 1: Understand Agenta's Health Endpoints

Agenta's FastAPI backend exposes OpenAPI documentation and a health check path:

# Self-hosted Agenta
curl https://agenta.yourcompany.com/api/health
# Returns: {"status": "ok"}

# Or check the OpenAPI docs endpoint (always available when FastAPI is running)
curl https://agenta.yourcompany.com/api/openapi.json
# Returns: JSON schema

For Agenta Cloud:

curl https://cloud.agenta.ai/api/health
# Returns health status JSON

A 500 or connection error from the health endpoint means the FastAPI backend is down, and prompt management, variant comparison, and evaluation features are all unavailable.

Step 2: Create a Vigilmon HTTP Monitor for the Backend Health

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://agenta.yourcompany.com/api/health (or https://cloud.agenta.ai/api/health).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ok (present in a healthy Agenta backend response).
  7. Label: Agenta backend health.
  8. Click Save.

This monitor catches:

  • Agenta FastAPI server crashes or container restarts
  • Docker Compose service failures (e.g., the backend service exits)
  • Database (MongoDB or PostgreSQL) connection failures causing 5xx responses
  • Redis connectivity issues that prevent celery workers from queuing evaluations
  • Kubernetes pod OOM kills after large prompt batch operations

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Agenta backend is down, all prompt experiments are blocked, and teams working on LLM quality improvements are completely stalled.

Step 3: Monitor the Agenta Web Application

The Agenta playground is the primary interface for prompt engineers. Monitor it separately from the backend API:

curl https://agenta.yourcompany.com
# Returns HTML with Agenta application shell
  1. Add Monitor → HTTP.
  2. URL: https://agenta.yourcompany.com (your Agenta deployment root).
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Agenta (appears in the web application HTML title).
  7. Label: Agenta web playground.
  8. Click Save.

When the web app monitor fires but the backend health monitor stays green, the issue is in the Next.js frontend container or the nginx reverse proxy — not the FastAPI backend. This pattern is common during frontend deployments or static asset caching failures.

Step 4: Monitor the Evaluation Service Endpoint

Agenta's evaluation pipeline runs LLM quality assessments (faithfulness, relevancy, exact match) as background celery tasks. The evaluation API is a separate concern from the playground:

# Check that the evaluations endpoint responds
curl -H "Authorization: ApiKey YOUR_API_KEY" \
  https://agenta.yourcompany.com/api/evaluations
# Returns: {"evaluations": [...]} or 200 with empty list
  1. Add Monitor → HTTP.
  2. URL: https://agenta.yourcompany.com/api/evaluations.
  3. HTTP headers: Authorization: ApiKey YOUR_API_KEY.
  4. Check interval: 5 minutes.
  5. Response timeout: 20 seconds.
  6. Expected status: 200.
  7. Label: Agenta evaluation API.
  8. Click Save.

Evaluation tasks can run for minutes per batch. A slow response from the evaluations endpoint (close to the 20-second timeout) may indicate a celery worker queue backlog rather than a full outage — investigate worker scaling if alerts are intermittent.

Step 5: Monitor the LLM Gateway Endpoint

Agenta proxies LLM calls through its own gateway to enable observability and cost tracking. If the LLM gateway is down, all prompt completions fail even when the playground UI is accessible:

curl https://agenta.yourcompany.com/api/completions/health
# Or check the proxy endpoint directly
curl -I https://agenta.yourcompany.com/api/chat/completions
# Expect 401 (auth required) — confirms the gateway is listening
  1. Add Monitor → HTTP.
  2. URL: https://agenta.yourcompany.com/api/completions/health (adjust to your deployment's gateway health path).
  3. Check interval: 2 minutes.
  4. Expected status: 200 or 401.
  5. Label: Agenta LLM gateway.
  6. Click Save.

Step 6: Monitor SSL Certificates

Agenta's web application and API must have valid TLS certificates. An expired certificate blocks prompt engineers from accessing the playground and breaks evaluation pipelines calling the API:

openssl s_client -connect agenta.yourcompany.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: agenta.yourcompany.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days.
  5. Click Save.

Self-hosted cert-manager note: Agenta's Docker Compose deployment often uses Let's Encrypt via Traefik or Caddy. Certificate renewal can fail silently if the ACME HTTP-01 challenge is blocked by a firewall rule or if rate limits are hit during testing. A 30-day alert window gives ample time to investigate renewal failures.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Runbook action | |---|---|---| | Backend health | Non-200 or keyword missing | Check Docker/K8s logs for FastAPI container; verify DB and Redis connectivity | | Web playground | Non-200 or keyword missing | Frontend container or nginx issue; check frontend logs; backend likely healthy | | Evaluation API | Non-200 or timeout | Check celery worker status; verify evaluation queue depth; scale workers if needed | | LLM gateway | Connection error or 5xx | LLM proxy is down; check gateway container logs; prompt completions will fail | | SSL certificate | < 30 days to expiry | Check Traefik/Caddy/cert-manager renewal logs; manually renew if auto-renewal failed |

Alert after: 1 consecutive failure for backend health and LLM gateway monitors. 2 consecutive failures for the web playground and evaluation API monitors.

Agenta Self-Hosted Architecture Monitoring Checklist

For self-hosted Agenta on Docker Compose, add monitors for each service layer:

| Component | Monitor type | What to watch | |---|---|---| | FastAPI backend (backend) | HTTP health check | /api/health → 200 | | Next.js frontend (web) | HTTP availability | Root URL → 200 + keyword | | MongoDB / PostgreSQL | TCP port check | Port 27017 or 5432 accessible from Vigilmon | | Redis (celery broker) | TCP port check | Port 6379 accessible | | Reverse proxy (nginx/Traefik) | HTTP + SSL | Root URL + certificate expiry |

Add TCP monitors for MongoDB and Redis by selecting Add Monitor → TCP with the host and port of your Agenta database tier.

Common Agenta Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | FastAPI container crash | Backend health fires; all API operations fail | | MongoDB connection lost | Backend health returns 500; playground loads but operations fail | | Redis celery broker down | Evaluation API becomes slow or times out; evaluation tasks queue but never execute | | Frontend container restart | Web playground monitor fires; backend health stays green | | LLM provider API key expired | LLM gateway returns 401/500; prompt completions fail while health endpoint stays green | | SSL certificate expired | SSL monitor fires; all browser and API access fails | | Out-of-memory on evaluation batch | Backend health intermittent; monitor for recovery after OOM kill | | Docker Compose update failure | Multiple monitors fire; watch for partial service restarts |

Agenta is the control plane for your LLM application quality — when it fails, prompt engineers lose the ability to iterate, evaluation pipelines stop producing quality signals, and production regressions go undetected. Vigilmon gives you external monitoring across every layer of your Agenta deployment so you catch failures before they block your team's LLM development workflow.

Start monitoring Agenta in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles