How to Monitor Ragas with Vigilmon

Ragas is the open-source evaluation framework for RAG (Retrieval-Augmented Generation) pipelines that data teams use to measure faithfulness, answer relevancy, context precision, and context recall — giving teams data-driven quality signals before and after deploying LLM applications to production. When the Ragas evaluation service is degraded, quality gates fail silently, regressions ship undetected, and LLM application teams lose the feedback loop that keeps RAG pipeline quality high. Vigilmon gives you external visibility into your Ragas evaluation infrastructure: the API server, LLM judge endpoints, result storage, and SSL certificates — so your RAG quality pipeline is never silently broken.

What You'll Build

• An HTTP monitor on the Ragas evaluation API server
• An LLM judge endpoint check to verify evaluation scoring is functional
• A results storage endpoint monitor to confirm evaluation data is being persisted
• SSL certificate monitoring for your Ragas deployment
• An alerting setup that isolates evaluation service failures from LLM backend failures

Prerequisites

• Ragas deployed as a service (FastAPI wrapper, Docker container, or as part of a larger MLOps platform like LangSmith, Langfuse, or a custom FastAPI application)
• Your Ragas API endpoint URL (e.g., https://ragas.yourcompany.com)
• LLM backend configured (OpenAI, Azure OpenAI, or a self-hosted model) for Ragas judges
• A free account at vigilmon.online

Step 1: Understand Ragas Deployment Patterns

Ragas is a Python library — it does not ship with a built-in HTTP server by default. Teams typically deploy Ragas in one of these patterns:

Pattern A: FastAPI evaluation service

# A custom FastAPI wrapper around Ragas
from fastapi import FastAPI
from ragas import evaluate

app = FastAPI()

@app.get("/health")
def health():
    return {"status": "ok"}

@app.post("/evaluate")
async def run_evaluation(dataset: dict):
    # run ragas evaluation
    ...

Pattern B: Integrated into an MLOps platform Ragas runs inside LangSmith, Langfuse, or Weights & Biases as an evaluation provider. In this case you monitor the platform's API rather than Ragas directly.

Pattern C: CI/CD evaluation job Ragas runs as a GitHub Actions or GitLab CI job. In this case, Vigilmon monitors the artifact storage endpoint where evaluation results are published.

This tutorial focuses on Pattern A (the most common production deployment for teams with continuous evaluation requirements).

Step 2: Create a Vigilmon HTTP Monitor for the Evaluation API Health

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://ragas.yourcompany.com/health.
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: ok (present in a healthy FastAPI health response).
7. Label: Ragas evaluation API health.
8. Click Save.

This monitor catches:

• Ragas FastAPI server crashes or container OOM kills
• Python dependency failures (e.g., missing OpenAI client library after a dependency update)
• LLM client initialization errors that prevent the server from starting
• Docker container restarts after SIGKILL from the orchestrator
• Kubernetes pod evictions due to memory pressure during large evaluation batches

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Ragas service is down, quality gate checks in your CI/CD pipeline will time out or return errors, potentially blocking all LLM application deployments.

Step 3: Monitor the Evaluation Endpoint

The evaluation endpoint is the core of the Ragas service. Monitor it to confirm the system can accept and process evaluation requests:

# Send a minimal evaluation request to verify the endpoint responds
curl -X POST https://ragas.yourcompany.com/evaluate \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"question": "ping", "answer": "pong", "contexts": ["test context"], "ground_truth": "pong"}'
# Returns: {"faithfulness": ..., "answer_relevancy": ..., ...}

For a lightweight liveness check that does not consume LLM tokens, create a dedicated probe endpoint:

@app.get("/ready")
def readiness():
    # Check that the LLM client is configured without making an API call
    try:
        assert openai_client is not None
        return {"status": "ready", "llm_backend": "configured"}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "error", "detail": str(e)})

1. Add Monitor → HTTP.
2. URL: https://ragas.yourcompany.com/ready.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ready.
7. Label: Ragas evaluation readiness.
8. Click Save.

Step 4: Monitor the LLM Backend Dependency

Ragas uses LLM calls to compute metrics like faithfulness and answer relevancy. If your LLM backend (OpenAI, Azure OpenAI, or a self-hosted model) is unreachable, all evaluations will fail with timeout errors even when the Ragas server itself is healthy:

# For OpenAI-backed Ragas
curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer YOUR_OPENAI_KEY"
# Returns: list of available models

1. Add Monitor → HTTP.
2. URL: https://api.openai.com/v1/models (or your Azure OpenAI endpoint).
3. HTTP headers: Authorization: Bearer YOUR_OPENAI_KEY.
4. Check interval: 5 minutes.
5. Response timeout: 15 seconds.
6. Expected status: 200.
7. Keyword: data (present in OpenAI model list responses).
8. Label: Ragas LLM backend (OpenAI).
9. Click Save.

When the LLM backend monitor fires independently, Ragas evaluations will time out or return incomplete metrics (faithfulness and answer relevancy require LLM calls; exact match and BLEU do not). This distinction helps you understand which evaluation metrics are affected without waiting for the full evaluation batch to fail.

Step 5: Monitor Result Storage

Ragas evaluation results are typically persisted to a database, object storage (S3/GCS), or a time-series store for trend analysis. If result storage is down, evaluations run but results are lost:

# If results are stored in a REST API (e.g., your own evaluation tracker)
curl https://ragas.yourcompany.com/results
# Returns: list of past evaluation runs

1. Add Monitor → HTTP.
2. URL: https://ragas.yourcompany.com/results (or your evaluation results API).
3. Check interval: 5 minutes.
4. Response timeout: 20 seconds.
5. Expected status: 200.
6. Label: Ragas result storage.
7. Click Save.

If results are stored in S3 or GCS, monitor the storage endpoint:

1. Add Monitor → HTTP.
2. URL: https://your-bucket.s3.amazonaws.com/ragas/results/.healthcheck (a small probe file you upload to verify bucket access).
3. Check interval: 10 minutes.
4. Expected status: 200.
5. Label: Ragas S3 result storage.

Step 6: Monitor SSL Certificates

All API calls between your CI/CD pipeline and the Ragas evaluation service must use valid TLS. An expired certificate blocks evaluation jobs:

openssl s_client -connect ragas.yourcompany.com:443 2>/dev/null | openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: ragas.yourcompany.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days.
5. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Runbook action | |---|---|---| | API health | Non-200 or keyword missing | Check Ragas container logs; verify Python environment and dependency installation | | Evaluation readiness | Non-200 or keyword missing | LLM client not configured; check environment variables for API keys | | LLM backend | Non-200 or timeout | OpenAI/Azure degradation; check LLM provider status page; evaluation metrics degraded | | Result storage | Non-200 or timeout | Database or S3 connectivity issue; evaluation results may be lost until resolved | | SSL certificate | < 30 days to expiry | Renew certificate; check cert-manager or Certbot auto-renewal |

Alert after: 1 consecutive failure for the API health and readiness monitors. 2 consecutive failures for the LLM backend and result storage monitors.

On-call routing: Route Ragas evaluation failures to the ML platform team or data engineering on-call rotation — not the application SRE team, since resolution requires Python/LLM expertise.

Ragas Evaluation Metrics and Monitoring Context

Ragas computes the following metric categories. Understanding which require LLM calls helps you triage partial outages:

| Metric | Requires LLM call | Vigilmon visibility | |---|---|---| | Faithfulness | Yes (LLM judge) | LLM backend monitor catches failures | | Answer Relevancy | Yes (LLM judge) | LLM backend monitor catches failures | | Context Precision | Yes (LLM judge) | LLM backend monitor catches failures | | Context Recall | Yes (LLM judge) | LLM backend monitor catches failures | | Answer Correctness | Yes (LLM + embedding) | LLM backend monitor catches failures | | Answer Similarity | Yes (embedding model) | LLM backend monitor catches failures | | Exact Match | No | API health monitor is sufficient | | BLEU / ROUGE | No | API health monitor is sufficient |

Common Ragas Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Ragas container OOM during large evaluation | API health fires; restart container with more memory | | OpenAI rate limit hit | LLM backend returns 429; evaluations time out; health endpoint stays green | | OpenAI API key expired | LLM backend returns 401; readiness monitor fires | | S3 result storage bucket permission error | Result storage monitor fires; evaluation runs but data is lost | | SSL certificate expired | SSL monitor fires; all CI/CD evaluation jobs fail with TLS errors | | Python dependency conflict after update | API health fails to start; container exits immediately | | Redis/celery queue full (async evaluation) | Evaluation requests accepted but never processed; API health green, jobs stall | | Database for result storage down | Result storage monitor fires; Ragas can still run evaluations but cannot persist them |

Ragas is the quality gate that keeps your RAG application from regressing in production. When the evaluation infrastructure fails, regressions ship undetected and the data-driven quality loop that LLM teams depend on breaks down silently. Vigilmon gives you external monitoring across every layer of your Ragas deployment — the evaluation API, LLM backend, result storage, and SSL certificates — so you know immediately when your quality pipeline needs attention.

Start monitoring Ragas in under 5 minutes — register free at vigilmon.online.