How to Monitor Haystack NLP Pipelines with Vigilmon (Latency, Retriever Health & Alerts)

Haystack is one of the most popular open-source frameworks for building production NLP and RAG (Retrieval-Augmented Generation) systems. When your Haystack pipeline powers a customer-facing search or question-answering feature, pipeline failures directly impact user experience — and the failures are often subtle: a document store that's too slow, a retriever returning stale embeddings, or an LLM component timing out silently.

In this tutorial you'll set up end-to-end monitoring for your Haystack application using Vigilmon, covering API health, pipeline latency, and component-level alerts.

Why Haystack applications need external monitoring

Haystack pipelines are multi-component systems. Each component — DocumentStore, Retriever, Reader, Generator — can degrade independently:

DocumentStore unavailable — your Elasticsearch or Weaviate cluster is down; the Haystack app process is healthy but every query fails with a connection error
Retriever returning empty results — an index mapping change or embedding dimension mismatch causes the retriever to return zero documents; the pipeline completes successfully but with garbage output
Slow embedding generation — the embedding model is under load or has been swapped; retriever latency spikes from 50ms to 3s
LLM component timeout — the generator (OpenAI, Hugging Face, or local) hits rate limits or is unreachable; requests queue up and the pipeline times out
Pipeline configuration regression — a pipeline.yaml change breaks component wiring; requests return 500 errors while the Flask/FastAPI wrapper is still running
Memory leak after long operation — a batch indexing job exhausts memory; subsequent queries fail while the process remains alive

External monitoring from outside your infrastructure is the only way to confirm that the full request path — from user query to ranked answer — is actually working.

What you'll need

A Haystack application with an HTTP API (Flask, FastAPI, or the built-in Haystack REST API)
A DocumentStore running (Elasticsearch, OpenSearch, Weaviate, Qdrant, etc.)
A free Vigilmon account (sign up takes 30 seconds)

Step 1: Add a health endpoint to your Haystack API

If you're using the Haystack REST API (haystack-rest), it already ships with a /health endpoint. For custom FastAPI wrappers, add one explicitly:

# app.py
from fastapi import FastAPI
from haystack import Pipeline
from haystack.document_stores.elasticsearch import ElasticsearchDocumentStore

app = FastAPI()

# Initialize pipeline at startup
document_store = ElasticsearchDocumentStore(host="localhost", port=9200)
pipeline = Pipeline()
# ... add your retrievers, readers, generators

@app.get("/health")
async def health_check():
    """Basic liveness check."""
    return {"status": "ok", "service": "haystack-api"}

@app.get("/ready")
async def readiness_check():
    """Deep readiness check — confirms document store connectivity."""
    try:
        # Ping the document store
        doc_count = document_store.get_document_count()
        return {
            "status": "ready",
            "document_count": doc_count,
            "document_store": "connected"
        }
    except Exception as e:
        return {"status": "degraded", "error": str(e)}, 503

@app.post("/query")
async def query(request: dict):
    result = pipeline.run(query=request["query"])
    return result

Test both endpoints:

curl http://localhost:8000/health
# {"status":"ok","service":"haystack-api"}

curl http://localhost:8000/ready
# {"status":"ready","document_count":15420,"document_store":"connected"}

The /ready endpoint is more valuable for monitoring because it confirms the DocumentStore connection, not just the HTTP process.

Step 2: Expose your Haystack API publicly

For Vigilmon to monitor from external regions, your API needs a public URL. Use a reverse proxy:

nginx:

server {
    listen 443 ssl;
    server_name haystack.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/haystack.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/haystack.yourdomain.com/privkey.pem;

    location /health {
        proxy_pass http://127.0.0.1:8000/health;
    }

    location /ready {
        proxy_pass http://127.0.0.1:8000/ready;
    }

    location /query {
        proxy_pass http://127.0.0.1:8000/query;
        proxy_read_timeout 60s;
    }
}

Docker Compose with Traefik:

version: "3.8"
services:
  haystack-api:
    image: your-registry/haystack-api:latest
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.haystack.rule=Host(`haystack.yourdomain.com`)"
      - "traefik.http.routers.haystack.tls.certresolver=letsencrypt"
      - "traefik.http.services.haystack.loadbalancer.server.port=8000"

  elasticsearch:
    image: elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    volumes:
      - es_data:/usr/share/elasticsearch/data

volumes:
  es_data:

Step 3: Set up HTTP monitoring in Vigilmon

With the API reachable, configure Vigilmon monitors:

Monitor 1: Liveness check

Log in to vigilmon.online → Monitors → New Monitor
Type: HTTP / HTTPS
URL: https://haystack.yourdomain.com/health
Check interval: 1 minute
Expected response: status code 200, body contains "status":"ok"
Save

Monitor 2: Readiness / DocumentStore check

Monitors → New Monitor → HTTP / HTTPS
URL: https://haystack.yourdomain.com/ready
Check interval: 2 minutes
Expected response: status code 200, body contains "status":"ready"
Save

The readiness monitor is your DocumentStore early-warning system. If Elasticsearch restarts or loses its index, this check catches it before users start getting empty search results.

Step 4: Add response time monitoring for pipeline latency

Haystack pipeline latency is a first-class metric. A retriever that normally responds in 200ms but suddenly takes 3s usually indicates embedding cache miss, DocumentStore overload, or an LLM component slowdown.

Set response time alerts on both monitors:

Open the readiness monitor → Alerts → Response Time
Set: Alert if response time exceeds 2000ms
Save

For the liveness monitor:

Open the health monitor → Alerts → Response Time
Set: Alert if response time exceeds 500ms

A slow /health endpoint (which should be near-instant) indicates the Python process itself is under load.

Step 5: Monitor your DocumentStore directly

Your Haystack pipeline is only as healthy as its underlying DocumentStore. If you're running Elasticsearch, OpenSearch, or Weaviate, add direct TCP and HTTP monitors for them too.

Elasticsearch HTTP monitor:

Monitors → New Monitor → HTTP / HTTPS
URL: http://elasticsearch.yourdomain.com:9200/_cluster/health
Check interval: 2 minutes
Expected response: status 200, body contains "status":"green" (or "yellow" if you run a single node)
Save

TCP port monitor for Elasticsearch:

Monitors → New Monitor → TCP Port
Host: elasticsearch.yourdomain.com
Port: 9200
Check interval: 1 minute
Save

If your document store goes down, Vigilmon will alert you independently of the Haystack API health checks — giving you a clearer picture of where the fault lies.

Step 6: Configure alert routing

For a production Haystack application, set up tiered alerts:

Immediate Slack alert — fires as soon as the health check fails:

Settings → Notifications → Add Channel → Slack
Paste your Slack webhook URL
Assign to all Haystack monitors under Alert Contacts

PagerDuty escalation — fires after 5 minutes of continuous failure:

Settings → Notifications → Add Channel → PagerDuty
Paste your PagerDuty integration key
Set alert delay to 5 minutes
Assign to the readiness monitor

Webhook for custom alerting — if you want to trigger auto-remediation (e.g., restart the Elasticsearch container):

Settings → Notifications → Add Channel → Webhook
URL: https://your-ops-tool.yourdomain.com/haystack-down
Vigilmon will POST to this URL with incident details when the monitor fails

Step 7: Create a status page for dependent services

If your Haystack pipeline is a shared service consumed by multiple internal teams or external customers, publish a status page:

Status Pages → New Status Page
Add the liveness and readiness monitors
Optionally add the Elasticsearch monitor
Name it "Search & QA API"
Share the URL in your internal developer docs

Use Maintenance Windows when reindexing large document collections (which can cause temporary DocumentStore degradation) to suppress false alerts and communicate scheduled downtime.

Monitoring checklist for Haystack in production

| Component | Monitor type | What it catches | |---|---|---| | Haystack API process | HTTP /health | Process crash, port unreachable | | DocumentStore connection | HTTP /ready | ES/Weaviate down, index missing | | Elasticsearch cluster | HTTP /_cluster/health | Red/yellow cluster state | | Elasticsearch TCP port | TCP | Port unreachable, firewall change | | Pipeline latency | Response time threshold | Slow retriever, LLM timeout |

Conclusion

Haystack pipelines are powerful but multi-layered: a failure in any component — the document store, the embedding model, the reader, or the generator — can silently degrade output quality or cause complete outages. With Vigilmon you get:

Liveness and readiness monitors that distinguish process failures from DocumentStore failures
Response time alerts that catch retriever slowdowns before they become user-facing
DocumentStore-level monitoring that gives you component-level fault isolation
Status pages to communicate NLP pipeline health to dependent teams