Monitoring Chroma with Vigilmon: Embedding Collection Health, Query Performance & LLM Application Retrieval SLAs

Chroma is the vector store that keeps RAG (Retrieval-Augmented Generation) pipelines grounded in real data. When Chroma becomes unavailable or a collection's embeddings are stale, your LLM application answers questions with outdated or invented context — without any obvious error. Vigilmon provides external uptime and health monitoring for your Chroma deployment, catching server failures, API unavailability, and collection-level issues before your retrieval SLAs are breached.

What You'll Build

• A liveness monitor on Chroma's /api/v1 heartbeat endpoint
• Collection-level health checks via the Chroma REST API
• Persistence layer health monitoring for server-mode deployments
• SSL certificate monitoring for your Chroma endpoint
• Alert thresholds tailored to LLM retrieval SLA requirements

Prerequisites

• A running Chroma instance in server mode (port 8000 by default) or Chroma Cloud
• An accessible endpoint (e.g., https://chroma.example.com)
• A free account at vigilmon.online

Step 1: Verify Chroma's Heartbeat Endpoint

Chroma's server mode exposes a heartbeat at /api/v1:

curl https://chroma.example.com/api/v1

A running Chroma server returns:

{ "nanosecond heartbeat": 1704067200000000000 }

This endpoint is your liveness signal — if it returns anything other than a 200 with a numeric heartbeat, the Chroma server process is down or overloaded.

You can also query the version endpoint for a softer health check:

curl https://chroma.example.com/api/v1/version

Returns the Chroma version string, useful for verifying that a deployment completed successfully.

Authentication: Chroma supports token-based auth. Pass it as: curl -H "Authorization: Bearer YOUR_TOKEN" https://chroma.example.com/api/v1. Chroma Cloud requires this header on all requests.

Step 2: Create a Vigilmon Heartbeat Monitor

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://chroma.example.com/api/v1.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: "heartbeat" (matches the response key).
7. Click Save.

This monitor catches Chroma server process failures, which silently break all retrieval operations in downstream LLM applications using LangChain, LlamaIndex, or direct Chroma clients.

Step 3: Monitor Collections

Chroma organizes embeddings into collections — each collection typically corresponds to a document corpus or knowledge base. List your collections to verify they're available:

curl https://chroma.example.com/api/v1/collections

Returns an array of collection objects:

[
  {
    "id": "abc123",
    "name": "knowledge-base",
    "metadata": { "embedding_model": "text-embedding-3-small" }
  }
]

For each critical collection, monitor its count endpoint to verify embeddings are present:

curl https://chroma.example.com/api/v1/collections/knowledge-base/count

Returns the number of embeddings: { "count": 52000 }.

Add a monitor for each business-critical collection:

1. Add Monitor → HTTP.
2. URL: https://chroma.example.com/api/v1/collections/knowledge-base/count.
3. Check interval: 5 minutes.
4. Expected status: 200.
5. Keyword: "count" (verifies the endpoint responds and the collection exists).
6. Label: Chroma collection: knowledge-base.

A 404 on this endpoint means the collection was accidentally deleted — a catastrophic event for RAG pipelines that can silently result in empty context windows.

Step 4: Monitor Collection Query Availability

Beyond existence checks, verify that collections are queryable. Chroma provides a query endpoint you can probe with a zero-result query:

curl -X POST https://chroma.example.com/api/v1/collections/knowledge-base/query \
  -H "Content-Type: application/json" \
  -d '{
    "query_embeddings": [[0.0, 0.0, 0.0]],
    "n_results": 1
  }'

A healthy response:

{
  "ids": [["doc-001"]],
  "distances": [[0.98]],
  "documents": [["First document content..."]]
}

Add a Vigilmon monitor that confirms query availability:

1. Add Monitor → HTTP (POST).
2. URL: https://chroma.example.com/api/v1/collections/knowledge-base/query.
3. Method: POST.
4. Body: {"query_embeddings": [[0.0]],"n_results": 1}.
5. Expected status: 200.
6. Keyword: "ids".
7. Label: Chroma query available: knowledge-base.

Step 5: Monitor Persistence Layer Health

Chroma in server mode writes embeddings to a SQLite or DuckDB persistence file. Disk pressure or filesystem errors cause silent write failures that don't immediately crash the server but corrupt data over time.

Monitor the Chroma server's ability to read-back data by checking collection count periodically:

curl https://chroma.example.com/api/v1/collections/knowledge-base/count

Compare the count over time — a count that suddenly drops to 0 (while the server is still responding) indicates persistence layer corruption or a database reset.

Add a monitor with a custom keyword to detect an empty collection:

1. Add Monitor → HTTP.
2. URL: https://chroma.example.com/api/v1/collections/knowledge-base/count.
3. Check interval: 10 minutes.
4. Expected status: 200.
5. Keyword: configure an alert when response body is {"count":0} — use Vigilmon's keyword alert to catch the zero-count state.
6. Label: Chroma persistence sanity: knowledge-base.

Step 6: Monitor Embedding Collection Size Trends

For LLM applications with continuous ingestion pipelines (document processors, web crawlers, database sync), embedding count should only grow. A stable or decreasing count signals a broken ingestion pipeline.

Track embedding count via a 10-minute monitor and alert on count regression:

1. Add Monitor → HTTP.
2. URL: https://chroma.example.com/api/v1/collections/knowledge-base/count.
3. Check interval: 10 minutes.
4. Expected status: 200.
5. Keyword: "count".
6. Label: Chroma ingestion health: knowledge-base.

Use Vigilmon's response time graph to visually confirm the embedding count endpoint responds within SLA — slow responses here often precede collection unavailability.

Step 7: SSL Certificate Monitoring

1. Add Monitor → SSL Certificate.
2. Domain: chroma.example.com.
3. Alert when expiry is within: 30 days.
4. Click Save.

LangChain and LlamaIndex clients connect to Chroma using the Python chromadb HTTP client, which validates SSL certificates strictly. A certificate expiry causes immediate SSLError exceptions in your LLM inference code with no graceful fallback.

Step 8: Configure Alert Routing

In Vigilmon under Settings → Notifications:

| Monitor | Trigger | Action | |---|---|---| | Heartbeat /api/v1 | No 200 response | LLM retrieval is broken; page immediately | | Collection existence | 404 on count endpoint | Collection deleted; restore from backup | | Query availability | Non-200 from query endpoint | Index may be corrupted; inspect Chroma logs | | Persistence sanity | Count returns 0 | Persistence layer reset; urgent data recovery | | Ingestion health | Count endpoint slow | Ingestion pipeline stalled or disk full | | SSL certificate | < 30 days to expiry | Renew; LLM client will fail on expiry |

Alert after: 1 failure for heartbeat and collection existence monitors. 2 failures for query availability (brief timeouts during large index builds are normal).

Common Chroma Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Chroma server process crash | Heartbeat monitor; alert in < 60 s | | Collection accidentally deleted | Collection count endpoint returns 404 | | Persistence file corruption | Count drops to 0; sanity monitor fires | | Disk full on Chroma host | Write failures; query response slows then times out | | Ingestion pipeline stalls | Collection size stops growing (manual review) | | Query timeout under high concurrency | Query availability monitor fires | | SSL certificate expires | SSL monitor alerts at 30-day threshold | | Chroma upgrade breaks API | Heartbeat keyword heartbeat absent |

Chroma failures are silent from the LLM's perspective — an empty context window looks the same as a well-retrieved one until a human notices the degraded response quality. Vigilmon's external checks on Chroma's heartbeat, collection availability, and query readiness give you the operational visibility to maintain retrieval SLAs for your AI applications.

Start monitoring Chroma in under 5 minutes — register free at vigilmon.online.