tutorial

Monitoring Weaviate: Vector Database Health, Query Latency, and Vigilmon for AI/ML Uptime

"How to monitor Weaviate vector databases in production — node health, import throughput, query latency percentiles, schema consistency, and Vigilmon for AI/ML application uptime."

Monitoring Weaviate: Vector Database Health, Query Latency, and Vigilmon for AI/ML Uptime

Weaviate is increasingly the backbone of production RAG pipelines, semantic search systems, and AI-powered recommendation engines. When Weaviate goes down or degrades, the AI features depending on it silently fail or return stale results. This guide covers the full observability stack for a production Weaviate deployment and how Vigilmon provides external uptime monitoring for AI/ML applications.

Weaviate Architecture and Failure Modes

A production Weaviate deployment consists of:

  • One or more nodes (multi-node cluster for HA)
  • Shards distributed across nodes for each class/collection
  • Vector indexes (HNSW by default) per shard
  • An import pipeline ingesting objects and computing embeddings
  • Query endpoints for semantic search, BM25, and hybrid queries

Common failure modes:

  • Node crash: a node goes offline, its shards become unavailable
  • Import stall: the import pipeline backs up, causing stale vectors
  • Query latency spike: HNSW index rebuilding or GC pauses cause P99 spikes
  • Schema drift: a class definition diverges across nodes in a split-brain scenario
  • Memory pressure: large HNSW indexes consume all available RAM, causing OOM kills

Node Health via the REST API

Weaviate exposes health endpoints out of the box:

# Basic liveness probe
curl http://localhost:8080/v1/.well-known/live
# Returns: {}  with 200 OK if alive

# Readiness probe (node ready to serve traffic)
curl http://localhost:8080/v1/.well-known/ready
# Returns: {}  with 200 OK if ready

# Node and shard status
curl http://localhost:8080/v1/nodes

The /v1/nodes response is the richest health signal:

{
  "nodes": [
    {
      "name": "weaviate-0",
      "status": "HEALTHY",
      "version": "1.24.0",
      "gitHash": "abc123",
      "stats": {
        "shardCount": 4,
        "objectCount": 1250000
      },
      "shards": [
        {
          "name": "shard-abc",
          "class": "Document",
          "objectCount": 312500,
          "vectorIndexingStatus": "READY",
          "vectorQueueLength": 0
        }
      ]
    }
  ]
}

Alert when:

  • Any node status is not HEALTHY
  • vectorIndexingStatus is not READY
  • vectorQueueLength > 0 for more than a few minutes (indicates import backlog)

Import Pipeline Throughput

The import throughput tells you how fast objects are being written and vectorized. Monitor via the Weaviate Python client:

import weaviate
import time

client = weaviate.connect_to_local()

def get_import_stats(class_name: str):
    meta = client.collections.get(class_name)
    # Query current object count
    result = client.collections.get(class_name).aggregate.over_all(
        total_count=True
    )
    return result.total_count

# Track throughput over time
prev_count = get_import_stats("Document")
time.sleep(60)
new_count = get_import_stats("Document")
throughput = (new_count - prev_count) / 60
print(f"Import rate: {throughput:.1f} objects/second")

For Prometheus scraping, Weaviate exposes a /metrics endpoint:

curl http://localhost:2112/metrics | grep -E 'weaviate_object|weaviate_batch'

Key import metrics:

# Objects added per second
weaviate_objects_durations_ms_bucket

# Batch import duration
weaviate_batch_durations_ms_bucket

# Failed batch operations
weaviate_batch_failed_refs_count

Query Latency Percentiles

Query latency is the most user-visible metric for AI applications. Weaviate exposes latency histograms via Prometheus:

# P50, P95, P99 query latency
histogram_quantile(0.50, rate(weaviate_queries_durations_ms_bucket[5m]))
histogram_quantile(0.95, rate(weaviate_queries_durations_ms_bucket[5m]))
histogram_quantile(0.99, rate(weaviate_queries_durations_ms_bucket[5m]))

Set Prometheus alert rules:

groups:
  - name: weaviate
    rules:
      - alert: WeaviateQueryLatencyHigh
        expr: |
          histogram_quantile(0.95, 
            rate(weaviate_queries_durations_ms_bucket[5m])
          ) > 500
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Weaviate P95 query latency > 500ms"

      - alert: WeaviateQueryLatencyCritical
        expr: |
          histogram_quantile(0.99, 
            rate(weaviate_queries_durations_ms_bucket[5m])
          ) > 2000
        for: 2m
        labels:
          severity: critical

      - alert: WeaviateNodeUnhealthy
        expr: weaviate_nodes_status{status!="HEALTHY"} > 0
        for: 1m
        labels:
          severity: critical

Schema Consistency

In a multi-node cluster, schema must be consistent across nodes. Check for drift:

import weaviate
import requests

def check_schema_consistency(nodes: list[str]):
    schemas = {}
    for node in nodes:
        r = requests.get(f"http://{node}:8080/v1/schema", timeout=5)
        schemas[node] = r.json()

    reference = schemas[nodes[0]]
    drifted = []
    for node, schema in schemas.items():
        if schema != reference:
            drifted.append(node)

    return drifted

nodes = ["weaviate-0", "weaviate-1", "weaviate-2"]
drifted = check_schema_consistency(nodes)
if drifted:
    print(f"CRITICAL: schema drift on nodes {drifted}")

Run this check every 5 minutes and alert on any drift — schema inconsistency causes query failures on the drifted node.

Memory and Resource Monitoring

Weaviate's HNSW index lives in memory. Monitor node memory usage:

# If running in Kubernetes
kubectl top pod -l app=weaviate -n weaviate

# Docker resource stats
docker stats weaviate --no-stream

From Prometheus:

# JVM heap usage (if using Java modules)
# Or process memory for Go process
process_resident_memory_bytes{job="weaviate"}

# Memory utilization threshold alert
process_resident_memory_bytes{job="weaviate"} / node_memory_MemTotal_bytes > 0.85

Also monitor vector index build status — a rebuilding HNSW index can spike CPU and memory:

curl http://localhost:8080/v1/nodes | jq '.nodes[].shards[].vectorIndexingStatus'

Prometheus Setup

Enable Weaviate metrics scraping:

# docker-compose.yml
services:
  weaviate:
    image: semitechnologies/weaviate:1.24.0
    environment:
      PROMETHEUS_MONITORING_ENABLED: "true"
      PROMETHEUS_MONITORING_PORT: "2112"
    ports:
      - "8080:8080"
      - "2112:2112"  # Prometheus metrics port

# prometheus.yml
scrape_configs:
  - job_name: weaviate
    static_configs:
      - targets: ["weaviate:2112"]
    scrape_interval: 15s

Vigilmon Integration

AI/ML applications have high expectations for availability — users notice immediately when semantic search returns empty results or recommendation feeds go blank. Vigilmon provides continuous external monitoring of Weaviate from outside your infrastructure.

Step 1 — build a health endpoint

from flask import Flask, jsonify
import requests
import time

app = Flask(__name__)

WEAVIATE_HOST = "http://weaviate:8080"

@app.route("/health/weaviate")
def health():
    results = {}

    # Liveness
    try:
        r = requests.get(f"{WEAVIATE_HOST}/v1/.well-known/live", timeout=3)
        results["live"] = r.status_code == 200
    except Exception as e:
        return jsonify({"live": False, "error": str(e)}), 503

    # Readiness
    try:
        r = requests.get(f"{WEAVIATE_HOST}/v1/.well-known/ready", timeout=3)
        results["ready"] = r.status_code == 200
        if not results["ready"]:
            return jsonify(results), 503
    except Exception as e:
        return jsonify({"ready": False, "error": str(e)}), 503

    # Node health
    try:
        r = requests.get(f"{WEAVIATE_HOST}/v1/nodes", timeout=5)
        nodes = r.json()["nodes"]
        unhealthy = [n["name"] for n in nodes if n["status"] != "HEALTHY"]
        results["unhealthy_nodes"] = unhealthy
        if unhealthy:
            return jsonify(results), 503

        # Check import queue
        queued_shards = [
            s["name"]
            for n in nodes
            for s in n.get("shards", [])
            if s.get("vectorQueueLength", 0) > 1000
        ]
        results["shards_with_queue_backlog"] = queued_shards
    except Exception as e:
        results["nodes_error"] = str(e)

    # Query smoke test
    try:
        start = time.time()
        r = requests.post(
            f"{WEAVIATE_HOST}/v1/graphql",
            json={"query": "{ __typename }"},
            timeout=5
        )
        latency_ms = (time.time() - start) * 1000
        results["query_latency_ms"] = round(latency_ms, 1)
        if latency_ms > 1000:
            results["query_slow"] = True
            return jsonify(results), 503
    except Exception as e:
        results["query_error"] = str(e)
        return jsonify(results), 503

    return jsonify({**results, "status": "ok"}), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9090)

Step 2 — add Vigilmon monitors

In Vigilmon, set up:

  1. HTTP monitor → https://your-host:9090/health/weaviate — interval 60s, expects 200.
  2. HTTP monitor → https://weaviate-host:8080/v1/.well-known/ready — interval 30s (for faster detection).
  3. TCP monitor → weaviate-host:8080 — basic port check.

For AI applications where Weaviate is customer-facing, add the monitors to a public status page so users see the impact of outages in real time.

Step 3 — set alert escalation

Configure multi-channel alerting:

  • P1 (node unhealthy, query failure): immediate PagerDuty page
  • P2 (high latency, import backlog): Slack alert within 5 minutes
  • P3 (schema drift): email to on-call engineer

Alert Thresholds Reference

| Metric | Warning | Critical | |--------|---------|----------| | Node status | Any non-HEALTHY | Any FAILED | | P95 query latency | > 200ms | > 500ms | | P99 query latency | > 500ms | > 2000ms | | Vector queue length | > 100 | > 1000 | | Import error rate | > 0.1% | > 1% | | Memory utilization | > 75% | > 90% | | Schema consistency | drift detected | drift detected |

Conclusion

Weaviate's built-in /v1/.well-known and /v1/nodes endpoints give you the foundation, and Prometheus metrics expose the query latency and import throughput data you need for production alerting. Layer Vigilmon HTTP monitors on top for external uptime verification that's independent of your internal observability stack — so you catch Weaviate outages from the same vantage point as your AI application users. With node health, query latency, import pipeline monitoring, and schema consistency checks in place, your vector database is production-ready.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles