tutorial

How to Monitor Milvus Vector Database with Vigilmon

Milvus is the leading open-source vector database powering AI applications — from semantic search and recommendation engines to retrieval-augmented generatio...

Milvus is the leading open-source vector database powering AI applications — from semantic search and recommendation engines to retrieval-augmented generation (RAG) pipelines. When Milvus goes down or degrades, your AI features go with it: search results vanish, embedding lookups time out, and LLM-backed features return empty or stale responses.

In this tutorial you'll set up comprehensive monitoring for Milvus using Vigilmon — covering health checks, TCP port monitoring, and real-time alerts.

Why Milvus needs dedicated monitoring

Milvus has a distributed, multi-component architecture that creates several unique failure modes:

  • Component failures — Milvus 2.x splits responsibilities across QueryNode, DataNode, IndexNode, and Proxy. Any one can fail silently while the others appear healthy.
  • etcd and MinIO dependencies — Milvus depends on etcd for metadata and MinIO (or local storage) for vector data. If either goes down, collections become unqueryable even though Milvus itself is "running."
  • Memory pressure — vector indexes are loaded entirely into memory. A collection load failure leaves queries returning zero results without an obvious error.
  • gRPC port saturation — under heavy concurrent load, the gRPC port (19530) can stop accepting new connections while existing ones keep working, causing partial service degradation.
  • Embedding pipeline disconnects — if your embedding service upstream of Milvus dies, inserts stop working but Milvus itself remains healthy — the failure is invisible to Milvus-level metrics.

External monitoring that probes the actual API surface is the most reliable way to detect all of these scenarios before users do.

What you'll need

  • A running Milvus instance (standalone or distributed)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Verify Milvus health endpoints

Milvus exposes two built-in endpoints you can monitor directly:

HTTP health check (port 9091):

curl http://your-milvus-host:9091/healthz
# Returns: {"status":"healthy"} or {"status":"unhealthy", "reason":"..."}

gRPC API port (19530) — the primary service port used by all SDKs.

For standalone Milvus started with Docker Compose:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-13T19-46-17Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.0
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      etcd:
        condition: service_healthy
      minio:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      timeout: 20s
      retries: 3

networks:
  default:
    name: milvus

The 9091 port exposes the Prometheus metrics and health check endpoint — keep it accessible for monitoring.

Step 2: Add a custom health check wrapper

The built-in /healthz endpoint checks whether Milvus is running but doesn't verify that your collections are actually queryable. For deeper validation, add a lightweight health check service:

# health_server.py
from http.server import HTTPServer, BaseHTTPRequestHandler
from pymilvus import connections, Collection, utility
import json
import os

MILVUS_HOST = os.getenv("MILVUS_HOST", "localhost")
MILVUS_PORT = int(os.getenv("MILVUS_PORT", "19530"))
COLLECTION_NAME = os.getenv("MILVUS_COLLECTION", "")

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return

        try:
            connections.connect("default", host=MILVUS_HOST, port=MILVUS_PORT)
            
            status = {"milvus": "ok"}
            
            if COLLECTION_NAME:
                if utility.has_collection(COLLECTION_NAME):
                    col = Collection(COLLECTION_NAME)
                    col.load()
                    status["collection"] = "loaded"
                    status["num_entities"] = col.num_entities
                else:
                    status["collection"] = "missing"
            
            connections.disconnect("default")
            
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps(status).encode())

        except Exception as e:
            self.send_response(503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({"status": "error", "detail": str(e)}).encode())

    def log_message(self, format, *args):
        pass  # suppress default access logs

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8080), HealthHandler)
    print("Health server running on :8080")
    server.serve_forever()

Run it alongside your Milvus instance:

pip install pymilvus
MILVUS_HOST=localhost MILVUS_PORT=19530 MILVUS_COLLECTION=my_collection python health_server.py

This gives Vigilmon a proper HTTP endpoint that actually exercises the Milvus connection and verifies your collection is loaded and queryable.

Step 3: Set up HTTP monitoring in Vigilmon

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to http://your-milvus-host:9091/healthz (or your custom health wrapper at :8080/health)
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • (Optional) Response body contains: "status":"healthy" or "milvus":"ok"
  6. Save the monitor

Vigilmon will probe your Milvus health endpoint from multiple regions every minute. A timeout or non-200 response immediately triggers an incident.

Step 4: Monitor the gRPC and HTTP ports with TCP checks

Milvus serves all SDK traffic over port 19530 (gRPC). A TCP check confirms the port is reachable independent of the HTTP health endpoint:

  1. In Vigilmon, go to Monitors → New Monitor
  2. Choose TCP Port as the type
  3. Set hostname to your Milvus server and port to 19530
  4. Save the monitor

Add a second TCP monitor for port 9091 if you want to track the metrics/health port separately.

For distributed Milvus, add TCP monitors for each component's port:

| Component | Default Port | Monitor | |-----------|-------------|---------| | Proxy (gRPC) | 19530 | TCP | | Proxy (HTTP) | 9091 | HTTP + TCP | | QueryNode metrics | 9091 | TCP | | DataNode metrics | 9091 | TCP | | MinIO API | 9000 | TCP | | etcd | 2379 | TCP |

Step 5: Configure alert channels

When Milvus fails, AI features fail with it — you want instant notification.

  1. Go to Alert Channels → Add Channel
  2. Choose Webhook for Slack/Discord/PagerDuty integration or Email for email alerts
  3. For Slack, paste your Slack incoming webhook URL
  4. Assign the channel to your Milvus monitors

Vigilmon sends a notification payload like:

{
  "monitor_name": "Milvus gRPC :19530",
  "status": "down",
  "host": "your-milvus-host",
  "port": 19530,
  "started_at": "2026-01-15T14:30:00Z",
  "duration_seconds": 120
}

Set up an escalation by adding both a Slack channel (immediate) and an email channel (as a fallback if Slack is also down).

Step 6: Monitor Milvus dependencies

Milvus is only as healthy as its dependencies. Add monitors for:

etcd (port 2379):

# Verify etcd is serving
curl http://your-etcd-host:2379/health
# Returns: {"health":"true"}

Add an HTTP monitor pointing to http://your-etcd-host:2379/health.

MinIO (port 9000):

curl http://your-minio-host:9000/minio/health/live
# Returns HTTP 200 when healthy

Add an HTTP monitor for the MinIO live endpoint.

Object storage (S3-compatible): If you're using AWS S3 or another S3-compatible store, use Vigilmon's heartbeat monitors to track your embedding pipeline's S3 write operations.

Step 7: Create a status page for AI infrastructure

If you expose Milvus to internal teams or external partners, a public status page clarifies which layer is failing when AI features degrade:

  1. Go to Status Pages → New Status Page
  2. Name it something like "AI Infrastructure"
  3. Add your Milvus monitors grouped by component:
    • Milvus Proxy (HTTP health)
    • Milvus gRPC port
    • etcd health
    • MinIO health
  4. Publish the page

Now when your ML team reports "search is broken," they can check the status page before filing a support ticket — and you'll already have an incident open if something is actually down.

Complete monitoring setup summary

| Monitor | Type | Endpoint | Catches | |---------|------|----------|---------| | Milvus health | HTTP | :9091/healthz | Milvus process health | | Custom health wrapper | HTTP | :8080/health | Collection load failures | | Milvus gRPC | TCP | :19530 | Port saturation, process crash | | etcd | HTTP | :2379/health | Metadata store failures | | MinIO | HTTP | :9000/minio/health/live | Object storage failures |

What's next

  • Heartbeat monitors — add a heartbeat check for your nightly index rebuild jobs; if the rebuild stops running, Vigilmon alerts you before stale indexes cause search quality degradation
  • SSL monitoring — if you front Milvus with an Attu or custom HTTPS proxy, Vigilmon checks your TLS certificate expiry automatically
  • Response time tracking — Vigilmon's response time history shows you when gRPC connection latency starts climbing before it causes timeouts

Start monitoring your Milvus instance for free at vigilmon.online — no credit card required, monitors start in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles