Monitoring VictoriaMetrics with Vigilmon

VictoriaMetrics is a fast, cost-efficient time-series database and monitoring solution built in Go. It ingests billions of metrics, handles complex MetricsQL queries, and powers production observability stacks — but the stack that monitors everything else still needs its own uptime watchdog. Vigilmon closes that loop, giving you external availability checks for VictoriaMetrics server, ingest endpoints, query APIs, and every cluster component without adding another internal dependency to your monitoring chain.

What You'll Set Up

• VictoriaMetrics server availability check (single-node and cluster modes)
• Data ingest endpoint health (Prometheus remote write, InfluxDB, OpenTSDB)
• MetricsQL query API response time monitoring
• vmagent scrape job health via heartbeat
• vmalert alerting service availability
• vmauth authentication proxy health
• Cluster component availability (vminsert / vmselect / vmstorage)
• Cron heartbeat for vmbackup jobs

Prerequisites

• VictoriaMetrics single-node (port 8428) or cluster mode (ports 8480/8481/8482) running and accessible
• Optional: vmagent, vmalert, vmauth deployed in your environment
• A free Vigilmon account

Step 1: Monitor the VictoriaMetrics HTTP API

VictoriaMetrics exposes a health endpoint at /health on its main HTTP port. In single-node mode that is port 8428; in cluster mode, each component exposes its own health route.

Single-node

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the health URL: http://victoriametrics.internal:8428/health
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Set Expected response body contains to ok (VictoriaMetrics returns the literal string ok).
7. Click Save.

The /health endpoint returns ok when the process is alive and not ok when it is shutting down, giving you a precise liveness signal.

Step 2: Monitor Data Ingest Endpoints

VictoriaMetrics accepts data over three protocols. Each ingest path should be probed independently so you know which pipeline is broken.

Prometheus remote write

Add an HTTP monitor for the remote write endpoint:

http://victoriametrics.internal:8428/api/v1/write

Set Expected HTTP status to 204 — VictoriaMetrics returns 204 on an empty-body POST and 400 on malformed data, so a 204 from a periodic probe confirms the path is open.

You can also probe it with a minimal metric payload from a script:

#!/bin/bash
# Minimal Prometheus remote write probe — send a single metric
PAYLOAD=$(printf '\x00' | curl -s -o /dev/null -w "%{http_code}" \
  -X POST http://victoriametrics.internal:8428/api/v1/write \
  -H "Content-Type: application/x-protobuf" \
  --data-binary @-)
echo "HTTP $PAYLOAD"
curl -s https://vigilmon.online/heartbeat/WRITE_PROBE_KEY

InfluxDB line protocol

http://victoriametrics.internal:8428/write

An HTTP GET to this URL returns 405 (Method Not Allowed) — which still confirms the endpoint is reachable. Set Expected HTTP status to 405 in Vigilmon for a lightweight availability check.

OpenTSDB HTTP

http://victoriametrics.internal:8428/api/put

Same pattern: a GET returns 405. Add an HTTP monitor with expected status 405.

Step 3: Monitor the MetricsQL Query API

Slow or unavailable query responses are the most visible VictoriaMetrics failure mode. Monitor the query path with a lightweight instant query:

1. Add an HTTP monitor for:

   http://victoriametrics.internal:8428/api/v1/query?query=1
   

2. Set Expected HTTP status to 200.
3. Set Expected response body contains to "status":"success".
4. Set Check interval to 1 minute.
5. Enable Response time alert and set the threshold to 2000 ms — a query for the literal 1 should resolve in milliseconds; anything over 2 seconds signals storage pressure or a compaction stall.

For the range query API, add a second monitor:

http://victoriametrics.internal:8428/api/v1/query_range?query=1&start=now-5m&end=now&step=60

Step 4: Monitor vmagent Scrape Job Health

vmagent runs as a separate process (default port 8429) and scrapes Prometheus targets on behalf of VictoriaMetrics. Its own health endpoint follows the same pattern:

http://vmagent.internal:8429/health

Add an HTTP monitor with expected status 200 and expected body ok.

For deeper scrape-job health, vmagent exposes a targets page. Set up a cron heartbeat to confirm vmagent is successfully completing scrape cycles:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected interval to match your scrape interval (e.g. 60 seconds for a 1-minute scrape interval).
3. Copy the heartbeat URL.
4. Add a scrape-success probe to your vmagent config or a sidecar script:

#!/bin/bash
# Check that vmagent has no scrape errors in the last 5 minutes
ERRORS=$(curl -s http://vmagent.internal:8429/metrics | grep 'vmagent_scrape_series_failed_total' | awk '{print $2}')
if [ "$ERRORS" = "0" ]; then
  curl -s https://vigilmon.online/heartbeat/VMAGENT_KEY
fi

Step 5: Monitor vmalert

vmalert evaluates alerting and recording rules against VictoriaMetrics (default port 8880). Add an HTTP availability monitor:

http://vmalert.internal:8880/health

Expected status: 200, expected body: ok.

Also monitor the rules evaluation endpoint:

http://vmalert.internal:8880/api/v1/rules

Set Expected HTTP status to 200 and Expected response body contains to "status":"success". If vmalert loses connectivity to VictoriaMetrics, the rules endpoint starts returning errors before your dashboards notice missing alert evaluations.

Step 6: Monitor vmauth

vmauth is the authentication proxy that sits in front of VictoriaMetrics in multi-tenant deployments (default port 8427). Add an HTTP availability monitor:

http://vmauth.internal:8427/health

Expected status: 200, expected body: ok.

If vmauth goes down, all upstream clients lose access to VictoriaMetrics even if the storage node is healthy — making this one of the highest-impact single points of failure in a secured cluster.

Step 7: Monitor vmbackup Jobs with a Heartbeat

vmbackup runs as a one-shot process on a schedule (e.g. from cron or a Kubernetes CronJob). Use a Vigilmon cron heartbeat to confirm it completes:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected interval to match your backup schedule (e.g. 1440 minutes for daily backups).
3. Copy the heartbeat URL.
4. Wrap your vmbackup call:

#!/bin/bash
vmbackup \
  -storageDataPath=/var/lib/victoriametrics \
  -dst=s3://my-bucket/vm-backup \
  -credsFilePath=/etc/vm/credentials.json \
  && curl -s https://vigilmon.online/heartbeat/VMBACKUP_KEY

If vmbackup exits non-zero (storage error, S3 credential failure, disk full), the heartbeat never fires and Vigilmon alerts after the expected interval passes.

Step 8: Monitor Cluster Components

In cluster mode, VictoriaMetrics splits into three services. Monitor each independently:

vminsert (port 8480)

http://vminsert.internal:8480/health

vminsert is the write path — if this goes down, metric ingestion stops silently for any client without write-failure alerting.

vmselect (port 8481)

http://vmselect.internal:8481/health

vmselect is the read path. Add both a health monitor and a lightweight query probe:

http://vmselect.internal:8481/select/0/prometheus/api/v1/query?query=1

vmstorage (port 8482)

http://vmstorage.internal:8482/health

vmstorage is the persistence layer. Add the health monitor and set a response-time alert at 500 ms — a sluggish vmstorage almost always indicates disk I/O pressure before it escalates to an outage.

Step 9: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your preferred channel (Slack, email, or webhook).
2. Set Consecutive failures before alert to 2 on all HTTP monitors — VictoriaMetrics restarts take a few seconds and should not generate false positives on the first probe failure.
3. For the query API monitor, set Response time alert to 2000 ms.
4. For vmstorage, set response time alert to 500 ms to catch I/O issues early.

Summary

| Monitor | Target | What It Catches | |---|---|---| | VictoriaMetrics health | :8428/health | Process crash, shutdown | | Prometheus ingest | :8428/api/v1/write | Write path unavailable | | InfluxDB ingest | :8428/write | InfluxDB path unavailable | | OpenTSDB ingest | :8428/api/put | OpenTSDB path unavailable | | MetricsQL query | :8428/api/v1/query?query=1 | Query API errors, slow reads | | vmagent health | :8429/health | Scraper down | | vmagent heartbeat | Heartbeat URL | Scrape cycle missed | | vmalert health | :8880/health | Alert evaluation down | | vmauth health | :8427/health | Auth proxy down | | vmbackup heartbeat | Heartbeat URL | Backup job failure | | vminsert (cluster) | :8480/health | Write path down | | vmselect (cluster) | :8481/health | Read path down | | vmstorage (cluster) | :8482/health | Storage layer down or slow |

VictoriaMetrics is the backbone of your observability stack — but a backbone needs its own safety net. With Vigilmon checking every ingest path, query API, and cluster component from the outside, you get independent uptime verification that does not rely on VictoriaMetrics itself being healthy enough to report its own problems.