Monitoring VictoriaLogs with Vigilmon

VictoriaLogs delivers Elasticsearch-scale log storage at a fraction of the resource cost — 30× lower disk usage and 15× lower RAM than Elasticsearch for equivalent workloads. It accepts logs from FluentBit, Logstash, Promtail, and OpenTelemetry Collector via compatible APIs, all from a single victoria-logs-prod binary. But when VictoriaLogs goes down, your entire log pipeline goes dark: no ingestion, no search, no alerting on log patterns. Vigilmon gives you real-time monitoring of process health, ingestion rates, query latency, disk capacity, and API compatibility endpoints.

What You'll Set Up

VictoriaLogs process health check via the /health endpoint
Log ingestion rate and ingestion error rate monitoring
Query latency (P99) alerting via Prometheus metrics
Disk and memory usage heartbeats
Elasticsearch-compatible and Loki-compatible API health checks
Web UI liveness monitoring

Prerequisites

VictoriaLogs running (single binary victoria-logs-prod) with HTTP port 9428 accessible
A free Vigilmon account

Step 1: Monitor VictoriaLogs Process Health

VictoriaLogs exposes a health endpoint at GET /health on port 9428. If the process crashes, all log ingestion and search stops immediately.

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
Enter the URL: http://your-victorialogs-host:9428/health
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Keyword: OK — VictoriaLogs returns the literal string OK on a healthy response.
Click Save.

This single monitor covers the entire VictoriaLogs binary. Downtime on /health means log ingestion, LogsQL search, and all API endpoints are unavailable.

If VictoriaLogs is behind a reverse proxy (nginx, Caddy), also add an HTTP monitor on the proxy's public URL so you catch both process failures and proxy misconfigurations.

Step 2: Monitor Log Ingestion Rate

A sudden drop in log ingestion rate is one of the most useful early signals — it means your upstream log shippers (FluentBit, Vector, Promtail) have lost their connection to VictoriaLogs, even if the process itself is alive.

VictoriaLogs exposes Prometheus-compatible metrics at GET /metrics. Key ingestion metric:

vm_rows_inserted_total{type="log"}

Add a metrics endpoint monitor:

Add Monitor → Type: HTTP / HTTPS.
URL: http://your-victorialogs-host:9428/metrics
Check interval: 1 minute.
Keyword: vm_rows_inserted_total — confirms the metrics endpoint is returning valid data.
Save.

For ingestion rate alerting, scrape /metrics with Prometheus or VictoriaMetrics and add an alert rule:

# prometheus-rules.yml
- alert: VictoriaLogsIngestionDrop
  expr: rate(vm_rows_inserted_total{type="log"}[5m]) < (avg_over_time(rate(vm_rows_inserted_total{type="log"}[5m])[1h:5m]) * 0.8)
  for: 5m
  annotations:
    summary: "VictoriaLogs ingestion rate dropped >20%"

This fires when the 5-minute ingestion rate drops more than 20% from the 1-hour average — a reliable signal that upstream shippers are failing.

Step 3: Monitor Ingestion Error Rate

Ingestion errors indicate format mismatches, protocol incompatibilities, or storage issues. Even one ingestion error is worth investigating.

Check the ingestion error counter from /metrics:

vm_http_request_errors_total{path=~"/insert/.*|/elasticsearch/.*|/loki/.*"}

Add a heartbeat monitor from a cron script:

#!/bin/bash
ERRORS=$(curl -s http://localhost:9428/metrics | \
  grep 'vm_http_request_errors_total' | \
  awk '{sum += $2} END {print sum+0}')

# Compare to previous reading stored in a state file
PREV=$(cat /tmp/victorialogs_errors.txt 2>/dev/null || echo 0)
echo "$ERRORS" > /tmp/victorialogs_errors.txt
NEW_ERRORS=$((ERRORS - PREV))

if [ "$NEW_ERRORS" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/$INGESTION_TOKEN"
fi

Run every 5 minutes via cron. Set Vigilmon heartbeat expected interval to 10 minutes. Any new ingestion errors in the last 5 minutes stop the heartbeat.

Step 4: Monitor Query Latency (P99)

VictoriaLogs query latency spikes indicate memory or CPU pressure — the query cache is being evicted, or full-text scans are hitting disk under load.

Key metrics from /metrics:

vm_request_duration_seconds{path="/select/logsql/query",quantile="0.99"}

Add a synthetic query latency check:

#!/bin/bash
START=$(date +%s%3N)
curl -s -G "http://localhost:9428/select/logsql/query" \
  --data-urlencode "query=*" \
  --data-urlencode "start=now-1m" \
  --data-urlencode "limit=1" > /dev/null
END=$(date +%s%3N)
LATENCY_MS=$((END - START))

# Alert if P99-equivalent synthetic latency > 5000ms
if [ "$LATENCY_MS" -lt 5000 ]; then
  curl -s "https://vigilmon.online/heartbeat/$LATENCY_TOKEN"
fi

Run every 5 minutes. Set heartbeat expected interval to 10 minutes. Latency above 5 seconds triggers an alert — investigate CPU/RAM usage or tune --search.maxConcurrentRequests.

Step 5: Monitor Disk Usage

VictoriaLogs compressed log storage grows with ingestion rate minus retention pruning. Disk exhaustion causes all writes to fail.

#!/bin/bash
# Default storage path is victoria-logs-data/ in the working directory
STORAGE_PATH="${VICTORIALOGS_DATA:-/var/lib/victoria-logs-data}"
USAGE=$(df "$STORAGE_PATH" | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$USAGE" -lt 80 ]; then
  curl -s "https://vigilmon.online/heartbeat/$DISK_TOKEN"
fi

Run every 5 minutes. Set Vigilmon heartbeat interval to 10 minutes. Alert at >80% disk usage to give yourself time to expand storage or adjust --retentionPeriod before writes fail.

Also monitor disk growth rate vs. ingest rate to catch retention enforcement failures:

#!/bin/bash
# If growth rate is accelerating despite stable ingest rate, retention may not be running
SIZE_NOW=$(du -sb "${VICTORIALOGS_DATA:-/var/lib/victoria-logs-data}" 2>/dev/null | cut -f1)
PREV=$(cat /tmp/victorialogs_size.txt 2>/dev/null || echo 0)
echo "$SIZE_NOW" > /tmp/victorialogs_size.txt

GROWTH=$((SIZE_NOW - PREV))
# Growth in 5 minutes should not exceed 500MB for most deployments — tune as needed
if [ "$GROWTH" -lt 524288000 ]; then
  curl -s "https://vigilmon.online/heartbeat/$RETENTION_TOKEN"
fi

Step 6: Monitor Memory Usage

VictoriaLogs holds its in-memory index and query cache in RAM. Sustained memory above 80% system RAM leads to OOM kills and restarts.

#!/bin/bash
PID=$(pgrep -f victoria-logs-prod | head -1)
if [ -z "$PID" ]; then exit 1; fi

RSS_KB=$(awk '/VmRSS/{print $2}' /proc/"$PID"/status)
TOTAL_KB=$(grep MemTotal /proc/meminfo | awk '{print $2}')
USAGE_PCT=$((RSS_KB * 100 / TOTAL_KB))

if [ "$USAGE_PCT" -lt 80 ]; then
  curl -s "https://vigilmon.online/heartbeat/$MEM_TOKEN"
fi

Run every 2 minutes. Set Vigilmon heartbeat interval to 5 minutes. Memory above 80% of system RAM warrants investigation into active query concurrency or in-flight ingest batch sizes.

Step 7: Monitor Elasticsearch and Loki API Compatibility

VictoriaLogs provides Elasticsearch-compatible (/elasticsearch/bulk) and Loki-compatible (/loki/api/v1/push) ingestion endpoints. These are used by FluentBit, Logstash, Promtail, and Grafana Agent respectively. If they return errors, your log shippers fail silently.

Elasticsearch bulk API health:

Add Monitor → Type: HTTP / HTTPS.
URL: http://your-victorialogs-host:9428/elasticsearch/bulk
Method: POST (send an empty body or a valid no-op payload)
Expected status: 200 (or 400 for empty body — either confirms the endpoint is alive).
Check interval: 5 minutes.

Loki push API health:

Add Monitor → Type: HTTP / HTTPS.
URL: http://your-victorialogs-host:9428/loki/api/v1/push
Method: POST
Expected status: 200 or 400 (endpoint alive).
Check interval: 5 minutes.

These monitors confirm the compatibility endpoints are responding — separate from the main process health check, which covers only /health.

Step 8: Monitor the Web UI

VictoriaLogs includes a built-in web UI for LogsQL queries at the root path (/). If the web UI returns non-200, it indicates a server-side error even when the health endpoint passes.

Add Monitor → Type: HTTP / HTTPS.
URL: http://your-victorialogs-host:9428/
Check interval: 5 minutes.
Expected status: 200.
Keyword: VictoriaLogs — confirms the web UI HTML loaded correctly, not a redirect or error page.
Save.

Step 9: Configure Alerting

In Vigilmon, navigate to Alert Channels and connect your notification methods:

Email — async alerts for disk and memory thresholds
Slack or Discord webhook — real-time team visibility on ingestion drops and query latency
PagerDuty — for production log pipelines where log loss has compliance implications

Recommended alert policy:

| Monitor | Condition | Priority | |---|---|---| | /health endpoint | Non-200 or missing OK | Page immediately | | Elasticsearch bulk API | Heartbeat missed | Alert within 5 min | | Loki push API | Heartbeat missed | Alert within 5 min | | Ingestion error rate | Heartbeat missed | Alert within 10 min | | Ingestion rate | Heartbeat missed | Alert within 10 min | | Query latency P99 | Heartbeat missed | Alert within 10 min | | Disk usage | Heartbeat missed | Alert within 10 min | | Memory usage | Heartbeat missed | Alert within 5 min | | Web UI | Non-200 | Alert within 5 min |

Conclusion

VictoriaLogs delivers exceptional resource efficiency for self-hosted log aggregation, but a crashed binary or full disk means your entire log pipeline goes silent — no alerts on application errors, no compliance audit trail, no debugging context. With Vigilmon monitoring the health endpoint, ingestion rate, error rate, query latency, disk and memory usage, and API compatibility endpoints, you know the moment VictoriaLogs starts degrading — not when a developer wonders why their log queries are failing. Add the heartbeat monitors, wire up alert channels, and your log infrastructure is fully observable.