Monitoring VictoriaLogs with Vigilmon

VictoriaLogs is a high-performance, cost-efficient log management solution from VictoriaMetrics. It accepts logs from Fluentd, Logstash, Vector, and other shippers over HTTP and provides a LogsQL query interface for searching and aggregating log data. Unlike heavyweight ELK stacks, VictoriaLogs runs as a single binary with low memory overhead — but that simplicity means there's no built-in availability dashboard. When the VictoriaLogs process crashes or the ingestion endpoint stops accepting data, your log pipeline silently breaks. Vigilmon provides the external visibility layer: health checks on the query API, ingestion endpoint probes, and heartbeat monitors that verify your log shippers are successfully delivering data.

What You'll Set Up

• HTTP health probe for the VictoriaLogs process
• Ingestion endpoint probe to verify the /insert API is accepting data
• Query API check to confirm the search interface is responsive
• Heartbeat from your log shipper to detect silent ingestion failures
• Alerting recommendations for log infrastructure failure modes

Prerequisites

• VictoriaLogs running (Docker, Kubernetes, or bare metal)
• VictoriaLogs accessible on a reachable hostname or IP (default port: 9428)
• A free Vigilmon account

Step 1: Probe the VictoriaLogs Health Endpoint

VictoriaLogs exposes a /health endpoint on port 9428 that returns HTTP 200 when the process is running and ready:

curl http://localhost:9428/health
# ok

Add a monitor in Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter http://victoria-logs.yourdomain.com:9428/health (or https:// if you've placed it behind TLS).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Optionally set Response body contains to ok.
7. Click Save.

For Kubernetes deployments, expose VictoriaLogs through a NodePort or Ingress:

apiVersion: v1
kind: Service
metadata:
  name: victoria-logs
  namespace: logging
spec:
  type: NodePort
  selector:
    app: victoria-logs
  ports:
  - name: http
    port: 9428
    targetPort: 9428
    nodePort: 30928

Then monitor http://<node-ip>:30928/health. This external check catches failures that Kubernetes liveness probes miss: load balancer routing issues and DNS failures that leave your log query frontend unreachable.

Step 2: Verify the Ingestion Endpoint

The VictoriaLogs /insert/elasticsearch/ endpoint accepts log data in Elasticsearch bulk format (used by Logstash, Filebeat, and Vector). Monitor this endpoint with a lightweight synthetic write to detect ingestion failures:

# Test ingestion manually
curl -X POST http://localhost:9428/insert/elasticsearch/_bulk \
  -H 'Content-Type: application/json' \
  --data-binary '{"index":{"_index":"vigilmon-probe"}}
{"_time":"2024-01-15T10:23:00Z","message":"vigilmon-health-check","host":"probe"}'

A healthy VictoriaLogs instance returns HTTP 204.

Add a Vigilmon HTTP monitor that issues a POST with a probe payload:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter http://victoria-logs.yourdomain.com:9428/insert/elasticsearch/_bulk.
3. Set Method to POST.
4. Under Advanced → Request body, paste:

   {"index":{"_index":"vigilmon-probe"}}
   {"_time":"2024-01-15T10:23:00Z","message":"vigilmon-health-check","host":"probe"}
   

5. Add header: Content-Type: application/json.
6. Set Expected HTTP status to 204.
7. Set Check interval to 5 minutes (ingestion probes write data; keep the interval low-frequency).
8. Click Save.

This catches a class of failures the /health check misses: the process is running but the storage backend is full, the data directory is unmounted, or write permissions have changed.

Step 3: Check the Query API

VictoriaLogs exposes a LogsQL query interface at /select/logsql/query. An unresponsive query API means your dashboards and alerts are running blind even if ingestion is working:

curl 'http://localhost:9428/select/logsql/query?query=_time%3A1m&limit=1'

Add a monitor in Vigilmon:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter http://victoria-logs.yourdomain.com:9428/select/logsql/query?query=_time%3A1m&limit=1.
3. Set Expected HTTP status to 200.
4. Set Check interval to 1 minute.
5. Click Save.

This verifies the full query path: log data is stored, indexed, and queryable. A response time spike on this monitor (visible in Vigilmon's response time chart) signals index pressure before it becomes a full outage.

Step 4: Heartbeat from Your Log Shipper

The most reliable VictoriaLogs availability check is a dead-man's-switch from your actual log shipper. Configure Vector (or Fluentd/Logstash) to send a test log entry to Vigilmon's heartbeat URL on a schedule:

Vector configuration (vector.toml)

[sources.heartbeat_timer]
type = "demo_logs"
format = "json"
interval = 300  # every 5 minutes

[transforms.heartbeat_filter]
type = "filter"
inputs = ["heartbeat_timer"]
condition = 'true'

[sinks.vigilmon_heartbeat]
type = "http"
inputs = ["heartbeat_filter"]
uri = "https://vigilmon.online/api/heartbeat/<YOUR_HEARTBEAT_ID>"
method = "GET"
encoding.codec = "json"
batch.max_events = 1

Fluentd configuration

<match vigilmon.heartbeat>
  @type http
  endpoint https://vigilmon.online/api/heartbeat/<YOUR_HEARTBEAT_ID>
  http_method get
  <buffer>
    flush_interval 300s
  </buffer>
</match>

<source>
  @type exec
  command echo '{"status":"heartbeat"}'
  tag vigilmon.heartbeat
  run_interval 5m
</source>

Configure the Vigilmon heartbeat

1. In Vigilmon, click Add Monitor → Heartbeat.
2. Set Expected interval to 5 minutes.
3. Set Grace period to 3 minutes.
4. Paste the heartbeat URL into your shipper config.
5. Restart your shipper to apply the config.

If VictoriaLogs stops accepting data and your shipper's buffer fills and halts, the heartbeat stops arriving and Vigilmon fires an alert — catching silent ingestion failures that health endpoint checks cannot detect.

Step 5: Configure Notifications

1. In Vigilmon, go to Settings → Notifications.
2. Add your Slack webhook, PagerDuty integration key, or on-call email.
3. Assign the notification channel to all monitors.
4. Recommended alert delays:
   • /health check: 2 minutes (process down affects all log visibility)
   • Ingestion endpoint: 5 minutes (allow for transient write backpressure)
   • Query API: 2 minutes (dashboards and alerts go blind immediately)
   • Shipper heartbeat: 10 minutes (allow for shipper buffer drain and reconnect)

VictoriaLogs-Specific Failure Modes to Watch

| Failure mode | Detected by | |---|---| | Process crash / OOM kill | /health HTTP monitor | | Storage disk full | Ingestion endpoint POST monitor | | Index corruption | Query API HTTP monitor | | Shipper disconnected | Heartbeat monitor | | DNS / network routing failure | Multi-region /health consensus | | TLS certificate expiry | Vigilmon SSL cert check on your domain |

What You've Built

| Monitor | Type | Checks | |---|---|---| | /health | HTTP | VictoriaLogs process alive and ready | | Ingestion /insert endpoint | HTTP POST | Write path accepting data | | Query /select/logsql/query | HTTP | Query API responsive | | Log shipper heartbeat | Heartbeat | End-to-end log delivery working |

This four-layer setup gives you visibility into the complete VictoriaLogs data path: process health, write availability, read availability, and actual shipper delivery. With Vigilmon monitoring your log infrastructure, you'll know about VictoriaLogs failures before your on-call team loses the log visibility they need to debug the next incident.