Monitoring VMAlert with Vigilmon

VMAlert is the alerting engine for VictoriaMetrics stacks: it evaluates Prometheus-compatible alerting and recording rules and fires alerts to Alertmanager or other notifiers. The irony of every alerting system is that it can fail silently — VMAlert going down means your alerts stop firing without anyone noticing. Vigilmon solves this by monitoring VMAlert from the outside: health checks, API availability, and heartbeat verification that alert evaluation is actually running.

What You'll Set Up

• HTTP health monitor on VMAlert's built-in /health endpoint
• HTTP monitor on the web UI for availability confirmation
• Cron heartbeat to verify alert rules are being evaluated
• TCP port monitor for the VMAlert listener

Prerequisites

• VMAlert running (standalone or as part of a VictoriaMetrics stack)
• Web UI enabled (-httpListenAddr flag configured)
• A free Vigilmon account

Step 1: Monitor the VMAlert Health Endpoint

VMAlert exposes a /health endpoint at its HTTP listen address (default port 8880). This returns 200 OK when the process is up.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: http://your-vmalert-host:8880/health.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Verify the endpoint manually first:

curl -s http://localhost:8880/health
# Returns: OK

If VMAlert is behind a reverse proxy with TLS termination, use the HTTPS URL and enable Monitor SSL certificate with a 21-day alert window.

Step 2: Monitor the VMAlert Web UI

VMAlert serves a web UI listing all loaded alert groups and their current state at http://your-vmalert-host:8880/. Add a second monitor for this:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: http://your-vmalert-host:8880/.
3. Set Expected HTTP status to 200.
4. Set Check interval to 5 minutes.
5. Click Save.

The web UI returning 200 confirms VMAlert loaded your rules files successfully — a misconfigured rules file causes VMAlert to fail at startup and this check will immediately alert you.

Step 3: Check the Rules API Endpoint

VMAlert exposes its loaded alert groups via the API at /api/v1/rules. This is the same endpoint Grafana Alertmanager integrations use to list active alerts.

Add a monitor:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: http://your-vmalert-host:8880/api/v1/rules.
3. Set Expected HTTP status to 200.
4. Optionally set Response body must contain to "groups" to confirm the rules payload is valid JSON.
5. Click Save.

An empty rules list ({"groups":[]}) or a malformed response indicates VMAlert lost its rules configuration — this check catches that without requiring you to watch the web UI manually.

Step 4: Add a TCP Port Monitor

Add a TCP port monitor as a low-level availability check independent of the HTTP layer:

1. Click Add Monitor → TCP Port.
2. Enter your VMAlert host and port 8880.
3. Set Check interval to 1 minute.
4. Click Save.

A TCP failure when the HTTP check is also failing confirms the process has crashed. A TCP success with an HTTP failure suggests the process is up but the web server is unresponsive — a different failure mode requiring different remediation.

Step 5: Heartbeat Monitoring for Alert Rule Evaluation

VMAlert's health endpoint confirms the process is alive — it does not confirm alert rules are being evaluated on schedule. Use a "deadman" alerting rule combined with a Vigilmon cron heartbeat to verify end-to-end rule evaluation.

Add a recording rule in your VMAlert rules file that sends a heartbeat ping periodically:

# vmalert-rules.yml
groups:
  - name: vigilmon-heartbeat
    interval: 5m
    rules:
      - record: vigilmon:heartbeat
        expr: |
          (
            count(up) * 0
          ) + 1
        # The expr always produces 1 — this rule fires every evaluation cycle

Then add an alerting rule that calls Vigilmon when evaluated:

      - alert: VigilmonHeartbeat
        expr: vigilmon:heartbeat == 1
        labels:
          severity: heartbeat
        annotations:
          summary: "VMAlert rule evaluation is healthy"

Configure VMAlert's notifier to send this alert to a webhook receiver that pings Vigilmon:

# In your Alertmanager config, route the 'heartbeat' severity to a webhook
route:
  routes:
    - match:
        severity: heartbeat
      receiver: vigilmon-heartbeat

receivers:
  - name: vigilmon-heartbeat
    webhook_configs:
      - url: 'https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID'

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 5 minutes.
3. Copy the heartbeat URL into your Alertmanager config.
4. Click Save.

If VMAlert stops evaluating rules — due to a crash, rules reload failure, or datasource connectivity issue — the heartbeat ping stops and Vigilmon alerts within 10 minutes.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your preferred channel (Slack, email, webhook).
2. Set Consecutive failures before alert to 2 on HTTP monitors — VMAlert may briefly return non-200 during a hot reload triggered by rules file changes.
3. For the cron heartbeat monitor, keep the default alerting threshold (missing 1 expected ping triggers alert) since rule evaluation gaps are always worth investigating.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health | :8880/health | Process crash | | HTTP web UI | :8880/ | Rules load failure, startup error | | HTTP rules API | :8880/api/v1/rules | Rules config loss | | TCP port | :8880 | Network-layer failure | | Cron heartbeat | Heartbeat URL | Rule evaluation halt, datasource outage |

VMAlert is the component that tells you when everything else is broken — which makes it the most important thing to monitor. With Vigilmon watching the health endpoint, rules API, and a deadman heartbeat, you have external assurance that your alerting pipeline is working, independent of the VictoriaMetrics stack it protects.