Monitoring Netdata with Vigilmon: Web UI, API Health, Agent Heartbeat & SSL Alerts

Netdata is a high-resolution, real-time monitoring agent deployed on every node in your infrastructure. It collects thousands of metrics per second per host and exposes them through a local web UI and REST API. When Netdata's agent goes down, you lose real-time visibility into CPU, memory, disk I/O, and network for that host — and you may not notice until something else fails. When Netdata Cloud loses connectivity to an agent, the host disappears silently from your dashboards. Vigilmon gives you external health checks on Netdata itself so your monitoring tool has a monitor.

What You'll Build

• A monitor on Netdata's web UI to detect process crashes
• An /api/v1/info endpoint check that validates the agent is collecting data
• An agent heartbeat check using the alarms API
• A collector health API monitor
• SSL certificate monitoring for your Netdata domain

Prerequisites

• Netdata agent running (v1.35+) on a publicly reachable host or behind a reverse proxy
• A domain or IP address for Netdata (e.g., https://netdata.example.com or http://host:19999)
• A free account at vigilmon.online

Step 1: Verify Netdata's Web UI Endpoint

Netdata's built-in web server runs on port 19999 by default. The root path returns the Netdata dashboard:

curl -s http://localhost:19999/ | head -5

For external monitoring, you'll typically expose Netdata through a reverse proxy (nginx, Caddy, Traefik) with HTTPS. Verify your public endpoint:

curl -o /dev/null -s -w "%{http_code}" https://netdata.example.com/

A running Netdata returns HTTP 200. If the agent crashes, the web server stops responding entirely — either connection refused or a timeout.

Firewall note: Port 19999 is not exposed to the public internet in a default Netdata install. Either use ufw allow 19999 (not recommended for production) or expose Netdata behind a reverse proxy with authentication. Use the HTTPS URL for Vigilmon monitoring.

Step 2: Monitor the /api/v1/info Endpoint

Netdata's REST API exposes agent information at /api/v1/info. This endpoint confirms the agent is running, collecting metrics, and connected:

curl https://netdata.example.com/api/v1/info

A healthy Netdata agent returns a JSON object:

{
  "version": "1.44.0",
  "uid": "abc123...",
  "mirrored_hosts": 1,
  "alarms": {
    "normal": 120,
    "warning": 2,
    "critical": 0
  },
  "collectors": [...]
}

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://netdata.example.com/api/v1/info.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: "version" (always present in a healthy response).
7. Label: Netdata API info.
8. Click Save.

This monitor catches:

• Netdata process crashes (web server stops responding)
• Configuration errors that cause the API to return errors instead of JSON
• Memory exhaustion causing Netdata to become unresponsive

Step 3: Agent Heartbeat via the Alarms API

Netdata's alarms API at /api/v1/alarms provides a real-time snapshot of all active alarms and the agent's current alarm state. Polling this endpoint acts as an agent heartbeat — if the endpoint goes silent, the agent is down:

curl "https://netdata.example.com/api/v1/alarms?all"

A healthy response includes an alarms key with per-alarm entries:

{
  "hostname": "myhost",
  "latest_alarm_log_unique_id": 42,
  "status": true,
  "now": 1719744000,
  "alarms": { ... }
}

1. Add Monitor → HTTP.
2. URL: https://netdata.example.com/api/v1/alarms.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: "hostname" (confirms the agent responded with structured alarm data).
7. Label: Netdata agent heartbeat.
8. Click Save.

Why a separate heartbeat monitor? The /api/v1/info endpoint is light and fast. The alarms endpoint exercises a slightly deeper path through the agent — it has to query the alarm registry. Using both gives you redundant signal and distinguishes between a crashed web server (both fail) and a degraded alarm subsystem (alarms API fails, info passes).

Step 4: Collector Health API

Netdata's /api/v1/charts endpoint lists all active collectors and the charts they produce. A missing chart means a collector stopped:

curl https://netdata.example.com/api/v1/charts | python3 -m json.tool | head -30

A healthy response lists hundreds of charts under the charts key:

{
  "hostname": "myhost",
  "version": "1.44.0",
  "charts": {
    "system.cpu": { ... },
    "system.ram": { ... },
    ...
  }
}

1. Add Monitor → HTTP.
2. URL: https://netdata.example.com/api/v1/charts.
3. Check interval: 5 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: "system.cpu" (the CPU chart is always present on any running Linux/macOS host).
7. Label: Netdata collector health.
8. Click Save.

If the collectors API returns an empty or malformed response, Netdata's data collection pipeline has stalled — usually caused by a plugin crash or a resource exhaustion event.

Step 5: Monitor SSL Certificates

If Netdata is exposed through a reverse proxy with TLS, monitor the certificate:

1. Add Monitor → SSL Certificate.
2. Domain: netdata.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again at: 14 days, 7 days, 3 days.
5. Click Save.

An expired Netdata SSL certificate means:

• Netdata Cloud agents can no longer connect to the parent node
• Cross-node metric streaming (Netdata's stream receiver) fails TLS validation
• You lose multi-host visibility in Netdata Cloud

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, set up your alert channels (email, Slack, PagerDuty, webhook):

| Monitor | Trigger | Response | |---|---|---| | Web UI (/) | Non-200 or timeout | Netdata process crash; restart the agent | | /api/v1/info | Non-200 or keyword missing | Agent API down; check systemctl status netdata | | Alarms heartbeat | Non-200 or keyword missing | Agent stopped responding; OOM or crash likely | | Collector health | Non-200 or keyword missing | Collector pipeline stalled; check Netdata logs | | SSL certificate | < 30 days to expiry | Renew cert; streaming connections will fail |

Alert sensitivity: Set the /api/v1/info and alarms monitors to alert after 1 consecutive failure. Netdata is a continuous-collection system — a single missed interval is already a data gap.

Common Netdata Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | netdata process OOM killed | Web UI and API monitors fire within 60 s | | Plugin crash (e.g., python.d) | Collector health fires; charts for that plugin disappear | | Netdata upgrade fails to start | All API monitors fire simultaneously | | Reverse proxy misconfiguration | All monitors fire; SSL and web UI alert first | | SSL certificate expires | SSL monitor alerts; Cloud streaming fails silently | | Disk full (RRD database writes stall) | Agent may still respond to API but /api/v1/info shows stale data | | Netdata Cloud parent loses connection | Alarms API may still pass locally; Cloud dashboard shows host offline | | High-cardinality plugins consuming all CPU | API response times increase; timeout-based alerts fire |

Netdata is designed to be always-on infrastructure monitoring — but infrastructure monitoring tools need monitoring too. Vigilmon provides external visibility into Netdata's own health: the web UI, the API, the agent heartbeat, collector pipeline, and SSL certificate are all checked from outside your host. When Netdata goes down silently, Vigilmon fires the alert so you can restore visibility before the next incident.

Monitor your Netdata agents free — register at vigilmon.online.