tutorial

How to Monitor Netdata Cloud with Vigilmon (Agent Connectivity, Streaming & ML Anomaly Health)

Netdata Cloud is a real-time infrastructure monitoring SaaS that aggregates metrics from Netdata agents running on your servers, containers, and Kubernetes c...

Netdata Cloud is a real-time infrastructure monitoring SaaS that aggregates metrics from Netdata agents running on your servers, containers, and Kubernetes clusters. Each Netdata agent streams metrics to the Cloud with per-second granularity, giving you the fastest feedback loop in the monitoring ecosystem. But that speed comes with complexity: the streaming pipeline, parent-child node synchronization, and ML anomaly detection all have to work in concert for the platform to deliver its promise.

In this tutorial you'll set up external monitoring for your Netdata Cloud deployment using Vigilmon — covering agent connectivity, streaming replication health, alert notification delivery, ML anomaly detection status, and parent-child node sync. Free tier, no credit card required.

Why Netdata Cloud needs external monitoring

Netdata Cloud's architecture — edge agents → streaming replication → cloud ingestion → ML pipeline — means a failure at any layer can make your infrastructure look healthy when it isn't:

  • Agent connectivity drops — a network policy change or firewall rule blocks the agent's outbound connection to Netdata Cloud; the Cloud shows the node as "unreachable" but no alert fires if you haven't set up external monitoring
  • Streaming replication stalls — a parent node in a hub-and-spoke setup disconnects, taking all its child nodes offline with it; child nodes show as offline in the Cloud but the parent's own metrics look fine
  • Alert notifications stop delivering — Netdata Cloud routes alerts through notification integrations (Slack, PagerDuty, email); a stale webhook URL or expired token silently swallows all alerts
  • ML anomaly detection falls behind — Netdata's Anomaly Advisor runs an ML model per metric; under high node count or metric cardinality load, the model training pipeline can fall behind and stop issuing anomaly scores
  • Parent-child sync breaks — a configuration mismatch between a parent streaming node and its children causes the child's metrics to appear in the parent's namespace but stop updating

External monitoring with Vigilmon gives you ground truth: the Netdata Cloud UI is reachable, the API responds, agent connectivity is healthy, and alerts are flowing — verified from outside your infrastructure.

What you'll need

  • A Netdata Cloud account with at least one Space
  • One or more Netdata agents installed on your infrastructure
  • A free Vigilmon account
  • Netdata Cloud API token (generate from User Settings → API Keys)

Step 1: Monitor Netdata Cloud reachability

Netdata Cloud's web UI and API are hosted at app.netdata.cloud. While this is a SaaS endpoint you don't control, you still need to know when it's unreachable from your network — because a Netdata Cloud outage or a local network issue that blocks access to it will look identical from your users' perspective.

  1. Log in to Vigilmon and click Add Monitor.
  2. Choose HTTP(S) as the monitor type.
  3. Enter https://app.netdata.cloud/api/v1/spaces as the URL.
  4. Set check interval to 60 seconds.
  5. Under Request Headers, add Authorization: Bearer YOUR_NETDATA_CLOUD_TOKEN.
  6. Under Expected Status Code, enter 200.
  7. Click Save.

This monitor confirms that Netdata Cloud's API is reachable from your monitoring region and that your token is still valid. If the API starts returning 401 (token expired) or 503 (service down), you'll know immediately.

Step 2: Monitor agent connectivity

Each Netdata agent is a "node" in your Netdata Cloud Space. When agents disconnect, the Cloud API reports them as offline. You can build a connectivity health check by polling the nodes endpoint:

#!/bin/bash
# agent-connectivity-check.sh
# Alerts if any critical node has been offline for more than 5 minutes

NETDATA_TOKEN="${NETDATA_TOKEN}"
SPACE_ID="${NETDATA_SPACE_ID}"  # found in Space Settings → General
MAX_OFFLINE_SECONDS=300

RESP=$(curl -sf \
  -H "Authorization: Bearer ${NETDATA_TOKEN}" \
  "https://app.netdata.cloud/api/v1/spaces/${SPACE_ID}/nodes?limit=100")

if [ $? -ne 0 ]; then
  echo "FAIL: Netdata Cloud API unreachable"
  exit 1
fi

NOW=$(date +%s)

# Find nodes that are offline and have been for too long
OFFLINE=$(echo "$RESP" | jq -r --argjson now "$NOW" --argjson threshold "$MAX_OFFLINE_SECONDS" '
  .nodes[]
  | select(.availability.online == false)
  | select(($now - (.availability.lastSeen // $now)) > $threshold)
  | .hostname
')

if [ -n "$OFFLINE" ]; then
  echo "FAIL: nodes offline: $(echo "$OFFLINE" | tr '\n' ', ')"
  exit 1
fi

TOTAL=$(echo "$RESP" | jq '.nodes | length')
echo "OK: all $TOTAL nodes online"
exit 0

Expose this script as an HTTP endpoint (see Step 6 for the full sidecar) and add it to Vigilmon as an HTTP monitor with expected response body "OK:".

Step 3: Monitor streaming replication health

Netdata supports streaming replication where child agents send their metrics to a parent node. The parent then forwards the aggregated stream to Netdata Cloud. When the parent-child connection breaks, all children appear offline simultaneously.

Check streaming replication by querying each parent node's local API for its connected children:

// streaming-health-check.js
import http from "http";
import fetch from "node-fetch";

// Parent nodes are the Netdata agents that accept child streams
// Their local APIs are accessible within your network
const PARENT_NODES = (process.env.PARENT_NODES || "http://parent1:19999,http://parent2:19999").split(",");

async function checkParentNode(url) {
  const r = await fetch(`${url}/api/v1/streaming`, { timeout: 5000 });
  if (!r.ok) throw new Error(`${url} returned ${r.status}`);
  const data = await r.json();

  return {
    url,
    receivingClients: data.receivingClients ?? 0,
    sendingClients: data.sendingClients ?? 0,
    ok: true,
  };
}

http.createServer(async (req, res) => {
  if (req.url !== "/streaming-health") {
    res.writeHead(404); res.end(); return;
  }

  const results = await Promise.allSettled(PARENT_NODES.map(checkParentNode));
  const failed = results
    .filter((r) => r.status === "rejected")
    .map((r) => r.reason.message);

  const healthy = results
    .filter((r) => r.status === "fulfilled")
    .map((r) => r.value);

  if (failed.length > 0) {
    res.writeHead(503, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ status: "degraded", failed, healthy }));
  } else {
    res.writeHead(200, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ status: "ok", parents: healthy }));
  }
}).listen(9093, () => console.log("Streaming health check on :9093"));

Deploy this sidecar within your network where it can reach parent node APIs. Add a Vigilmon HTTP monitor for https://monitoring.your-domain.com/streaming-health.

Step 4: Monitor alert notification delivery

Netdata Cloud routes alerts through notification integrations you configure in Space Settings. If a Slack webhook expires or a PagerDuty token rotates, alerts are silently swallowed. The only way to catch this is to periodically fire a test alert and verify delivery.

Create a simple alert notification verification script:

#!/bin/bash
# alert-notification-test.sh
# Sends a test notification via Netdata Cloud API and verifies delivery

NETDATA_TOKEN="${NETDATA_TOKEN}"
SPACE_ID="${NETDATA_SPACE_ID}"
NOTIFICATION_INTEGRATION_ID="${NOTIFICATION_INTEGRATION_ID}"  # from Space Settings → Notifications

# Trigger a test notification
RESP=$(curl -sf -X POST \
  -H "Authorization: Bearer ${NETDATA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"type": "test"}' \
  "https://app.netdata.cloud/api/v1/spaces/${SPACE_ID}/notifications/${NOTIFICATION_INTEGRATION_ID}/test")

HTTP_CODE=$?

if [ $HTTP_CODE -ne 0 ]; then
  echo "FAIL: could not reach notification API"
  exit 1
fi

STATUS=$(echo "$RESP" | jq -r '.status // "unknown"')
if [ "$STATUS" = "success" ] || [ "$STATUS" = "sent" ]; then
  echo "OK: test notification sent successfully"
  exit 0
else
  echo "FAIL: test notification returned status: $STATUS"
  exit 1
fi

Schedule this as a daily cron job and expose its result via a status endpoint. Add a Vigilmon monitor for the status endpoint — if the notification test fails, you'll know before your next real alert is due.

Step 5: Monitor ML anomaly detection status

Netdata's Anomaly Advisor uses local ML models trained per metric per node. You can check the training and inference status via the local Netdata agent API:

#!/usr/bin/env python3
# ml-health-check.py — check ML model training status on a parent node

import http.server
import json
import urllib.request
import urllib.error

NETDATA_URL = "http://localhost:19999"  # local agent API

class MLHealthHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/ml-health":
            self.send_response(404)
            self.end_headers()
            return

        try:
            with urllib.request.urlopen(
                f"{NETDATA_URL}/api/v1/ml_info", timeout=5
            ) as r:
                data = json.loads(r.read())

            # Check if models are trained and inference is running
            models_trained = data.get("models_trained", 0)
            models_total = data.get("models_total", 1)
            anomaly_rate = data.get("current_anomaly_rate", 0)
            training_pct = (models_trained / models_total * 100) if models_total > 0 else 0

            if training_pct < 80:
                body = json.dumps({
                    "status": "degraded",
                    "modelsCoverage": f"{training_pct:.1f}%",
                    "trained": models_trained,
                    "total": models_total,
                })
                self.send_response(503)
            else:
                body = json.dumps({
                    "status": "ok",
                    "modelsCoverage": f"{training_pct:.1f}%",
                    "currentAnomalyRate": anomaly_rate,
                })
                self.send_response(200)

            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(body.encode())

        except Exception as e:
            self.send_response(503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({"status": "error", "error": str(e)}).encode())

    def log_message(self, *args):
        pass

if __name__ == "__main__":
    with http.server.HTTPServer(("", 9094), MLHealthHandler) as httpd:
        print("ML health check on :9094")
        httpd.serve_forever()

Deploy this on your Netdata parent node and expose it via an ingress. Add a Vigilmon monitor checking for "status":"ok" in the response.

Step 6: Configure Vigilmon alert channels

With all five monitors in place, configure where alerts go:

  1. In the Vigilmon dashboard, click Alert Channels.
  2. Add a Slack webhook for #infra-alerts.
  3. Add a PagerDuty integration for your on-call rotation if agent connectivity is P1.
  4. For each monitor, set Alert after N failures to 2 to filter transient blips.
  5. Enable Recovery notifications so you know when agents reconnect after a network partition.

Step 7: Parent-child node sync verification

Beyond individual monitors, verify parent-child sync health by comparing the node counts the parent reports locally versus what Netdata Cloud reports for the same Space:

#!/bin/bash
# parent-child-sync-check.sh

NETDATA_TOKEN="${NETDATA_TOKEN}"
SPACE_ID="${NETDATA_SPACE_ID}"
PARENT_URL="${PARENT_URL:-http://localhost:19999}"

# Count children the parent sees locally
LOCAL_CHILDREN=$(curl -sf "${PARENT_URL}/api/v1/nodes" | jq '[.nodes[] | select(.streaming == true)] | length')

# Count nodes Netdata Cloud shows as online in this space
CLOUD_ONLINE=$(curl -sf \
  -H "Authorization: Bearer ${NETDATA_TOKEN}" \
  "https://app.netdata.cloud/api/v1/spaces/${SPACE_ID}/nodes" \
  | jq '[.nodes[] | select(.availability.online == true)] | length')

if [ "$LOCAL_CHILDREN" -ne "$CLOUD_ONLINE" ]; then
  echo "MISMATCH: parent sees $LOCAL_CHILDREN children, Cloud shows $CLOUD_ONLINE online"
  exit 1
fi

echo "OK: $LOCAL_CHILDREN nodes in sync"
exit 0

Schedule this as a 5-minute cron job and expose the result as an HTTP endpoint. Add a final Vigilmon monitor for the sync check endpoint.

What you're monitoring now

| Monitor | What it catches | |---|---| | Netdata Cloud API endpoint | Cloud outage, token expiry, network access blocked | | Agent connectivity check | Individual agents offline >5 minutes | | Streaming replication health | Parent node disconnected, child nodes falling silent | | Alert notification delivery | Webhook expired, token rotated, integration broken | | ML anomaly detection status | Model training below coverage threshold | | Parent-child sync check | Local vs Cloud node count mismatch |

Operational tips for Netdata Cloud

  • Use space-level tokens, not user tokens — space tokens don't rotate when a user changes their password; they're explicitly managed and won't silently expire in the middle of the night.
  • Tag your critical nodes — Netdata Cloud allows node tags; filter your connectivity check to only alert on nodes tagged critical to avoid noise from short-lived or dev nodes.
  • Monitor your parent nodes' disk — Netdata's per-second metrics generate significant on-disk data on parent nodes. If a parent node's disk fills up, streaming stops immediately. Add a simple disk space check via Vigilmon's HTTP monitor on the parent's local API.
  • Keep Netdata agents pinned to a minor version — patch versions occasionally change the streaming protocol in ways that break parent-child compatibility. Pin agents to the same minor version across your fleet and update deliberately.

External monitoring with Vigilmon gives you the independent, outside-in view of your Netdata Cloud setup that no internal metric can provide — especially useful when Netdata itself is the thing that's malfunctioning.

Start monitoring your Netdata Cloud setup with Vigilmon →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles