tutorial

How to Monitor Groundcover with Vigilmon (eBPF Sensor Health, SLO Tracking & APM Pipeline)

Groundcover is a cloud-native APM platform built on eBPF. Instead of instrumenting your application code or deploying sidecar proxies, it installs a single D...

Groundcover is a cloud-native APM platform built on eBPF. Instead of instrumenting your application code or deploying sidecar proxies, it installs a single DaemonSet of kernel-level sensors that automatically capture traces, metrics, and logs from every workload on your Kubernetes nodes — with zero code changes.

Zero instrumentation is a huge operational win, but it also introduces a unique monitoring challenge: when Groundcover's sensors stop working, you lose observability silently. There's no instrumentation agent inside your app to fail loudly; the sensor just stops capturing data at the kernel level. In this tutorial you'll use Vigilmon to monitor Groundcover's own health — sensor agent status, service topology accuracy, SLO pipeline health, and anomaly detection — so your observability platform doesn't become a blind spot. Free tier, no credit card required.

Why Groundcover needs external monitoring

Groundcover's eBPF architecture means that failures at the sensor layer have outsized impact:

  • Sensor DaemonSet pod crashes — a kernel version incompatibility or cgroup v2 misconfiguration causes sensor pods to crash-loop; Groundcover's UI still loads but shows zero new data, making it look like your services stopped receiving traffic
  • Service topology goes stale — after a Kubernetes node replacement or namespace rename, the topology map stops updating but doesn't show an error; engineers make routing decisions based on a map that's days out of date
  • SLO tracking pauses silently — if the SLO computation pipeline falls behind its event stream, your SLO charts flatline rather than showing an error; you only discover the issue when a compliance report is due
  • Anomaly detection stops firing — the ML pipeline that drives Groundcover's anomaly alerts can fall behind under high cardinality load; alerts stop arriving and teams assume everything is fine

External monitoring with Vigilmon adds the independent layer you need: reachability checks, health endpoint polling, and alerting that works even when Groundcover itself is the thing that's down.

What you'll need

  • A Groundcover installation on a Kubernetes cluster (the inCloud or self-hosted edition)
  • The Groundcover DaemonSet deployed on your nodes
  • A free Vigilmon account
  • kubectl access to check DaemonSet status

Step 1: Verify sensor agent health

Groundcover deploys its eBPF sensors as a Kubernetes DaemonSet named groundcover-sensor (or similar, depending on your install). Check that all pods are running:

kubectl get daemonset groundcover-sensor -n groundcover \
  -o jsonpath='{.status.numberReady}/{.status.desiredNumberScheduled}'

A healthy output is 5/5 (or whatever matches your node count). A mismatch like 3/5 means two nodes aren't being monitored.

To expose this as an HTTP endpoint that Vigilmon can probe, deploy a small status exporter:

# sensor-health-exporter.py
import subprocess
import json
import http.server
import socketserver

PORT = 8765

class SensorHealthHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/sensor-health":
            self.send_response(404)
            self.end_headers()
            return

        try:
            result = subprocess.run(
                [
                    "kubectl", "get", "daemonset", "groundcover-sensor",
                    "-n", "groundcover",
                    "-o", "jsonpath={.status.numberReady}/{.status.desiredNumberScheduled}"
                ],
                capture_output=True, text=True, timeout=10
            )
            parts = result.stdout.strip().split("/")
            ready = int(parts[0])
            desired = int(parts[1])

            if ready == desired and desired > 0:
                body = json.dumps({"status": "ok", "ready": ready, "desired": desired})
                self.send_response(200)
            else:
                body = json.dumps({"status": "degraded", "ready": ready, "desired": desired})
                self.send_response(503)

            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(body.encode())
        except Exception as e:
            self.send_response(503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({"status": "error", "error": str(e)}).encode())

    def log_message(self, format, *args):
        pass  # suppress access log noise

with socketserver.TCPServer(("", PORT), SensorHealthHandler) as httpd:
    print(f"Sensor health exporter on :{PORT}")
    httpd.serve_forever()

Deploy this as a Kubernetes Deployment with a ServiceAccount that has get permissions on DaemonSets in the groundcover namespace:

# sensor-health-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sensor-health-exporter
  namespace: groundcover
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sensor-health-exporter
  template:
    metadata:
      labels:
        app: sensor-health-exporter
    spec:
      serviceAccountName: sensor-health-exporter-sa
      containers:
        - name: exporter
          image: python:3.12-slim
          command: ["python", "/app/sensor-health-exporter.py"]
          volumeMounts:
            - name: script
              mountPath: /app
          ports:
            - containerPort: 8765
      volumes:
        - name: script
          configMap:
            name: sensor-health-script
---
apiVersion: v1
kind: Service
metadata:
  name: sensor-health-exporter
  namespace: groundcover
spec:
  selector:
    app: sensor-health-exporter
  ports:
    - port: 8765
      targetPort: 8765
  type: ClusterIP

Expose this service via your ingress and add it to Vigilmon as an HTTP monitor checking for "status":"ok".

Step 2: Monitor the Groundcover API / UI endpoint

If you're running the Groundcover inCloud edition, the UI and API are hosted by Groundcover. If you're self-hosting, your Groundcover backend exposes a UI and API over HTTP(S). Add that endpoint to Vigilmon:

  1. In the Vigilmon dashboard, click Add Monitor.
  2. Choose HTTP(S) as the monitor type.
  3. Enter your Groundcover URL: https://groundcover.your-domain.com (or the inCloud URL you were given).
  4. Set the check interval to 60 seconds.
  5. Under Expected Status Code, enter 200.
  6. Click Save.

This checks end-to-end reachability: DNS resolution, TLS termination, load balancer health, and the backend returning a valid response. If any layer breaks, Vigilmon alerts you within two minutes.

Step 3: Monitor service topology accuracy

Groundcover builds its service map by analyzing network traffic captured by the eBPF sensors. When the topology stops updating, it usually means the sensor is no longer sending data to the backend. You can detect topology staleness by polling the service count endpoint:

#!/bin/bash
# topology-freshness-check.sh
# Checks that the service topology has been updated within the last 5 minutes

GROUNDCOVER_API="${GROUNDCOVER_API:-https://groundcover.your-domain.com}"
GROUNDCOVER_TOKEN="${GROUNDCOVER_TOKEN}"

RESP=$(curl -sf \
  -H "Authorization: Bearer ${GROUNDCOVER_TOKEN}" \
  "${GROUNDCOVER_API}/api/v1/topology/services")

if [ $? -ne 0 ]; then
  echo "FAIL: could not reach topology API"
  exit 1
fi

SERVICE_COUNT=$(echo "$RESP" | jq '.services | length')
LAST_UPDATE=$(echo "$RESP" | jq -r '.meta.lastUpdated // empty')

if [ -z "$LAST_UPDATE" ] || [ "$SERVICE_COUNT" -eq 0 ]; then
  echo "FAIL: topology is empty or has no lastUpdated timestamp"
  exit 1
fi

# Check that last update is within 5 minutes
LAST_TS=$(date -d "$LAST_UPDATE" +%s 2>/dev/null || date -j -f "%Y-%m-%dT%H:%M:%SZ" "$LAST_UPDATE" +%s)
NOW=$(date +%s)
AGE=$((NOW - LAST_TS))

if [ "$AGE" -gt 300 ]; then
  echo "STALE: topology last updated ${AGE}s ago (threshold 300s)"
  exit 1
fi

echo "OK: topology has $SERVICE_COUNT services, updated ${AGE}s ago"
exit 0

Wrap this in an HTTP endpoint and add a Vigilmon monitor for it. If the topology hasn't refreshed in 5 minutes, your sensors may have stopped sending data.

Step 4: Monitor SLO tracking pipeline health

Groundcover's SLO engine computes error budgets in near real-time. If the SLO pipeline falls behind, your error budget charts will show stale data. Check SLO freshness by querying the SLO status endpoint:

// slo-health-check.js
import http from "http";
import fetch from "node-fetch";

const GROUNDCOVER_API = process.env.GROUNDCOVER_API || "https://groundcover.your-domain.com";
const TOKEN = process.env.GROUNDCOVER_TOKEN;

http.createServer(async (req, res) => {
  if (req.url !== "/slo-health") {
    res.writeHead(404); res.end(); return;
  }

  try {
    const r = await fetch(`${GROUNDCOVER_API}/api/v1/slos`, {
      headers: { Authorization: `Bearer ${TOKEN}` },
    });

    if (!r.ok) throw new Error(`SLO API returned ${r.status}`);
    const data = await r.json();

    const stale = (data.slos || []).filter((slo) => {
      if (!slo.lastEvaluated) return true;
      const ageMs = Date.now() - new Date(slo.lastEvaluated).getTime();
      return ageMs > 10 * 60 * 1000; // 10 minutes
    });

    if (stale.length > 0) {
      res.writeHead(503, { "Content-Type": "application/json" });
      res.end(JSON.stringify({
        status: "degraded",
        staleSLOs: stale.map((s) => s.name),
      }));
    } else {
      res.writeHead(200, { "Content-Type": "application/json" });
      res.end(JSON.stringify({ status: "ok", sloCount: data.slos?.length ?? 0 }));
    }
  } catch (err) {
    res.writeHead(503, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ status: "error", error: err.message }));
  }
}).listen(9092);

Add this as a Vigilmon HTTP monitor with the expected response body containing "status":"ok".

Step 5: Monitor anomaly detection pipeline health

Groundcover's ML-based anomaly detection can silently fall behind under high cardinality load. Check whether anomaly alerts are being generated at the expected cadence:

#!/bin/bash
# anomaly-pipeline-check.sh
# Checks that at least one anomaly has been evaluated in the past 15 minutes
# (assumes at least one service in the cluster will show some anomaly signals)

GROUNDCOVER_API="${GROUNDCOVER_API:-https://groundcover.your-domain.com}"
TOKEN="${GROUNDCOVER_TOKEN}"

RESP=$(curl -sf \
  -H "Authorization: Bearer ${TOKEN}" \
  "${GROUNDCOVER_API}/api/v1/anomalies?limit=1&since=15m")

if [ $? -ne 0 ]; then
  echo "FAIL: anomaly API unreachable"
  exit 1
fi

COUNT=$(echo "$RESP" | jq '.anomalies | length')
PIPELINE_STATUS=$(echo "$RESP" | jq -r '.pipelineStatus // "unknown"')

if [ "$PIPELINE_STATUS" = "delayed" ] || [ "$PIPELINE_STATUS" = "stopped" ]; then
  echo "FAIL: anomaly pipeline is $PIPELINE_STATUS"
  exit 1
fi

echo "OK: anomaly pipeline active, $COUNT recent anomalies"
exit 0

Adapt this to match the actual Groundcover API shape for your version. The key signal to check is whether the pipeline status field indicates the engine is processing events — not whether anomalies exist (a healthy system with no anomalies is fine).

Step 6: Configure Vigilmon alerts

  1. In the Vigilmon dashboard, click Alert Channels.
  2. Add a Slack webhook for your #platform-observability channel.
  3. Add a PagerDuty integration if sensor health is P1 for your team.
  4. For each Groundcover monitor, set Alert after N failures to 2.
  5. Set Recovery notification to ON — you want to know when sensors come back just as quickly as when they go down.

What you're monitoring now

| Monitor | What it catches | |---|---| | Sensor DaemonSet health endpoint | Node sensors crash-looping, kernel incompatibility | | Groundcover UI/API endpoint | Backend down, ingress misconfiguration, TLS expiry | | Topology freshness endpoint | Sensors stopped sending data, backend pipeline stalled | | SLO pipeline health endpoint | SLO engine behind, error budget charts stale | | Anomaly detection pipeline | ML engine delayed, high-cardinality overload |

Operational tips for Groundcover

  • Keep kernel versions pinned on nodes where Groundcover sensors run — eBPF programs are kernel-version sensitive, and a node pool upgrade can silently break sensors without the DaemonSet entering CrashLoopBackoff.
  • Watch for cgroup v2 migrations — if your cluster moves to cgroup v2, Groundcover sensors may need a configuration update. The sensor health exporter above catches this early.
  • Set a DaemonSet coverage alert — if your node count changes (autoscaling) and the DaemonSet desired count doesn't keep up, you'll have unmonitored nodes. The ready/desired check in Step 1 covers this.

External monitoring with Vigilmon gives you the independent layer that Groundcover itself cannot provide: confirmation that the platform you rely on for observability is actually working.

Start monitoring your Groundcover environment with Vigilmon →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles