tutorial

How to Monitor Coroot Node Agents, Service Maps, and SLOs with Vigilmon

Coroot's eBPF-based zero-instrumentation observability is powerful but needs external watchdogs. Learn how to monitor Coroot node agent health, service map accuracy, RED metrics per service, deployment tracking, SLO alerting, and integrate Vigilmon endpoint checks for production confidence.

Coroot is a zero-instrumentation observability platform that uses eBPF to automatically discover services, build accurate service maps, and compute RED metrics — all without a single line of application code. It correlates deployments with latency spikes, tracks error rates per service, and supports SLO-based alerting out of the box. What it doesn't do is monitor itself. If the Coroot node agent crashes on a host, that host disappears silently from your service map — and you won't notice until someone asks why their service data looks thin.

Vigilmon fills this gap with external HTTP and heartbeat monitors that watch Coroot's own health endpoints, validate data freshness, and alert before silent blind spots become production incidents.

Why Coroot Needs External Monitoring

Coroot's node agent (coroot-node-agent) runs as a DaemonSet and streams eBPF-collected metrics to the Coroot server. Several failure modes are invisible from within Coroot itself:

  • Node agent crash: The host stops reporting. Service map edges from that node disappear with no alert.
  • Coroot server overload: Ingestion slows, RED metrics become stale, SLO calculations lag behind reality.
  • ClickHouse backend failure: Coroot's storage backend degraded — queries succeed but return incomplete data windows.
  • Deployment tracking gaps: If the agent misses a rollout window, change correlation breaks and post-deploy regression analysis is unreliable.

External monitoring with Vigilmon adds a second opinion that catches these failures before users report degraded dashboards.

Step 1: Verify Coroot Node Agent Health

Coroot's node agent exposes a Prometheus metrics endpoint and an HTTP health check. Verify both are reachable:

# Check node agent health endpoint (default port 80 on agent pod)
curl -s http://NODE_AGENT_IP:80/health

# Check Prometheus metrics exposure
curl -s http://NODE_AGENT_IP:80/metrics | grep coroot_node_agent_up

For Kubernetes deployments, expose the health endpoint via a Service and Ingress so Vigilmon can reach it externally:

# coroot-agent-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: coroot-node-agent-health
  namespace: coroot
spec:
  selector:
    app: coroot-node-agent
  ports:
    - name: health
      port: 80
      targetPort: 80
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: coroot-agent-health-ingress
  namespace: coroot
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /health
spec:
  rules:
    - host: coroot-health.your-domain.com
      http:
        paths:
          - path: /agent-health
            pathType: Prefix
            backend:
              service:
                name: coroot-node-agent-health
                port:
                  number: 80

Apply and verify:

kubectl apply -f coroot-agent-health-svc.yaml
curl -s https://coroot-health.your-domain.com/agent-health
# Expected: HTTP 200

Step 2: Monitor Service Map Accuracy and RED Metrics

Coroot computes RED (Rate, Errors, Duration) metrics per service from eBPF data. Build a health check that validates the Coroot API returns fresh service data:

# coroot_health.py
from flask import Flask, jsonify
import requests
import os
from datetime import datetime, timezone

app = Flask(__name__)

COROOT_URL = os.environ.get('COROOT_URL', 'http://coroot:8080')
COROOT_TOKEN = os.environ.get('COROOT_API_TOKEN', '')
STALE_THRESHOLD_MINUTES = int(os.environ.get('STALE_THRESHOLD_MINUTES', '5'))

@app.route('/health/coroot/services')
def coroot_services_health():
    headers = {'Authorization': f'Bearer {COROOT_TOKEN}'} if COROOT_TOKEN else {}
    try:
        resp = requests.get(
            f'{COROOT_URL}/api/projects',
            headers=headers,
            timeout=10,
        )
        resp.raise_for_status()
        projects = resp.json()
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

    if not projects:
        return jsonify(status='degraded', reason='no_projects_found'), 503

    # Validate at least one project has active services
    for project in projects:
        project_id = project.get('id') or project.get('name')
        try:
            svc_resp = requests.get(
                f'{COROOT_URL}/api/projects/{project_id}/overview',
                headers=headers,
                timeout=10,
            )
            if svc_resp.status_code == 200:
                overview = svc_resp.json()
                applications = overview.get('applications', [])
                if applications:
                    return jsonify(
                        status='ok',
                        project=project_id,
                        service_count=len(applications),
                    ), 200
        except Exception:
            continue

    return jsonify(status='degraded', reason='no_active_services_found'), 503


@app.route('/health/coroot/server')
def coroot_server_health():
    try:
        resp = requests.get(f'{COROOT_URL}/health', timeout=5)
        if resp.status_code == 200:
            return jsonify(status='ok'), 200
        return jsonify(status='degraded', code=resp.status_code), 503
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=int(os.environ.get('PORT', '5010')))

Run the sidecar:

pip install flask requests
export COROOT_URL=http://coroot.coroot.svc.cluster.local:8080
python coroot_health.py &

Step 3: Track Deployment Correlation Health

Coroot's deployment tracking relies on continuous agent data ingestion. If agent coverage drops below 100% of your nodes, deployment change correlation becomes unreliable. Add a DaemonSet readiness check:

#!/bin/bash
# check_coroot_agent_coverage.sh
DESIRED=$(kubectl get daemonset coroot-node-agent -n coroot -o jsonpath='{.status.desiredNumberScheduled}')
READY=$(kubectl get daemonset coroot-node-agent -n coroot -o jsonpath='{.status.numberReady}')

if [ "$READY" -lt "$DESIRED" ]; then
  echo "{\"status\":\"degraded\",\"desired\":$DESIRED,\"ready\":$READY}"
  exit 1
fi

echo "{\"status\":\"ok\",\"desired\":$DESIRED,\"ready\":$READY}"
exit 0

Expose this as an HTTP endpoint using a lightweight wrapper:

import subprocess
import json
from flask import Flask, Response

app = Flask(__name__)

@app.route('/health/coroot/agent-coverage')
def agent_coverage():
    result = subprocess.run(
        ['bash', '/scripts/check_coroot_agent_coverage.sh'],
        capture_output=True, text=True, timeout=10
    )
    body = result.stdout.strip() or '{"status":"unknown"}'
    code = 200 if result.returncode == 0 else 503
    return Response(body, status=code, mimetype='application/json')

Step 4: Configure SLO Alerting Monitoring

Coroot's SLO engine evaluates error budgets based on eBPF-collected data. Monitor SLO health by checking the Coroot UI's SLO summary API:

@app.route('/health/coroot/slos')
def coroot_slos_health():
    headers = {'Authorization': f'Bearer {os.environ.get("COROOT_API_TOKEN", "")}'} if os.environ.get("COROOT_API_TOKEN") else {}
    try:
        resp = requests.get(
            f'{COROOT_URL}/api/projects/default/slos',
            headers=headers,
            timeout=10,
        )
        if resp.status_code == 404:
            # No SLOs configured — not an error
            return jsonify(status='ok', slo_count=0, note='no_slos_configured'), 200
        resp.raise_for_status()
        slos = resp.json()
        breached = [s for s in slos if s.get('status') == 'breached']
        if breached:
            return jsonify(
                status='degraded',
                reason='slo_breach_detected',
                breached_count=len(breached),
                breached=[s.get('name') for s in breached],
            ), 503
        return jsonify(status='ok', slo_count=len(slos)), 200
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

Step 5: Configure Vigilmon HTTP Monitors for Coroot

Monitor 1: Coroot Server Health

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set URL to https://coroot.your-domain.com/health
  4. Interval: 1 minute
  5. Expected: Status code 200
  6. Response timeout: 5000ms
  7. Alert: Slack + PagerDuty (P1 — Coroot down means no observability)
  8. Save

Monitor 2: Node Agent Coverage

  • URL: https://coroot-health.your-domain.com/health/coroot/agent-coverage
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert: Slack + PagerDuty (agents down means blind spots in service map)

Monitor 3: Service Map Validity

  • URL: https://coroot-health.your-domain.com/health/coroot/services
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert: Slack (P2 — stale data, not process crash)

Monitor 4: SLO Breach Detection

  • URL: https://coroot-health.your-domain.com/health/coroot/slos
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert: Slack + PagerDuty (active SLO breach warrants immediate response)

Step 6: Heartbeat Monitor for Coroot Data Ingestion

Validate that the Coroot server is actively receiving eBPF data from agents by pinging Vigilmon when the metrics pipeline is live:

#!/bin/bash
# coroot_heartbeat.sh — run via cron every 3 minutes on a node with the agent running
METRICS_COUNT=$(curl -sf http://localhost:80/metrics 2>/dev/null | grep -c "coroot_")
if [ "$METRICS_COUNT" -gt 10 ]; then
  curl -sf "$VIGILMON_HEARTBEAT_URL" > /dev/null
fi

Add to cron on each monitored node:

*/3 * * * * /opt/coroot/coroot_heartbeat.sh

In Vigilmon, create a Heartbeat monitor:

  1. Monitors → New Monitor → Heartbeat
  2. Name: coroot-data-ingestion
  3. Expected interval: 5 minutes, Grace: 10 minutes
  4. Save and set VIGILMON_HEARTBEAT_URL in your environment

Alert Routing Summary

| Monitor | Signal | Priority | |---|---|---| | Coroot server /health | Coroot process liveness | P1 | | Node agent coverage | Agent DaemonSet readiness | P1 | | Service map validity | Active service data present | P2 | | SLO breach detection | Active error budget burn | P1 | | Data ingestion heartbeat | eBPF metric pipeline alive | P2 |

Summary

Coroot's zero-instrumentation design is a strength in production — but it creates a blind spot when the instrumentation layer itself fails. External monitoring with Vigilmon closes this loop:

| What Vigilmon Monitors | Why It Matters | |---|---| | Coroot server health | Catches full observability outages immediately | | Node agent DaemonSet | Detects agent crashes before service maps go stale | | Service data freshness | Confirms eBPF data is flowing to the service map | | SLO breach state | Adds an external alert path for error budget burns | | Ingestion heartbeat | Validates the full agent → server pipeline is live |

Get started free at vigilmon.online — your first Coroot monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles