Monitoring Tetragon eBPF Security Observability with Vigilmon

Tetragon is Cilium's eBPF-based security observability and runtime enforcement engine. It gives you kernel-level visibility into process execution, file access, network connections, and privilege escalation—in real time, with near-zero overhead. But Tetragon fails in silent ways: the daemon crashes, eBPF programs detach, TracingPolicy rules stop matching, or the export pipeline backs up. When that happens, your security observability disappears without any alert.

This tutorial shows how to monitor Tetragon's health, process event throughput, enforcement policy status, and Prometheus metrics pipeline using Vigilmon—so you know immediately when your eBPF security layer stops working.

Why Tetragon needs external monitoring

Tetragon and eBPF can fail in ways that aren't visible from the Kubernetes control plane:

• eBPF programs detach silently — a kernel update or kernel version mismatch causes Tetragon's BPF programs to fail to load; the DaemonSet pod stays Running but generates zero events
• TracingPolicy stops matching — a malformed or overly restrictive policy silences an entire event class without any error visible in kubectl get tp
• Export pipeline backs up — gRPC export to an aggregator (Hubble, Tetragon aggregator, SIEM) stalls; events queue in memory and eventually drop
• Process event throughput drops to zero — during a high-noise security incident, Tetragon ringbuffer fills and drops events; you lose visibility exactly when you need it most
• DaemonSet pod crashes on a specific node — one node runs without security observability while the rest look healthy

Kubernetes liveness probes and kubectl describe don't catch any of these. You need external observation of Tetragon's actual event output, not just its pod status.

What you'll need

• A Kubernetes cluster with Tetragon installed (Helm or kubectl)
• Tetragon's Prometheus metrics endpoint exposed
• A free Vigilmon account

Step 1: Verify Tetragon's Prometheus metrics endpoint

Tetragon exposes Prometheus metrics on port 2112 by default. Confirm it's reachable:

# Port-forward to a Tetragon pod to verify
kubectl port-forward -n kube-system ds/tetragon 2112:2112

curl http://localhost:2112/metrics | grep tetragon

You should see metrics like:

# HELP tetragon_events_total Number of Tetragon events
# TYPE tetragon_events_total counter
tetragon_events_total{type="PROCESS_EXEC"} 4821
tetragon_events_total{type="PROCESS_EXIT"} 4810
tetragon_events_total{type="PROCESS_KPROBE"} 1204
tetragon_events_total{type="PROCESS_TRACEPOINT"} 893

Expose the metrics service so your health bridge can reach it:

# tetragon-metrics-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: tetragon-metrics
  namespace: kube-system
spec:
  selector:
    app.kubernetes.io/name: tetragon
  ports:
    - name: metrics
      port: 2112
      targetPort: 2112

Step 2: Build a Tetragon health bridge

Create a lightweight service that queries Tetragon metrics and exposes a simple HTTP health endpoint for Vigilmon to probe:

# tetragon-health-bridge.py
import os, time, requests
from flask import Flask, jsonify

app = Flask(__name__)
TETRAGON_METRICS = os.environ.get("TETRAGON_METRICS_URL", "http://tetragon-metrics.kube-system:2112/metrics")
MIN_EVENTS_PER_MINUTE = int(os.environ.get("MIN_EVENTS_PER_MINUTE", "5"))

_last_counts = {}
_last_check = 0

def parse_prometheus(text):
    metrics = {}
    for line in text.splitlines():
        if line.startswith("#"):
            continue
        if "{" in line:
            name_part, value_part = line.rsplit(" ", 1)
            metrics[name_part.strip()] = float(value_part.strip())
    return metrics

def get_event_rate():
    global _last_counts, _last_check
    resp = requests.get(TETRAGON_METRICS, timeout=5)
    resp.raise_for_status()
    current = parse_prometheus(resp.text)
    now = time.time()

    exec_key = 'tetragon_events_total{type="PROCESS_EXEC"}'
    current_exec = current.get(exec_key, 0)

    if _last_check == 0 or _last_check not in _last_counts:
        _last_counts = {exec_key: current_exec}
        _last_check = now
        return None

    elapsed = now - _last_check
    delta = current_exec - _last_counts.get(exec_key, 0)
    rate_per_min = (delta / elapsed) * 60

    _last_counts = {exec_key: current_exec}
    _last_check = now
    return rate_per_min

@app.route("/health")
def health():
    try:
        rate = get_event_rate()
        if rate is None:
            return jsonify({"status": "ok", "message": "Initializing baseline"}), 200
        if rate < MIN_EVENTS_PER_MINUTE:
            return jsonify({
                "status": "error",
                "message": f"Event rate too low: {rate:.1f}/min (minimum {MIN_EVENTS_PER_MINUTE})"
            }), 503
        return jsonify({"status": "ok", "exec_events_per_min": round(rate, 1)}), 200
    except Exception as e:
        return jsonify({"status": "error", "message": str(e)}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Deploy it:

# tetragon-bridge-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tetragon-health-bridge
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tetragon-health-bridge
  template:
    metadata:
      labels:
        app: tetragon-health-bridge
    spec:
      containers:
        - name: bridge
          image: python:3.12-slim
          command: ["python", "/app/tetragon-health-bridge.py"]
          env:
            - name: MIN_EVENTS_PER_MINUTE
              value: "5"
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tetragon-health-bridge
  namespace: monitoring
spec:
  selector:
    app: tetragon-health-bridge
  ports:
    - port: 80
      targetPort: 8080

Step 3: Monitor TracingPolicy enforcement status

Tetragon's TracingPolicy resources control what eBPF programs are active. Monitor their health:

# Check TracingPolicy status
kubectl get tracingpolicy -A

# A healthy policy looks like:
# NAME                      AGE   PODS    ERRORS   INFO
# sys-calls                 2d    3       0        Enabled on 3/3 nodes
# network-observe           2d    3       0        Enabled on 3/3 nodes

Add a TracingPolicy check to the health bridge:

import subprocess, json

def check_tracing_policies():
    result = subprocess.run(
        ["kubectl", "get", "tracingpolicy", "-A", "-o", "json"],
        capture_output=True, text=True, timeout=10
    )
    if result.returncode != 0:
        return False, "kubectl failed"

    policies = json.loads(result.stdout)
    errors = []
    for item in policies.get("items", []):
        name = item["metadata"]["name"]
        status = item.get("status", {})
        if status.get("errors", 0) > 0:
            errors.append(f"{name}: {status['errors']} errors")

    if errors:
        return False, "; ".join(errors)
    return True, f"{len(policies['items'])} policies healthy"

Step 4: Set up Vigilmon monitors

Monitor 1 — Tetragon metrics endpoint liveness:

1. Go to vigilmon.online → Add Monitor
2. Type: HTTP
3. URL: http://tetragon-metrics.kube-system.svc:2112/metrics (or your ingress URL)
4. Keyword check: tetragon_events_total
5. Interval: 1 minute

Monitor 2 — Event throughput health bridge:

1. Add Monitor → HTTP
2. URL: http://tetragon-health-bridge.monitoring.svc/health
3. Expected status: 200
4. Interval: 2 minutes

Monitor 3 — Tetragon gRPC export endpoint (if using Hubble or aggregator):

1. Add Monitor → TCP
2. Host: tetragon-aggregator.monitoring.svc
3. Port: 4317 (or your configured gRPC port)
4. Interval: 1 minute

Step 5: Configure Prometheus alerting rules

Add AlertManager rules for critical Tetragon conditions:

# tetragon-alerts.yaml
groups:
  - name: tetragon
    rules:
      - alert: TetragonEventRateLow
        expr: rate(tetragon_events_total{type="PROCESS_EXEC"}[5m]) < 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tetragon PROCESS_EXEC event rate dropped"
          description: "EXEC events below 6/min for 5m — eBPF programs may have detached"

      - alert: TetragonDroppedEvents
        expr: rate(tetragon_map_drops_total[5m]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Tetragon ringbuffer dropping events"
          description: "Events are being dropped — increase ringbuffer size or reduce noise"

      - alert: TetragonExportErrors
        expr: rate(tetragon_export_errors_total[5m]) > 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Tetragon export pipeline errors"
          description: "gRPC export errors detected — SIEM may not be receiving events"

Apply:

kubectl apply -f tetragon-alerts.yaml -n monitoring

Step 6: Set up alert notifications in Vigilmon

1. In Vigilmon, go to Alerts → Notification Channels
2. Add a Slack or email channel for your security team
3. On each Tetragon monitor, set:
   • Notify after: 1 failure
   • Escalation: immediate (security tools must not fail silently)

What you're now monitoring

With this setup you have visibility into:

• eBPF program liveness — if PROCESS_EXEC events stop, Vigilmon fires within 2 minutes
• Ringbuffer health — dropped events trigger alerts before you lose forensic coverage
• TracingPolicy enforcement — policy errors surface before security gaps open
• Export pipeline integrity — gRPC or file export failures are caught at the source

Tetragon's eBPF approach gives you unprecedentedly deep kernel-level visibility—but only when it's actually running. Vigilmon ensures you're always the first to know when that visibility goes dark.