How to Monitor bpftrace with Vigilmon

bpftrace is a high-level tracing language built on top of Linux eBPF that lets you write one-liner and script-based probes to instrument kernel functions, user-space applications, and hardware performance counters in real time. In Kubernetes environments, bpftrace is commonly deployed as a DaemonSet running alongside production workloads to capture system call latency, page fault storms, network packet drops, and OOM events across every node.

But bpftrace infrastructure is itself unmonitored by default. If the tracer DaemonSet crashes on a subset of nodes, you silently lose observability coverage for those nodes — and you won't know until you look. In this tutorial you'll set up uptime monitoring for your bpftrace observability infrastructure using Vigilmon — free tier, no credit card required.

Why bpftrace deployments need external monitoring

Internal Kubernetes probes on a bpftrace DaemonSet confirm that the container is running, but miss a wider class of operational failures:

• Kernel version mismatch — bpftrace probes fail silently when the running kernel lacks the required BTF or eBPF features; the container stays up while all probes return empty output
• Probe attachment failures — kprobes or uprobes fail to attach when the target function has been inlined by the compiler; no error is surfaced to the DaemonSet's readiness probe
• Missing kernel headers or BTF — nodes without /sys/kernel/btf/vmlinux prevent CO-RE compilation; bpftrace starts but every script exits immediately with an error
• DaemonSet not scheduled on all nodes — taints, resource pressure, or node selector drift cause gaps in eBPF coverage; pod-level health probes cannot detect missing coverage on unscheduled nodes
• Metric exporter HTTP server crash — when bpftrace output is piped to a Prometheus exporter sidecar, a sidecar crash means metrics disappear without affecting the main container's liveness

External monitoring from Vigilmon watches the HTTP endpoints your bpftrace stack exposes and alerts you when coverage gaps appear.

What you'll need

• A Kubernetes cluster with bpftrace running as a DaemonSet (or a standalone bpftrace host)
• An HTTP health or metrics endpoint exposed by your bpftrace deployment (see Step 1)
• A free Vigilmon account

Step 1: Expose a health endpoint for your bpftrace DaemonSet

bpftrace itself is a CLI tool — it does not expose HTTP endpoints natively. The standard production pattern wraps bpftrace in a sidecar or init container that continuously runs a probe script and exposes the result via a small HTTP server.

Here is a minimal DaemonSet that runs a bpftrace script and exposes a health endpoint:

# bpftrace-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: bpftrace-monitor
  namespace: observability
spec:
  selector:
    matchLabels:
      app: bpftrace-monitor
  template:
    metadata:
      labels:
        app: bpftrace-monitor
    spec:
      hostPID: true
      hostNetwork: true
      tolerations:
        - effect: NoSchedule
          operator: Exists
      containers:
        - name: bpftrace
          image: quay.io/iovisor/bpftrace:latest
          securityContext:
            privileged: true
          command:
            - /bin/sh
            - -c
            - |
              # Continuously run a minimal probe to verify eBPF is functional
              while true; do
                if bpftrace -e 'BEGIN { printf("ok\n"); exit(); }' > /tmp/health 2>&1; then
                  echo "healthy" > /tmp/status
                else
                  echo "unhealthy" > /tmp/status
                fi
                sleep 30
              done
          volumeMounts:
            - name: status
              mountPath: /tmp
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if [ "$STATUS" = "healthy" ]; then
                  printf "HTTP/1.1 200 OK\r\nContent-Length: 7\r\n\r\nhealthy" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Length: 9\r\n\r\nunhealthy" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              hostPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}

kubectl apply -f bpftrace-daemonset.yaml

# Verify the health endpoint on a node
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:8080/
# healthy

For clusters using a dedicated metrics exporter, expose the Prometheus metrics port via a NodePort or Ingress instead:

apiVersion: v1
kind: Service
metadata:
  name: bpftrace-metrics
  namespace: observability
spec:
  type: NodePort
  selector:
    app: bpftrace-monitor
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30080

Step 2: Verify bpftrace eBPF capability on each node

Before monitoring, confirm that bpftrace can attach probes on all nodes in your cluster. This script checks every node and logs any that are missing eBPF support:

#!/bin/bash
# check-bpftrace-nodes.sh
for NODE in $(kubectl get nodes -o name | cut -d/ -f2); do
  POD=$(kubectl get pod -n observability -o wide | grep bpftrace | grep "$NODE" | awk '{print $1}')
  if [ -z "$POD" ]; then
    echo "MISSING: No bpftrace pod on $NODE"
    continue
  fi
  RESULT=$(kubectl exec -n observability "$POD" -c bpftrace -- \
    bpftrace -e 'BEGIN { printf("ok\n"); exit(); }' 2>&1)
  if echo "$RESULT" | grep -q "ok"; then
    echo "OK: $NODE"
  else
    echo "FAIL: $NODE — $RESULT"
  fi
done

chmod +x check-bpftrace-nodes.sh
./check-bpftrace-nodes.sh

This identifies nodes with kernel header mismatches, missing BTF data, or eBPF permission issues before you start relying on the data.

Step 3: Set up HTTP monitoring in Vigilmon

With your health endpoint exposed, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | bpftrace DaemonSet health | http://your-node-ip:30080/ | 200 | | bpftrace metrics scrape | http://your-node-ip:30080/metrics | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match healthy in the response body
6. Save each monitor

If you route through an Ingress, use the Ingress host URL instead of the NodePort directly. Vigilmon probes from multiple geographic regions, so an Ingress with external DNS is the most reliable target.

Step 4: Monitor the bpftrace TCP port

Add a TCP monitor to catch lower-level failures where the health server port is unreachable even when the pod is Running:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your node IP or Ingress hostname and port 8080 (or your NodePort)
4. Save the monitor

A TCP failure here indicates a deeper networking issue — iptables rules, network policy blocking, or a host-port conflict — that would prevent any external observability data from being collected.

Step 5: Configure alert channels

When bpftrace coverage drops on a node, you lose kernel-level visibility for every workload on that node. Catch this before it silently affects incident investigation.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your platform or SRE on-call address
3. Assign the channel to all bpftrace monitors

Webhook alerts for incident routing

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. The payload Vigilmon sends:

{
  "monitor_name": "bpftrace DaemonSet health",
  "status": "down",
  "url": "http://your-node-ip:30080/",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Wire this alert into a runbook that checks which nodes have lost eBPF coverage and whether any active incident investigations are relying on bpftrace data from those nodes.

Step 6: Correlate Vigilmon alerts with bpftrace diagnostics

When you receive a bpftrace downtime alert, run these checks:

# 1. Check DaemonSet pod status across all nodes
kubectl get pods -n observability -l app=bpftrace-monitor -o wide

# 2. Check for pods not scheduled on expected nodes
kubectl describe ds bpftrace-monitor -n observability | grep -A5 "Scheduled\|Desired\|Ready"

# 3. Check bpftrace container logs for eBPF errors
kubectl logs -n observability -l app=bpftrace-monitor -c bpftrace --tail=50 | grep -i "error\|fail\|BTF\|not found"

# 4. Verify kernel BTF is available on the affected node
kubectl debug node/<node-name> -it --image=ubuntu -- ls /sys/kernel/btf/vmlinux

# 5. Check eBPF feature availability
kubectl exec -n observability <pod-name> -c bpftrace -- \
  bpftrace --info 2>&1 | grep -E "BTF|CO-RE|kernel"

# 6. Verify host PID and privileged mode are granted
kubectl get pod <pod-name> -n observability -o jsonpath='{.spec.hostPID}{" "}{.spec.containers[0].securityContext.privileged}'

If Vigilmon shows the health endpoint returning 503, the bpftrace container is running but eBPF probe attachment is failing — check for a kernel version change after a node upgrade.

Step 7: Create a status page for eBPF observability infrastructure

Your bpftrace DaemonSet is the foundation of kernel-level observability. Make its status visible to the broader engineering team without requiring cluster access:

1. Go to Status Pages → New Status Page
2. Name it: "eBPF Observability Infrastructure"
3. Add your monitors: bpftrace DaemonSet health, bpftrace metrics scrape
4. Share the URL with SRE and platform engineering teams

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on health endpoint | DaemonSet crash, eBPF probe attachment failures | | HTTP monitor on metrics endpoint | Prometheus exporter sidecar failures | | TCP monitor on health port | Networking issues blocking eBPF data collection | | Email + webhook alert channels | Immediate notification when kernel tracing coverage drops | | Status page | Team visibility into eBPF observability health |

bpftrace is one of the most powerful debugging tools in a Kubernetes operator's toolkit — but only when it is actually running. External monitoring is the only reliable way to know when your kernel tracing infrastructure has silently failed.

Get started at vigilmon.online — free tier, no credit card required.