How to Monitor kubectl-trace with Vigilmon

kubectl-trace is a kubectl plugin that schedules and runs bpftrace programs directly on Kubernetes nodes without requiring privileged access to the node shell. It submits trace runner Jobs to the cluster — each job mounts the node's host PID namespace, attaches eBPF probes, and streams the output back to your terminal. Teams use kubectl-trace to profile system calls, investigate network packet loss, and diagnose latency spikes across specific pods or node-level kernel functions.

But kubectl-trace depends on a set of cluster-level components to operate: the trace runner image must be pullable on each node, the necessary host namespaces must be grantable, and (when using the kubectl-trace operator) a controller must be healthy. When any of these fail, every trace attempt silently produces no output or an opaque permission error. In this tutorial you'll set up monitoring for the kubectl-trace operator and its trace runner infrastructure using Vigilmon — free tier, no credit card required.

Why kubectl-trace needs external monitoring

kubectl-trace failures are often invisible until an operator actually tries to run a trace during an incident:

• Operator controller down — when using the kubectl-trace operator, a crashed controller means all TraceRun custom resources queue without execution; the outage is only discovered when debugging is urgently needed
• Trace runner image pull failures — if the trace runner image is unavailable (registry outage, network policy change, image removed), every trace job fails to start with ErrImagePull
• RBAC revoked — a cluster policy change removes the service account permissions needed to create privileged Jobs; every kubectl trace run returns a permissions error
• BTF/kernel incompatibility introduced by a node upgrade — new kernel versions on upgraded nodes break existing bpftrace programs silently; coverage gaps appear as some nodes start returning empty trace output
• Node taints blocking trace runner scheduling — a node that received a new taint after a hardware event cannot schedule trace runner Pods; the trace appears to start but never outputs data

External monitoring from Vigilmon catches the operator health failures before they become an urgent problem during an incident.

What you'll need

• kubectl-trace installed as a kubectl plugin (kubectl krew install trace)
• Optionally: the kubectl-trace operator deployed to your cluster
• An HTTP endpoint exposed by the operator or a wrapper health check (see Step 1)
• A free Vigilmon account

Step 1: Expose a health endpoint for kubectl-trace infrastructure

The kubectl-trace operator exposes a metrics and health endpoint when deployed. If you are using the operator, expose it via a Service:

# kubectl-trace-operator-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kubectl-trace-operator-metrics
  namespace: kubectl-trace
  labels:
    app: kubectl-trace-operator
spec:
  type: ClusterIP
  selector:
    app: kubectl-trace-operator
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubectl-trace-operator-ingress
  namespace: kubectl-trace
spec:
  ingressClassName: nginx
  rules:
    - host: trace-ops.yourdomain.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: kubectl-trace-operator-metrics
                port:
                  number: 8080

kubectl apply -f kubectl-trace-operator-service.yaml

# Verify operator health
kubectl port-forward svc/kubectl-trace-operator-metrics 8080:8080 -n kubectl-trace &
curl http://localhost:8080/healthz
# {"status":"ok"}

If you are using kubectl-trace without the operator (direct Job submission), deploy a lightweight infrastructure checker:

# kubectl-trace-checker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubectl-trace-checker
  namespace: kubectl-trace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubectl-trace-checker
  template:
    metadata:
      labels:
        app: kubectl-trace-checker
    spec:
      serviceAccountName: kubectl-trace-checker
      containers:
        - name: checker
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              # Simple HTTP server that checks whether trace runner prerequisites are met
              while true; do
                # Verify the trace runner image is pullable by checking recent jobs
                FAILED=$(kubectl get jobs -n kubectl-trace --field-selector=status.failed=1 \
                  -o jsonpath='{.items[*].metadata.name}' 2>/dev/null | wc -w)
                if [ "$FAILED" -gt 0 ]; then
                  echo "warn: $FAILED failed trace jobs" > /tmp/status
                else
                  echo "ok" > /tmp/status
                fi
                sleep 60
              done &
              # Serve health via netcat
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                printf "HTTP/1.1 200 OK\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 9090 -q 1
              done
          ports:
            - containerPort: 9090

Step 2: Verify trace runner job scheduling on all nodes

Before relying on kubectl-trace during incidents, validate that trace runners can schedule on every node:

#!/bin/bash
# verify-kubectl-trace-nodes.sh
echo "Checking trace runner schedulability on all nodes..."

for NODE in $(kubectl get nodes -o name | cut -d/ -f2); do
  # Check for disqualifying conditions
  TAINTS=$(kubectl get node "$NODE" -o jsonpath='{.spec.taints[*].effect}' 2>/dev/null)
  if echo "$TAINTS" | grep -q "NoSchedule\|NoExecute"; then
    echo "WARN: $NODE has taints — trace runners may not schedule: $TAINTS"
  else
    echo "OK: $NODE — no blocking taints"
  fi
done

# Check for any stuck trace runner jobs
echo ""
echo "Checking for stuck trace jobs..."
kubectl get jobs -n kubectl-trace -o wide 2>/dev/null || echo "No trace jobs found (or namespace missing)"

# Check recent trace runner pod failures
kubectl get pods -n kubectl-trace -o wide 2>/dev/null | grep -E "Error|CrashLoop|ImagePull" \
  && echo "WARNING: failed trace runner pods found" || echo "No failed trace runner pods"

chmod +x verify-kubectl-trace-nodes.sh
./verify-kubectl-trace-nodes.sh

Step 3: Set up HTTP monitoring in Vigilmon

With your health endpoint exposed, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add monitors for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | kubectl-trace operator health | https://trace-ops.yourdomain.com/healthz | 200 | | kubectl-trace metrics | https://trace-ops.yourdomain.com/metrics | 200 | | Infrastructure checker | https://trace-ops.yourdomain.com/status | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match ok in the response body
6. Save each monitor

Step 4: Monitor the trace operator TCP port

Add a TCP-layer monitor to catch situations where the operator port is unreachable independently of HTTP-level health:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your operator service hostname and port 8080
4. Save the monitor

A TCP failure when the operator pod is Running usually indicates a network policy change or Service selector drift — exactly the kind of silent configuration error that breaks trace scheduling without any alert.

Step 5: Configure alert channels

kubectl-trace is the tool your engineers reach for when debugging production incidents. Alert on its health proactively so it is ready when you need it.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your platform or SRE on-call address
3. Assign the channel to all kubectl-trace monitors

Webhook alerts for incident routing

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. The payload Vigilmon sends:

{
  "monitor_name": "kubectl-trace operator health",
  "status": "down",
  "url": "https://trace-ops.yourdomain.com/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Wire this alert to a runbook that checks operator controller health, verifies trace runner image availability, and confirms RBAC is intact.

Step 6: Correlate Vigilmon alerts with kubectl-trace diagnostics

When you receive a kubectl-trace downtime alert, run these checks:

# 1. Check operator controller pod status
kubectl get pods -n kubectl-trace -l app=kubectl-trace-operator

# 2. Check operator logs for errors
kubectl logs -n kubectl-trace -l app=kubectl-trace-operator --tail=100 | grep -i "error\|fail"

# 3. Check for TraceRun CRDs stuck in pending state
kubectl get tracerun -A 2>/dev/null || echo "TraceRun CRD not installed"

# 4. Check recent trace runner job failures
kubectl get jobs -n kubectl-trace -o wide
kubectl describe jobs -n kubectl-trace | grep -A5 "Failed\|Error"

# 5. Verify trace runner image is pullable
kubectl run test-pull --image=quay.io/iovisor/kubectl-trace-runner:latest \
  --restart=Never --dry-run=client -n kubectl-trace

# 6. Verify RBAC permissions are intact
kubectl auth can-i create jobs -n kubectl-trace \
  --as=system:serviceaccount:kubectl-trace:kubectl-trace-operator

# 7. Run a quick test trace to validate end-to-end
NODE=$(kubectl get nodes -o name | head -1 | cut -d/ -f2)
timeout 15 kubectl trace run node/$NODE -e 'BEGIN { printf("trace-ok\n"); exit(); }' \
  && echo "TRACE OK" || echo "TRACE FAILED"

If Vigilmon shows the operator health endpoint down, start with the operator pod logs — the most common cause is a Kubernetes API server connectivity issue or an expired service account token.

Step 7: Create a status page for distributed tracing infrastructure

kubectl-trace readiness is a hidden dependency for your incident response playbooks. Make its status visible:

1. Go to Status Pages → New Status Page
2. Name it: "Distributed Kernel Tracing"
3. Add your monitors: operator health, metrics, TCP port
4. Share the URL with SRE leads and platform engineering teams

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on operator /healthz | Controller crash, API server connectivity failures | | HTTP monitor on /metrics | Metrics collection gaps for the tracing stack | | TCP monitor on operator port | Network policy changes blocking operator access | | Email + webhook alert channels | Immediate notification before tracing is needed in an incident | | Status page | SRE visibility into trace infrastructure readiness |

kubectl-trace is your emergency diagnostic tool — the one you reach for when nothing else explains what is happening on a node. External monitoring ensures it is actually ready when that moment arrives.

Get started at vigilmon.online — free tier, no credit card required.