How to Monitor Inspektor Gadget Infrastructure with Vigilmon

Inspektor Gadget is an eBPF-based toolkit for debugging and inspecting Kubernetes applications — it provides real-time visibility into system calls, network events, file activity, and process behaviour directly from the Linux kernel. When Inspektor Gadget is running, your platform engineering teams can diagnose Kubernetes issues in seconds. When it breaks, that visibility disappears silently and engineers are flying blind.

In this tutorial you'll set up external uptime monitoring for your Inspektor Gadget deployment using Vigilmon — free tier, no credit card required.

Why Inspektor Gadget needs monitoring

Inspektor Gadget runs as a privileged DaemonSet (the gadget DaemonSet) and an operator/controller. Both have failure modes that won't surface through standard Kubernetes tooling:

• DaemonSet pod crash — a gadget pod on one node crashes and isn't replaced; that node loses all eBPF tracing visibility without any cluster-level alert
• eBPF program load failure — a kernel version mismatch or BTF (BPF Type Format) incompatibility causes eBPF programs to fail loading; the pod stays Running but all traces return empty
• Privileged access revocation — a cluster security policy change strips the CAP_SYS_ADMIN or CAP_BPF capabilities from gadget pods; gadgets silently stop working
• Gadget image registry outage — when a node is re-provisioned, it can't pull the gadget image, leaving that node unmonitored
• API server connectivity loss — the gadget operator loses contact with the Kubernetes API and stops reconciling trace jobs

External monitoring of the Inspektor Gadget API surface catches these failures before engineers notice they're missing observability data.

What you'll need

• A Kubernetes cluster with Inspektor Gadget installed via the kubectl gadget plugin or Helm
• Access to expose the Inspektor Gadget API service
• A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose the Inspektor Gadget health endpoint

Inspektor Gadget's operator exposes a health endpoint at /healthz. Expose it for external monitoring:

# Check the current operator service
kubectl get svc -n gadget

# Port-forward to verify the health endpoint
kubectl port-forward -n gadget svc/gadget 8080:8080
curl http://localhost:8080/healthz
# {"status":"ok"}

Create a NodePort service for external monitoring:

# gadget-health-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: gadget-health
  namespace: gadget
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: gadget
    app.kubernetes.io/component: operator
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30808
      protocol: TCP

kubectl apply -f gadget-health-nodeport.yaml

# Verify
curl http://<node-ip>:30808/healthz
# {"status":"ok"}

Step 2: Set up a gadget DaemonSet coverage monitor

Beyond the operator, you need to know that the gadget DaemonSet has a healthy pod on every node. Create a custom health endpoint that reports DaemonSet coverage:

# gadget-coverage-probe.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gadget-coverage-probe
  namespace: gadget
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gadget-coverage-probe
  template:
    metadata:
      labels:
        app: gadget-coverage-probe
    spec:
      serviceAccountName: gadget-coverage-reader
      containers:
        - name: probe
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                DESIRED=$(kubectl get daemonset gadget -n gadget \
                  -o jsonpath='{.status.desiredNumberScheduled}')
                READY=$(kubectl get daemonset gadget -n gadget \
                  -o jsonpath='{.status.numberReady}')
                
                if [ "$DESIRED" = "$READY" ] && [ "$DESIRED" -gt "0" ]; then
                  echo "status=ok desired=$DESIRED ready=$READY" > /tmp/health
                else
                  echo "status=degraded desired=$DESIRED ready=$READY" > /tmp/health
                fi
                sleep 30
              done
          ports:
            - containerPort: 8888
          livenessProbe:
            exec:
              command: ["grep", "-q", "status=ok", "/tmp/health"]
            periodSeconds: 60

Pair this with a service account that has read access to the gadget DaemonSet:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gadget-coverage-reader
  namespace: gadget
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gadget-coverage-reader
rules:
  - apiGroups: ["apps"]
    resources: ["daemonsets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gadget-coverage-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gadget-coverage-reader
subjects:
  - kind: ServiceAccount
    name: gadget-coverage-reader
    namespace: gadget

Step 3: Set up HTTP monitoring in Vigilmon

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS as the type
3. Set the URL to your gadget health endpoint: http://your-node.example.com:30808/healthz
4. Set the check interval to 1 minute
5. Under Expected response, set:
   • Status code: 200
   • Response body contains: ok
6. Save the monitor

Vigilmon probes from multiple regions simultaneously. If the Inspektor Gadget operator crashes or becomes unresponsive, you'll know within a minute — not when an engineer notices their trace commands returning nothing.

Step 4: Monitor the gadget gRPC API port

Inspektor Gadget exposes a gRPC API for the kubectl gadget client. TCP monitoring on this port catches API server failures:

# gadget-grpc-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: gadget-grpc
  namespace: gadget
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: gadget
    app.kubernetes.io/component: operator
  ports:
    - name: grpc
      port: 4444
      targetPort: 4444
      nodePort: 30444
      protocol: TCP

Then in Vigilmon:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your-node.example.com / 30444
4. Save the monitor

Step 5: Create an eBPF functionality heartbeat

The deepest test of Inspektor Gadget is whether it can actually run an eBPF gadget. Create a CronJob that runs a lightweight trace and pings a heartbeat:

# gadget-ebpf-verify.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: gadget-ebpf-verify
  namespace: gadget
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: verify
              image: ghcr.io/inspektor-gadget/ig:latest
              securityContext:
                privileged: true
              command:
                - /bin/sh
                - -c
                - |
                  # Run a 5-second DNS trace to verify eBPF programs load
                  timeout 10 ig run trace_dns --timeout 5s \
                    --output json 2>&1 | head -5
                  
                  if [ $? -eq 0 ]; then
                    curl -fsS "https://hb.vigilmon.online/your-heartbeat-token"
                    echo "eBPF trace verified"
                  else
                    echo "ERROR: eBPF trace failed" >&2
                    exit 1
                  fi

In Vigilmon, create a Heartbeat monitor with a 15-minute interval. If eBPF programs stop loading — due to kernel incompatibility, capability loss, or a gadget image issue — the heartbeat stops pinging and you get an alert.

Step 6: Configure alert channels

1. In Vigilmon, go to Alert Channels → Add Channel → Webhook
2. Enter your Slack, PagerDuty, or Opsgenie webhook URL
3. Assign all Inspektor Gadget monitors to the channel

When an alert fires, run immediate triage:

# DaemonSet status
kubectl get daemonset gadget -n gadget
kubectl get pods -n gadget -o wide

# Check for eBPF load errors
kubectl logs -n gadget -l app.kubernetes.io/name=gadget --tail=50 | grep -i "error\|fail\|btf"

# Verify kernel BTF support on nodes
kubectl get nodes -o custom-columns=\
  NAME:.metadata.name,\
  KERNEL:.status.nodeInfo.kernelVersion

# Check capabilities
kubectl get pod -n gadget -l app.kubernetes.io/name=gadget \
  -o jsonpath='{.items[0].spec.containers[0].securityContext}'

Step 7: Create a status page

For platform engineering teams relying on Inspektor Gadget for production debugging:

1. In Vigilmon, go to Status Pages → New Status Page
2. Name it "Observability Infrastructure"
3. Add Inspektor Gadget monitors alongside your other observability tools
4. Publish and share with your platform engineering teams

Putting it all together

| Monitor | Type | What it catches | |---------|------|-----------------| | http://node:30808/healthz | HTTP | Operator crashes, API server connectivity loss | | node:30444 | TCP | gRPC API failures | | eBPF trace heartbeat | Heartbeat | eBPF program load failures, kernel incompatibility | | DaemonSet coverage probe | HTTP | Partial DaemonSet failures (nodes losing gadget pods) |

What's next

• Kernel version alerts — monitor node kernel versions across your cluster; kernel upgrades can break BTF compatibility with existing eBPF gadgets
• Gadget image tag pinning — use Vigilmon's SSL/TLS certificate monitoring on your container registry endpoints to catch registry outages before node re-provisioning fails
• Integration with Prometheus — Inspektor Gadget exports Prometheus metrics; add HTTP monitoring on your Prometheus scrape endpoints so you know when gadget metrics stop flowing

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.