How to Monitor Cilium with Vigilmon

Cilium is the eBPF-based networking and security layer for Kubernetes that powers everything from pod-to-pod traffic enforcement to multi-cluster connectivity. It replaces kube-proxy with eBPF datapath programs, enforces NetworkPolicy at L3/L4/L7, and — through Hubble — provides deep network observability without any application instrumentation.

When Cilium degrades, the consequences are severe and non-obvious. NetworkPolicy enforcement can become inconsistent (some traffic gets through that shouldn't; other traffic gets dropped that should flow). Hubble relay goes down and your network observability disappears. The Cilium agent on a node crashes and that node's pods lose connectivity. None of these events generate user-visible errors immediately — they surface as intermittent connectivity issues that are extremely hard to debug without knowing that the Cilium agent was unhealthy.

In this tutorial you'll set up external health monitoring for Cilium using Vigilmon — covering the Cilium agent API, Hubble relay, and policy enforcement availability.

Why Cilium needs external monitoring

Cilium exposes health endpoints precisely because its internal state is complex. The agent can be in a partially-initialized state where eBPF programs are loaded but not enforcing policy correctly. The Hubble relay can be disconnected from agents. The operator can lose sync with the Kubernetes API.

These failures don't always crash the agent — they produce degraded operation that looks fine from outside until something depends on the broken functionality.

External monitoring with Vigilmon catches:

• Cilium agent process failure — pods on the affected node may lose network connectivity entirely, or traffic bypasses policy enforcement
• Hubble relay unavailability — your network observability layer is blind; you can't trace connections or debug policy violations
• Operator failure — new nodes joining the cluster won't get Cilium initialized properly; NodePort services may break
• API server connectivity loss — the agent loses sync with Kubernetes and starts operating on stale policy state
• eBPF map pressure — the agent health endpoint degrades when BPF maps are close to capacity, a common production failure mode

What you'll need

• A Kubernetes cluster running Cilium (1.12+ recommended)
• Cilium agent API accessible on each node (default port: 9879)
• Hubble relay deployed (port 4245 by default, or via the Hubble UI service)
• A free Vigilmon account

Step 1: Expose the Cilium agent health API

The Cilium agent exposes a REST API on each node at port 9879. By default this is only accessible within the node. To monitor it externally, expose it via a Kubernetes Service:

# cilium-agent-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: cilium-agent-health
  namespace: kube-system
  annotations:
    description: "Exposes Cilium agent health API for external monitoring"
spec:
  type: NodePort
  selector:
    k8s-app: cilium
  ports:
    - name: health
      port: 9879
      targetPort: 9879
      nodePort: 30879
      protocol: TCP

kubectl apply -f cilium-agent-health-service.yaml

# Verify the endpoint
curl -s http://<node-ip>:30879/healthz
# Expected: {"cilium":{"ok":true},"kube-apiserver":{"ok":true},"kvstore":{"ok":true},"nodeConnectivity":{...}}

# Or use the cilium CLI directly on a node
cilium status --brief
# Expected: KVStore: Ok   API Server: Ok   Kubelet: Ok

For a quick check on agent readiness:

curl -s http://<node-ip>:30879/ready
# Expected: HTTP 200

Step 2: Monitor the Cilium agent health endpoint

This is your primary health signal — it reflects agent connectivity to the KV store, API server, and inter-node networking.

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL: http://<node-ip>:30879/healthz
4. Check interval: 1 minute
5. Expected response:
   • Status code: 200
6. Save the monitor

The /healthz endpoint returns a non-200 when any of the following are broken: KV store connectivity (etcd or CRD mode), Kubernetes API server connectivity, or inter-node network health checks.

For a cluster with multiple nodes, monitor a representative sample — at minimum the control plane node(s) and one or two worker nodes:

| Monitor | URL | Node role | |---------|-----|-----------| | Cilium agent (control-plane) | http://control-plane:30879/healthz | Control plane | | Cilium agent (worker-1) | http://worker-1:30879/healthz | Worker | | Cilium agent (worker-2) | http://worker-2:30879/healthz | Worker |

Step 3: Monitor Hubble relay health

Hubble relay aggregates flow data from all Cilium agents and serves it to the Hubble UI and CLI. It exposes an HTTP health endpoint:

# Check Hubble relay health
kubectl port-forward -n kube-system svc/hubble-relay 4245:80
curl -s http://localhost:4245/healthz
# Expected: HTTP 200 and JSON with relay connectivity status

To make Hubble relay reachable externally for monitoring, expose it via a NodePort or LoadBalancer service:

# hubble-relay-monitor-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: hubble-relay-health
  namespace: kube-system
spec:
  type: LoadBalancer
  selector:
    k8s-app: hubble-relay
  ports:
    - name: http
      port: 80
      targetPort: 4245

kubectl apply -f hubble-relay-monitor-service.yaml
kubectl get svc -n kube-system hubble-relay-health
# Note the EXTERNAL-IP

Then add the monitor in Vigilmon:

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: http://<hubble-relay-external-ip>/healthz
3. Check interval: 1 minute
4. Expected response:
   • Status code: 200
5. Save the monitor

Step 4: Monitor the Cilium operator

The Cilium operator manages CiliumNetworkPolicy resources, node initialization, and load balancer IP allocation. If it fails, new workloads won't get proper networking configuration:

# Check operator health via its HTTP endpoint
kubectl port-forward -n kube-system deployment/cilium-operator 9234:9234
curl -s http://localhost:9234/healthz
# Expected: HTTP 200

Expose via NodePort:

# cilium-operator-health.yaml
apiVersion: v1
kind: Service
metadata:
  name: cilium-operator-health
  namespace: kube-system
spec:
  type: NodePort
  selector:
    name: cilium-operator
  ports:
    - port: 9234
      targetPort: 9234
      nodePort: 30234

Add to Vigilmon:

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: http://<node-ip>:30234/healthz
3. Check interval: 2 minutes
4. Expected response: Status code 200
5. Save the monitor

Step 5: Add TCP monitoring for the agent API port

Add a TCP-level check to catch cases where the agent port is completely unreachable (process crash, port binding failure):

1. Monitors → New Monitor → TCP Port
2. Host: <node-ip>
3. Port: 30879
4. Check interval: 1 minute
5. Save the monitor

If the TCP monitor fires but the HTTP health monitor was showing 200, you have a race condition where the health check endpoint was up briefly and then the agent crashed. The TCP monitor will fire slightly before the HTTP monitor would catch the next failure.

Step 6: Configure alert channels

Cilium failures can result in complete network connectivity loss for affected pods. Alert routing should be treated as infrastructure-critical:

Email alerts

1. Alert Channels → Add Channel → Email
2. Add your platform/infrastructure team address and on-call rotation
3. Assign to all Cilium monitors

Webhook alerts

1. Alert Channels → Add Channel → Webhook
2. Configure PagerDuty or OpsGenie for agent health monitors (P1)
3. Configure Slack for operator and Hubble relay monitors (P2)

Example Vigilmon alert payload:

{
  "monitor_name": "Cilium agent healthz (worker-1)",
  "status": "down",
  "url": "http://worker-1:30879/healthz",
  "started_at": "2026-01-15T14:23:00Z",
  "duration_seconds": 180
}

Recommended alert priorities

| Monitor | Priority | Rationale | |---------|----------|-----------| | Cilium agent /healthz | P1 | Pod network connectivity at risk | | Agent TCP port | P1 | Agent process may be dead | | Hubble relay | P2 | Network observability degraded | | Cilium operator | P2 | New workloads may not get networking |

Step 7: Diagnostic runbook when alerts fire

When Vigilmon fires a Cilium agent alert, follow this sequence:

# 1. Check overall Cilium status
kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium status
# Look for: any component showing "Error" or "Warning"

# 2. Check agent logs for errors
kubectl logs -n kube-system -l k8s-app=cilium --tail 50 | grep -E 'error|Error|failed|Fatal'

# 3. Check node connectivity
kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-health status
# Shows inter-node probe results

# 4. Check BPF map pressure (a common culprit)
kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium bpf metrics list | grep "too many open files"

# 5. Restart the agent on the affected node if needed
kubectl delete pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName=<node-name>

Complete monitoring summary

| Monitor | Type | Endpoint | Priority | |---------|------|----------|----------| | Cilium agent health | HTTP | http://node:30879/healthz | P1 | | Agent port | TCP | node:30879 | P1 | | Hubble relay | HTTP | http://relay-lb/healthz | P2 | | Cilium operator | HTTP | http://node:30234/healthz | P2 |

Summary

Cilium failures are silent and potentially catastrophic — inconsistent policy enforcement, broken pod networking, and vanishing observability data can all occur without obvious error messages. External monitoring with Vigilmon gives you a reliable signal that doesn't depend on Cilium being healthy enough to report its own problems.

Start monitoring Cilium for free at vigilmon.online.