tutorial

Advanced Cilium Network Policy, Tetragon Security, and Clustermesh Monitoring with Vigilmon

Cilium's BPF dataplane and Tetragon security engine expose dozens of failure modes that basic health checks miss. Learn how to monitor BPF map pressure, policy enforcement drop rates, XDP load balancer health, Hubble flow metrics, Cilium identity allocation, and Clustermesh connectivity with Vigilmon.

Cilium replaces kube-proxy with eBPF, enforces network policies at the kernel level, and — with Tetragon — adds runtime security enforcement for syscall and process-level events. This power comes with operational complexity: BPF maps fill up silently under high-scale workloads, policy enforcement metrics reveal drops that never appear in application logs, XDP load balancer programs fail across kernel version boundaries, and Hubble loses flow visibility when the ring buffer overflows.

This guide goes beyond basic Cilium health checks to cover the advanced monitoring signals that matter in production: BPF map pressure, policy drop rates, XDP health, Tetragon enforcement metrics, Cilium identity allocation pressure, and Clustermesh connectivity — integrated with Vigilmon external endpoint monitoring for end-to-end confidence.

Why Advanced Cilium Monitoring Matters

Standard Kubernetes health checks (kubectl get pods) confirm Cilium agent pods are running. They don't tell you:

  • BPF map pressure: CT tables and LB maps have hard size limits. Overflow silently breaks connections without any pod restart or error log.
  • Policy enforcement drops: Packets dropped by network policy don't produce application errors — they just never arrive. Without metrics, policy misconfiguration is invisible.
  • XDP load balancer: XDP programs operate below the kernel networking stack. When they fail or miss a NUMA node, traffic routing breaks in ways that bypass application-level monitoring.
  • Hubble ring buffer overflow: High-throughput clusters fill Hubble's ring buffer and drop flow records, degrading network observability exactly when you need it most.
  • Clustermesh connectivity: Cross-cluster service discovery failures surface as timeout errors in applications, not as Cilium errors.

Step 1: Monitor BPF Map Pressure and Memory Metrics

Cilium exports BPF map utilization metrics via its Prometheus endpoint. Map overflow causes silent connection failures — this is one of the most dangerous Cilium failure modes.

# Verify Cilium Prometheus metrics are accessible
kubectl -n kube-system exec -ti ds/cilium -- cilium metrics list | grep map

# Direct Prometheus endpoint check
curl -s http://CILIUM_AGENT_IP:9962/metrics | grep -E "cilium_bpf_map|cilium_drop"

Key BPF map metrics to alert on:

# Conntrack table utilization (drops connections when full)
cilium_bpf_map_ops_total{map_name="cilium_ct4_global", operation="delete"}
cilium_bpf_map_ops_total{map_name="cilium_ct4_global", operation="insert"}

# LB services map — fills under high service count
cilium_bpf_map_ops_total{map_name="cilium_lb4_services_v2"}

# Policy map errors indicate enforcement pressure
cilium_bpf_map_ops_errors_total

Build an advanced health aggregator:

# cilium_health.py
from flask import Flask, jsonify
import requests
import subprocess
import json
import os

app = Flask(__name__)

CILIUM_METRICS_URL = os.environ.get(
    'CILIUM_METRICS_URL',
    'http://cilium-agent.kube-system.svc.cluster.local:9962/metrics'
)
BPF_ERROR_THRESHOLD = int(os.environ.get('BPF_ERROR_THRESHOLD', '10'))


def parse_prometheus_metrics(text):
    metrics = {}
    for line in text.splitlines():
        if line.startswith('#') or not line.strip():
            continue
        parts = line.rsplit(' ', 1)
        if len(parts) == 2:
            try:
                metrics[parts[0]] = float(parts[1])
            except ValueError:
                pass
    return metrics


@app.route('/health/cilium/bpf-maps')
def cilium_bpf_health():
    try:
        resp = requests.get(CILIUM_METRICS_URL, timeout=10)
        resp.raise_for_status()
        metrics = parse_prometheus_metrics(resp.text)

        # Sum all BPF map operation errors
        total_bpf_errors = sum(
            v for k, v in metrics.items()
            if 'cilium_bpf_map_ops_errors_total' in k
        )

        if total_bpf_errors > BPF_ERROR_THRESHOLD:
            return jsonify(
                status='degraded',
                reason='bpf_map_errors_elevated',
                total_bpf_errors=total_bpf_errors,
                threshold=BPF_ERROR_THRESHOLD,
            ), 503

        return jsonify(
            status='ok',
            total_bpf_errors=total_bpf_errors,
        ), 200

    except Exception as e:
        return jsonify(status='down', error=str(e)), 503


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=int(os.environ.get('PORT', '5013')))

Step 2: Monitor Policy Enforcement Metrics and Drop Rates

Cilium tracks policy enforcement drops per reason code. Sudden spikes in drops indicate policy misconfiguration or policy propagation delays causing legitimate traffic to be rejected.

DROP_RATE_THRESHOLD = int(os.environ.get('DROP_RATE_THRESHOLD', '50'))


@app.route('/health/cilium/policy-drops')
def cilium_policy_drops():
    try:
        resp = requests.get(CILIUM_METRICS_URL, timeout=10)
        resp.raise_for_status()
        metrics = parse_prometheus_metrics(resp.text)

        # Sum all policy-reason drops
        policy_drops = {}
        total_drops = 0
        for key, value in metrics.items():
            if 'cilium_drop_count_total' in key and value > 0:
                # Extract reason label from metric key
                reason = key.split('reason="')[-1].split('"')[0] if 'reason=' in key else 'unknown'
                policy_drops[reason] = policy_drops.get(reason, 0) + value
                total_drops += value

        # Alert on policy-specific drop reasons
        policy_related_drops = sum(
            v for k, v in policy_drops.items()
            if 'policy' in k.lower() or 'denied' in k.lower()
        )

        if policy_related_drops > DROP_RATE_THRESHOLD:
            return jsonify(
                status='degraded',
                reason='policy_drops_elevated',
                policy_related_drops=policy_related_drops,
                threshold=DROP_RATE_THRESHOLD,
                drop_breakdown=policy_drops,
            ), 503

        return jsonify(
            status='ok',
            total_drops=total_drops,
            policy_related_drops=policy_related_drops,
        ), 200

    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

For deeper policy visibility, use cilium monitor output analysis:

# Monitor policy drop events in real time
kubectl -n kube-system exec -ti ds/cilium -- cilium monitor --type drop | head -100

# Get policy enforcement status
kubectl -n kube-system exec -ti ds/cilium -- cilium endpoint list | grep -E "policy|DENIED"

Step 3: Monitor XDP Load Balancer Health

Cilium's XDP load balancer bypasses iptables entirely for north-south traffic. XDP program health is critical — a failed XDP attach silently falls back to slower paths or breaks traffic routing.

@app.route('/health/cilium/xdp')
def cilium_xdp_health():
    try:
        result = subprocess.run(
            ['kubectl', '-n', 'kube-system', 'exec', '-ti', 'ds/cilium', '--',
             'cilium', 'bpf', 'lb', 'list', '--o', 'json'],
            capture_output=True, text=True, timeout=15,
        )

        if result.returncode != 0:
            # Fall back to checking XDP metrics
            resp = requests.get(CILIUM_METRICS_URL, timeout=10)
            resp.raise_for_status()
            metrics = parse_prometheus_metrics(resp.text)

            xdp_handled = sum(
                v for k, v in metrics.items()
                if 'cilium_xdp' in k and 'handled' in k
            )

            return jsonify(
                status='ok' if xdp_handled >= 0 else 'unknown',
                xdp_handled=xdp_handled,
                note='lb_list_unavailable_using_metrics',
            ), 200

        lb_entries = json.loads(result.stdout) if result.stdout.strip() else []
        if not lb_entries:
            return jsonify(
                status='degraded',
                reason='no_xdp_lb_entries_found',
            ), 503

        return jsonify(status='ok', lb_entry_count=len(lb_entries)), 200

    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

Step 4: Monitor Cilium Identity Allocation and Hubble Flow Metrics

Cilium assigns numeric identities to endpoint label sets. Under high pod churn, identity allocation can become the bottleneck — and identity exhaustion causes pod networking to fail silently.

IDENTITY_WARN_THRESHOLD = int(os.environ.get('IDENTITY_WARN_THRESHOLD', '60000'))


@app.route('/health/cilium/identity')
def cilium_identity_health():
    try:
        resp = requests.get(CILIUM_METRICS_URL, timeout=10)
        resp.raise_for_status()
        metrics = parse_prometheus_metrics(resp.text)

        # Total allocated identities
        identity_count = sum(
            v for k, v in metrics.items()
            if 'cilium_identity' in k and 'total' in k
        )

        # Hubble ring buffer drop rate (lost flow records)
        hubble_lost_events = sum(
            v for k, v in metrics.items()
            if 'hubble_lost_events_total' in k
        )

        issues = []
        if identity_count > IDENTITY_WARN_THRESHOLD:
            issues.append(f'identity_count_elevated:{identity_count}')

        if hubble_lost_events > 1000:
            issues.append(f'hubble_ring_buffer_overflow:{hubble_lost_events}')

        if issues:
            return jsonify(
                status='degraded',
                issues=issues,
                identity_count=identity_count,
                hubble_lost_events=hubble_lost_events,
            ), 503

        return jsonify(
            status='ok',
            identity_count=identity_count,
            hubble_lost_events=hubble_lost_events,
        ), 200

    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

Step 5: Monitor Clustermesh Connectivity with Vigilmon

Clustermesh connects multiple Cilium-managed clusters for cross-cluster service discovery. Connectivity failures surface as DNS resolution errors in application pods — not as Cilium errors.

@app.route('/health/cilium/clustermesh')
def cilium_clustermesh_health():
    try:
        result = subprocess.run(
            ['kubectl', '-n', 'kube-system', 'exec', 'ds/cilium', '--',
             'cilium', 'clustermesh', 'status', '--o', 'json'],
            capture_output=True, text=True, timeout=15,
        )

        if result.returncode != 0:
            # Clustermesh may not be configured — treat as ok
            if 'not enabled' in result.stderr.lower() or 'not configured' in result.stderr.lower():
                return jsonify(status='ok', note='clustermesh_not_configured'), 200
            return jsonify(status='down', error=result.stderr.strip()), 503

        status = json.loads(result.stdout) if result.stdout.strip() else {}
        clusters = status.get('clusters', [])

        disconnected = [c for c in clusters if not c.get('ready', True)]
        if disconnected:
            return jsonify(
                status='degraded',
                reason='clustermesh_peers_disconnected',
                disconnected_count=len(disconnected),
                disconnected_clusters=[c.get('name') for c in disconnected],
            ), 503

        return jsonify(
            status='ok',
            connected_clusters=len(clusters),
        ), 200

    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

Step 6: Configure Vigilmon HTTP Monitors for Cilium

Monitor 1: BPF Map Health

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://cilium-health.your-domain.com/health/cilium/bpf-maps
  4. Interval: 2 minutes
  5. Expected: Status 200, body contains "status":"ok"
  6. Alert: Slack + PagerDuty (P1 — BPF map errors can cause silent connection failures)
  7. Save

Monitor 2: Policy Drop Rate

  • URL: https://cilium-health.your-domain.com/health/cilium/policy-drops
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert: Slack + PagerDuty (elevated policy drops indicate misconfiguration causing real traffic loss)

Monitor 3: XDP Load Balancer Health

  • URL: https://cilium-health.your-domain.com/health/cilium/xdp
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert: Slack + PagerDuty (XDP failure degrades load balancing performance significantly)

Monitor 4: Identity and Hubble Flow Health

  • URL: https://cilium-health.your-domain.com/health/cilium/identity
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert: Slack (P2 — identity pressure and Hubble overflow are leading indicators)

Monitor 5: Clustermesh Connectivity

  • URL: https://cilium-health.your-domain.com/health/cilium/clustermesh
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert: Slack + PagerDuty (disconnected cluster peers cause cross-cluster service failures)

Tetragon Security Enforcement Monitoring

Tetragon adds kernel-level security enforcement that can terminate processes violating policy. Monitor Tetragon health to ensure security enforcement is active:

# Check Tetragon agent health
kubectl -n kube-system get pods -l app.kubernetes.io/name=tetragon

# Validate Tetragon is processing events
kubectl -n kube-system exec -ti ds/tetragon -- tetra status

Add Tetragon metrics to the health sidecar:

TETRAGON_METRICS_URL = os.environ.get(
    'TETRAGON_METRICS_URL',
    'http://tetragon.kube-system.svc.cluster.local:2112/metrics'
)


@app.route('/health/tetragon')
def tetragon_health():
    try:
        resp = requests.get(TETRAGON_METRICS_URL, timeout=10)
        resp.raise_for_status()
        metrics = parse_prometheus_metrics(resp.text)

        events_total = sum(v for k, v in metrics.items() if 'tetragon_events_total' in k)
        policy_errors = sum(v for k, v in metrics.items() if 'tetragon_policy' in k and 'error' in k)

        if policy_errors > 0:
            return jsonify(
                status='degraded',
                reason='tetragon_policy_errors',
                policy_errors=policy_errors,
            ), 503

        return jsonify(
            status='ok',
            events_total=events_total,
            policy_errors=policy_errors,
        ), 200

    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

Configure in Vigilmon:

  • URL: https://cilium-health.your-domain.com/health/tetragon
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert: Slack + PagerDuty (Tetragon down means security enforcement is disabled)

Alert Routing Summary

| Monitor | Signal | Priority | |---|---|---| | BPF map health | Map errors causing silent connection drops | P1 | | Policy drop rate | Policy enforcement misconfiguration | P1 | | XDP load balancer | XDP program health and traffic routing | P1 | | Identity and Hubble | Identity pressure and flow visibility | P2 | | Clustermesh connectivity | Cross-cluster service discovery | P1 | | Tetragon enforcement | Runtime security policy enforcement active | P1 |

Summary

Cilium's eBPF dataplane and Tetragon's security enforcement are powerful precisely because they operate below the application layer — which also means their failures are invisible at the application layer. External monitoring with Vigilmon creates the independent signal layer that confirms Cilium and Tetragon are working correctly:

| What Vigilmon Monitors | Why It Matters | |---|---| | BPF map error rate | Detects silent connection failures before applications notice | | Policy enforcement drops | Catches policy misconfiguration causing real traffic loss | | XDP load balancer health | Validates kernel-level LB is attached and routing correctly | | Identity allocation pressure | Warns before exhaustion breaks new pod networking | | Hubble ring buffer | Flags observability gaps before troubleshooting sessions fail | | Clustermesh connectivity | Detects cross-cluster link failures before service timeouts cascade | | Tetragon policy health | Confirms runtime security enforcement is active |

Get started free at vigilmon.online — your first Cilium advanced monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles