tutorial

Monitoring Hubble with Vigilmon

Hubble is Cilium's network observability layer using eBPF. Learn how to monitor Hubble relay health, flow export rate, peer connectivity, DNS tracking, network policy drops, and Vigilmon integration.

Hubble is the network observability component of Cilium — the eBPF-based Kubernetes networking and security platform. While Cilium handles network policy enforcement and load balancing at the kernel level, Hubble provides visibility into those network flows: which pods are communicating, what DNS queries are being made, which network policy rules are dropping traffic, and where HTTP errors are occurring in the service mesh. Hubble's architecture runs an agent on each node (as part of the Cilium DaemonSet), which exports flow data to the Hubble Relay for aggregation, and optionally to the Hubble UI for visualization. When Hubble Relay goes down, network flow observability stops cluster-wide. When individual node agents lose connectivity, flows from those nodes disappear silently. Vigilmon gives you the external monitoring layer to catch Hubble infrastructure failures before they blind your network security team.

What You'll Set Up

  • HTTP monitor for the Hubble Relay health endpoint
  • Hubble UI availability check
  • Peer connectivity status monitoring
  • Flow export rate validation
  • DNS resolution tracking health
  • Network policy drop rate monitoring via Vigilmon endpoint checks

Prerequisites

  • Cilium with Hubble enabled (hubble.enabled: true in Helm values) deployed on Kubernetes 1.24+
  • Hubble Relay running (hubble.relay.enabled: true)
  • The Hubble Relay service exposed via an ingress or LoadBalancer for external checks
  • Optionally, Hubble UI deployed (hubble.ui.enabled: true)
  • A free Vigilmon account

Step 1: Monitor the Hubble Relay Health Endpoint

The Hubble Relay is the central aggregation point for all flow data from node-level Hubble agents. If Relay goes down, you lose the ability to query flows cluster-wide, run hubble observe, or see data in Hubble UI.

Hubble Relay exposes a gRPC health check, but for HTTP-based external monitoring, expose its health endpoint via a sidecar or metrics port:

GET https://hubble-relay.example.com/healthz

In the cilium Helm chart, the Relay deployment includes a health probe on port 4245 (gRPC) and an optional metrics port on 9966. To expose an HTTP health endpoint accessible from Vigilmon, configure an ingress for the Relay metrics port:

# hubble-relay-healthz-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hubble-relay-healthz
  namespace: kube-system
spec:
  rules:
  - host: hubble-relay.example.com
    http:
      paths:
      - path: /metrics
        pathType: Prefix
        backend:
          service:
            name: hubble-relay
            port:
              number: 9966

Then configure Vigilmon:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter URL: https://hubble-relay.example.com/metrics.
  4. Set Expected HTTP status to 200.
  5. Set Expected body contains to hubble_relay (confirms Relay-specific metrics are present).
  6. Set Check interval to 2 minutes.
  7. Click Save.

If Relay is unreachable, all hubble observe commands and Hubble UI stop updating — an immediate signal that network observability has gone dark.

Step 2: Monitor the Hubble UI

The Hubble UI is a React-based web interface for visualizing network flows and service dependencies. While not critical-path (operators can still use hubble observe CLI without the UI), the UI is the primary interface for security teams reviewing network policy violations and flow anomalies.

  1. Add Monitor → HTTP / HTTPS.
  2. URL: https://hubble-ui.example.com.
  3. Set Expected HTTP status to 200.
  4. Set Expected body contains to Hubble (matches the page title or app name in the response).
  5. Set Check interval to 5 minutes.
  6. Click Save.

The Hubble UI itself is a static SPA fronted by a Go backend that proxies gRPC calls to the Hubble Relay. A 200 on the root path confirms the backend is running, but a deeper check asserts both the frontend and its Relay connection are working:

  1. Add Monitor → HTTP / HTTPS.
  2. URL: https://hubble-ui.example.com/api/config.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 5 minutes.
  5. Click Save.

The /api/config endpoint is served by the Hubble UI backend and confirms the backend process is alive and configured.

Step 3: Monitor Hubble Peer Connectivity

Hubble Relay discovers and connects to each node's Hubble agent via the "Hubble Peers" API. If a node's Hubble agent is unreachable, Relay marks it as disconnected and flows from that node disappear from all queries. On a large cluster, partial peer disconnection is easy to miss.

Expose peer status via the Hubble CLI as a lightweight HTTP endpoint (using a small wrapper service, or check via hubble status in a scheduled job):

# Check Hubble peer connectivity
hubble status --server hubble-relay.example.com:80

# Sample output when healthy:
# Hubble: Ok  Current/Max Flows: 13,140/16,384  Flows/s: 42.1
# Nodes: 5/5 Connected  Uptime: 3h5m58.888s

For Vigilmon, configure a lightweight monitoring exporter that runs hubble status and exposes the result as an HTTP endpoint:

# hubble-status-exporter (example)
from flask import Flask, jsonify
import subprocess, json

app = Flask(__name__)

@app.route('/healthz')
def health():
    result = subprocess.run(
        ['hubble', 'status', '--output', 'json'],
        capture_output=True, text=True
    )
    data = json.loads(result.stdout)
    connected = data.get('nodes', {}).get('connected', 0)
    total = data.get('nodes', {}).get('total', 0)
    ok = connected == total
    return jsonify({'connected': connected, 'total': total, 'ok': ok}), 200 if ok else 503

app.run(host='0.0.0.0', port=8080)

Deploy this exporter and configure Vigilmon:

  1. Add Monitor → HTTP / HTTPS.
  2. URL: https://hubble-status.example.com/healthz.
  3. Set Expected HTTP status to 200.
  4. Set Expected body contains to "ok":true.
  5. Set Check interval to 5 minutes.
  6. Click Save.

When the check returns 503, partial or total peer disconnection has occurred — Vigilmon alerts your team before they notice missing flows in dashboards.

Step 4: Monitor Flow Export Rate

Hubble's value is the continuous stream of network flows from every node. If the flow export rate drops to zero, eBPF probes may have detached, the Hubble agent may have crashed, or Relay may have lost connections.

The Hubble Relay metrics endpoint includes hubble_flows_processed_total — a counter of flows processed. You can build a simple check that this counter is incrementing by polling the metrics endpoint and checking for a recent value:

Alternatively, the Hubble Relay metrics include the current flows-per-second rate:

GET https://hubble-relay.example.com/metrics

# Look for:
hubble_relay_num_connected_nodes{} 5
hubble_flows_processed_total{...} 1234567

Configure Vigilmon to check the connected nodes metric:

  1. Add Monitor → HTTP / HTTPS.
  2. URL: https://hubble-relay.example.com/metrics.
  3. Set Expected body contains to hubble_relay_num_connected_nodes.
  4. Set Check interval to 3 minutes.
  5. Click Save.

For a more precise check, use Expected body contains with the actual node count (e.g. 5) after hubble_relay_num_connected_nodes{} — this is brittle if your node count changes, so prefer checking the metric name exists and the Relay is running.

Step 5: Monitor DNS Resolution Tracking Health

One of Hubble's most valuable features is DNS observability — it captures every DNS query made by pods in the cluster, which is critical for security auditing (detecting DNS exfiltration, C2 beaconing) and debugging (missing DNS config, split-horizon DNS failures).

DNS tracking in Hubble is captured by the cilium-dns-proxy component embedded in Cilium. If DNS tracking stops, Hubble's dns_queries table becomes empty. Validate DNS tracking health:

# Check for recent DNS flows in Hubble
hubble observe --type l7 --protocol DNS --last 100 --output json

For a Vigilmon-compatible check, expose a DNS tracking health endpoint from your Hubble status exporter:

@app.route('/dns-health')
def dns_health():
    result = subprocess.run(
        ['hubble', 'observe', '--type', 'l7', '--protocol', 'DNS',
         '--last', '10', '--output', 'json'],
        capture_output=True, text=True, timeout=10
    )
    events = [json.loads(l) for l in result.stdout.strip().split('\n') if l]
    return jsonify({'dns_events': len(events), 'ok': len(events) > 0}), \
           200 if events else 503
  1. Add Monitor → HTTP / HTTPS.
  2. URL: https://hubble-status.example.com/dns-health.
  3. Set Expected HTTP status to 200.
  4. Set Expected body contains to "ok":true.
  5. Set Check interval to 10 minutes.
  6. Click Save.

DNS tracking failure often precedes broader Hubble agent issues — it's an early-warning indicator that eBPF probes are degrading.

Step 6: Monitor Network Policy Drop Rate

Hubble captures all network policy drops with the drop reason, source pod, destination pod, and policy rule. A spike in drops can indicate:

  • A recently applied Cilium NetworkPolicy that is too restrictive
  • A misconfigured service selector causing traffic to be blocked
  • A security incident (unexpected connection attempts)

For Vigilmon-based monitoring of drop rates, expose a drop-rate check endpoint:

@app.route('/policy-drops')
def policy_drops():
    result = subprocess.run(
        ['hubble', 'observe', '--verdict', 'DROPPED', '--last', '50',
         '--output', 'json'],
        capture_output=True, text=True, timeout=10
    )
    drops = [json.loads(l) for l in result.stdout.strip().split('\n') if l]
    # Alert if more than 20 drops in the last 50 events
    high_drop_rate = len(drops) > 20
    return jsonify({'drops': len(drops), 'high_rate': high_drop_rate}), \
           503 if high_drop_rate else 200
  1. Add Monitor → HTTP / HTTPS.
  2. URL: https://hubble-status.example.com/policy-drops.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 5 minutes.
  5. Click Save.

When this check returns 503, Vigilmon alerts your team to investigate unexpected network policy drops — the kind of misconfiguration that silently breaks inter-service communication without any application-level error logs.

Step 7: SSL Certificate Monitoring

Hubble UI and the Relay API endpoint should both be monitored for certificate expiry:

  1. Add Monitor → SSL Certificate.
  2. URL: https://hubble-ui.example.com.
  3. Set Alert when certificate expires in less than 30 days.
  4. Click Save.

Add a second SSL check for https://hubble-relay.example.com if you expose the Relay via a separate TLS-terminated endpoint. Relay uses mTLS internally between nodes, but its external management interface often uses a separate certificate.

Step 8: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon → add Slack, PagerDuty, or email.
  2. For the Relay health and peer connectivity checks: Consecutive failures before alert = 1 — losing Relay means losing cluster-wide network visibility immediately.
  3. For the Hubble UI and DNS tracking checks: 2 consecutive failures (the UI has known brief interruptions during Cilium DaemonSet rolling updates).
  4. For the policy drop rate check: 1 consecutive failure — a high drop rate warrants immediate investigation.
  5. Add Maintenance windows for Cilium upgrades (DaemonSet rolling updates cause brief connectivity interruptions per node).

Going Further

  • Cilium health monitoring: Hubble runs as part of Cilium. If Cilium itself is unhealthy (BPF map exhaustion, kernel version incompatibility), Hubble stops capturing data. Add a Vigilmon check for https://cilium-health.example.com/healthz or cilium status — Hubble health is a superset of Cilium health.
  • Prometheus integration: Hubble Relay exports Prometheus metrics at :9966/metrics. You can add these metrics to your Prometheus/Mimir stack and set alerting rules there, then use Vigilmon as an independent external cross-check that confirms the alerts from your internal stack are firing correctly.
  • Service dependency mapping validation: Hubble's service map is a live view of inter-service communication. Export service dependency data via hubble observe --output json | jq 'select(.flow.Type=="L7")' and compare it against your expected dependency graph. Unexpected new connections or missing expected connections indicate service configuration drift.
  • Multi-cluster Hubble federation: In multi-cluster Cilium setups using ClusterMesh, Hubble can observe cross-cluster flows. Add per-cluster Relay health checks and peer-count checks — a ClusterMesh connectivity failure shows up as a reduced peer count in the affected cluster's Relay before it shows up as application errors.

With Vigilmon monitoring Hubble Relay health, peer connectivity, flow export rate, DNS tracking, and network policy drop rates, you have an independent external view of your Cilium network observability stack — so you know the moment Hubble stops seeing your network before your security and platform teams notice the gap.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles