tutorial

Monitoring Flannel CNI with Vigilmon

Flannel is the simple Kubernetes CNI network fabric — but when pod networking breaks, everything in the cluster stops communicating. Here's how to monitor Flannel health, pod connectivity, and network-dependent services with Vigilmon.

Flannel is one of the most widely deployed Kubernetes CNI (Container Network Interface) plugins, providing a simple overlay network that enables pod-to-pod communication across cluster nodes. Because Flannel operates below the application layer, its failures are silent at the service level: pods appear running, but they can't reach each other or external services. Vigilmon gives you external uptime monitoring that catches Flannel-related network failures through their observable effects — service connectivity loss, pod communication failures, and DNS resolution breakdowns.

What You'll Set Up

  • Node-level HTTP monitors to detect pod networking failures per node
  • Cross-node connectivity probes using a canary deployment
  • CoreDNS uptime monitoring (Flannel failures often manifest as DNS failures first)
  • Cron heartbeat for Flannel DaemonSet health verification

Prerequisites

  • Kubernetes cluster with Flannel CNI (any version)
  • kubectl access to the cluster
  • At least one externally reachable service for baseline connectivity testing
  • A free Vigilmon account

Step 1: Understand How Flannel Failures Surface

Flannel does not expose an HTTP health endpoint that Vigilmon can probe directly. Instead, Flannel failures surface through their effects on cluster networking:

  • Pod-to-pod failures: Services stop responding because backend pods can't receive traffic across nodes.
  • Pod-to-service failures: Cluster IP routing breaks down.
  • DNS failures: CoreDNS pods lose connectivity and stop resolving names.
  • Node isolation: A node's Flannel daemonset pod crashes, isolating all pods on that node.

The monitoring strategy is to watch observable symptoms — not Flannel internals — because by the time Flannel breaks, the most important thing is detecting which services have stopped working.

Step 2: Deploy a Canary Service Across Multiple Nodes

A canary deployment that spreads pods across nodes lets Vigilmon detect per-node Flannel failures:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: network-canary
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: network-canary
  template:
    metadata:
      labels:
        app: network-canary
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: network-canary
      containers:
      - name: echo
        image: ealen/echo-server:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: network-canary
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: network-canary
  ports:
  - port: 80
    targetPort: 80

The topologySpreadConstraints ensures canary pods land on different nodes. If Flannel fails on any node, that node's pod stops responding and the service becomes degraded.

# Get the external IP for the canary service
kubectl get service network-canary -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Step 3: Add Vigilmon Monitors for the Canary Service

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the canary service URL: http://CANARY_EXTERNAL_IP/.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

This monitor catches the scenario where Flannel fails on enough nodes to degrade or break the service — load balancer health checks remove failing pods but the service becomes unavailable if too many nodes are affected simultaneously.

Step 4: Monitor CoreDNS

Flannel failures on nodes that host CoreDNS pods cause immediate DNS resolution failures across the cluster. DNS is a prerequisite for almost all service-to-service communication, making CoreDNS a critical proxy metric for Flannel health.

Expose CoreDNS health externally:

apiVersion: v1
kind: Service
metadata:
  name: coredns-health
  namespace: kube-system
spec:
  type: LoadBalancer
  selector:
    k8s-app: kube-dns
  ports:
  - name: health
    port: 8080
    targetPort: 8080

Add a Vigilmon monitor for the CoreDNS health endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter the URL: http://COREDNS_HEALTH_IP:8080/health.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Click Save.

CoreDNS's /health endpoint returns OK as long as the CoreDNS process is running. A failure here means CoreDNS pods are down, which is frequently caused by Flannel-level network isolation on the nodes hosting them.

Step 5: Heartbeat Monitoring for Flannel DaemonSet Verification

The Flannel DaemonSet runs on every node. You can verify all pods are running by sending a heartbeat from a CronJob that checks DaemonSet status:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: flannel-health-check
  namespace: kube-system
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: flannel-health-sa
          containers:
          - name: checker
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - |
              DESIRED=$(kubectl get daemonset kube-flannel-ds -n kube-flannel \
                -o jsonpath='{.status.desiredNumberScheduled}')
              READY=$(kubectl get daemonset kube-flannel-ds -n kube-flannel \
                -o jsonpath='{.status.numberReady}')
              if [ "$DESIRED" = "$READY" ]; then
                curl -s https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID
              fi
          restartPolicy: OnFailure

Create the required RBAC for the CronJob:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel-health-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: flannel-health-reader
rules:
- apiGroups: ["apps"]
  resources: ["daemonsets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: flannel-health-reader-binding
subjects:
- kind: ServiceAccount
  name: flannel-health-sa
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: flannel-health-reader
  apiGroup: rbac.authorization.k8s.io

In Vigilmon, create the heartbeat monitor:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 5 minutes.
  3. Copy the heartbeat URL into the CronJob YAML above.
  4. Click Save.

If any Flannel pod is NotReady, the CronJob skips the heartbeat ping and Vigilmon alerts after the interval passes.

Step 6: Monitor VXLAN/Backend Connectivity

Flannel uses VXLAN (default), host-gw, or UDP for its overlay backend. VXLAN traffic runs on UDP port 8472. A failure at the VXLAN layer allows pods to appear running while blocking all cross-node communication.

To detect VXLAN failures, deploy a cross-node communication probe as a CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cross-node-probe
  namespace: default
spec:
  schedule: "*/3 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: probe
            image: curlimages/curl:latest
            command:
            - /bin/sh
            - -c
            - |
              # Probe the canary service (pods spread across nodes)
              if curl -sf --max-time 5 http://network-canary.default.svc.cluster.local/; then
                curl -s https://vigilmon.online/heartbeat/CROSS_NODE_HEARTBEAT_ID
              fi
          restartPolicy: OnFailure

This probes the canary service (from Step 2) via its ClusterIP from inside the cluster. If Flannel's VXLAN tunnel is broken between any two nodes, this internal probe fails and the heartbeat stops.

Step 7: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, PagerDuty, or a webhook.
  2. For the canary HTTP monitor, set Consecutive failures before alert to 2 to avoid false alerts from a single transient probe failure.
  3. For the CoreDNS health monitor, set it to 1 — DNS failures are immediate and broad in impact.
  4. For the heartbeat monitors, a single missed heartbeat should alert, since the CronJob is designed to be reliable.
  5. Route Flannel-related alerts to your infrastructure on-call team, not just application-level alerts.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Canary service HTTP | External canary service IP | Per-node Flannel failure, pod networking down | | CoreDNS health | :8080/health on CoreDNS pods | DNS failure from Flannel node isolation | | DaemonSet heartbeat | Heartbeat URL (5-min CronJob) | Flannel pod crash, NotReady pods | | Cross-node probe heartbeat | Heartbeat URL (3-min CronJob) | VXLAN tunnel failure, cross-node comm broken |

Flannel is a foundational layer that every workload in your cluster depends on, but it has no first-class health endpoint to probe. The monitoring strategy here watches Flannel's observable symptoms — canary service availability, CoreDNS health, and cross-node connectivity — and uses Vigilmon heartbeats driven by in-cluster CronJobs to surface failures that only manifest inside the overlay network. Together these monitors give you early warning of Flannel failures before a networking outage cascades across the entire cluster.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles