How to Monitor Kube-VIP with Vigilmon (VIP Reachability, Leader Election, Failover Alerts)

Kube-VIP gives bare-metal and on-premises Kubernetes clusters the same LoadBalancer VIP experience that cloud providers deliver automatically. It runs as a DaemonSet or static pod on your control plane nodes, elects a leader via ARP or BGP, and assigns a virtual IP that floats between nodes when failures occur.

That floating VIP is both Kube-VIP's greatest strength and its most critical failure point. When the leader election breaks, when ARP announcements stop reaching switches, or when the VIP stops responding during a failover — your entire cluster's external access can vanish in seconds. In this tutorial you'll set up comprehensive monitoring for Kube-VIP using Vigilmon, catching every category of failure before users notice.

Why Kube-VIP needs external monitoring

Kube-VIP is typically monitored through Prometheus metrics scraped from inside the cluster. But internal metrics have a fundamental blind spot: if the VIP itself is unreachable, the monitoring stack may be unreachable too.

External monitoring fills the gaps Prometheus can't:

VIP unreachable after failover — a node goes down, Kube-VIP re-elects a leader on a different node, but ARP propagation fails and switches still send traffic to the dead node's MAC address. Pods show healthy; VIP doesn't respond.
Leader election stalls — the RAFT or ARP-based leader election enters a split-brain state. No node believes it's the leader. The VIP address exists in DNS but is unbound.
Gratuitous ARP blocked — a switch ACL or VLAN configuration change starts dropping the gratuitous ARP packets Kube-VIP sends on leader change. The VIP migrates logically but not physically.
API server VIP drifts — Kube-VIP managing the control plane VIP loses the election while the API server is healthy on individual nodes. kubectl works from inside the cluster; external access is broken.

Each of these scenarios produces the same observable symptom from outside: the VIP stops responding. Vigilmon catches all of them with a single HTTP or TCP check.

What you'll need

A Kubernetes cluster running Kube-VIP (v0.5+ recommended)
Kube-VIP configured in ARP or BGP mode with a VIP assigned
A free Vigilmon account

Step 1: Verify your Kube-VIP health endpoint

Kube-VIP exposes a Prometheus-compatible metrics and health endpoint. By default it listens on port 2112:

# From a node that can reach the VIP
curl http://<kube-vip-VIP>:2112/metrics | head -20

# Or check the health path if your version exposes it
curl http://<kube-vip-node-ip>:2112/metrics

To confirm Kube-VIP is serving on its assigned VIP, check the DaemonSet status:

kubectl get daemonset kube-vip-ds -n kube-system
kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-vip -o wide

Identify the current leader — it will be the pod with the VIP bound:

# The leader node will have the VIP assigned on its network interface
# Run this on each control plane node to find it
ip addr show | grep <your-vip-address>

Make note of your VIP address and the port your services use (commonly 6443 for the API server VIP, or 80/443 for service LoadBalancer VIPs).

Step 2: Monitor VIP reachability with Vigilmon HTTP check

The most critical check is whether the VIP responds at all. If Kube-VIP is managing your Kubernetes API server VIP, monitor it directly:

Log in to vigilmon.online and go to Monitors → New Monitor
Choose HTTP / HTTPS
Set the URL: https://<your-vip>:6443/healthz (for API server VIP) or http://<your-vip>/health (for service VIP)
Set check interval: 30 seconds — VIP failovers should be caught fast
Under Expected response, set status code 200
Save as "Kube-VIP API Server VIP"

For the API server's /healthz endpoint, the expected response body should contain ok. Add a body check:

Response body contains: ok

Vigilmon probes this from multiple geographic regions. A failed probe from one region but not others may indicate a routing asymmetry; consensus failure across all regions confirms the VIP is truly unreachable.

Step 3: Add a TCP reachability check

HTTP checks confirm the application layer responds. TCP checks confirm the VIP port is reachable at the network layer — this catches cases where the VIP is bound but the application process has crashed or its port is not listening:

Go to Monitors → New Monitor
Choose TCP Port
Enter your VIP address and the port: <your-vip> / 6443
Set check interval: 30 seconds
Save as "Kube-VIP TCP Port 6443"

If you manage multiple LoadBalancer services through Kube-VIP, add one TCP monitor per exposed port:

| Monitor Name | Host | Port | |---|---|---| | Kube-VIP API Server | <api-vip> | 6443 | | Kube-VIP HTTPS Services | <lb-vip> | 443 | | Kube-VIP HTTP Services | <lb-vip> | 80 |

Step 4: Monitor the metrics endpoint on each control plane node

Kube-VIP's metrics endpoint exposes leader election state and ARP/BGP status. Monitor it on each control plane node individually — not just the VIP — so you can detect when a node's Kube-VIP pod is unhealthy even if another node holds the VIP:

# Get node IPs
kubectl get nodes -o wide | grep control-plane

For each control plane node, add a monitor:

Go to Monitors → New Monitor
Choose HTTP / HTTPS
URL: http://<control-plane-node-ip>:2112/metrics
Check interval: 1 minute
Expected status code: 200
Save as "Kube-VIP Node 1 Metrics", "Kube-VIP Node 2 Metrics", etc.

If any node's metrics endpoint stops responding, its Kube-VIP pod has crashed — even if the VIP is currently held by a different node, you've lost redundancy and the next failure will cause a leader election with a reduced node pool.

Step 5: Configure failover alerting

Kube-VIP failovers are expected events — they happen when nodes are rebooted for maintenance or fail unexpectedly. The goal is to distinguish between:

Clean failover — VIP moves to another node and comes back up within seconds (normal maintenance)
Stuck failover — VIP goes unreachable and stays unreachable for more than 30–60 seconds (incident)

Configure Vigilmon's confirmation threshold to avoid alerting on clean failovers:

Open your "Kube-VIP API Server VIP" monitor settings
Under Confirmation, set Confirm outage after: 2 consecutive failures
With a 30-second check interval, this means you alert after ~60 seconds of unreachability

This window is long enough to skip a clean Kube-VIP failover (typically 5–15 seconds) but short enough to catch stuck elections.

Alert channels

Configure alert escalation for VIP failures, which can take down your entire cluster's external access:

Go to Alert Channels → Add Channel → Email
Add your on-call team email
Go to Alert Channels → Add Channel → Webhook
Add your PagerDuty or Opsgenie webhook URL

Assign both channels to all Kube-VIP monitors. For the API server VIP, treat any confirmed outage as P1 — your cluster is externally unreachable.

Step 6: Correlate Vigilmon alerts with Kube-VIP state

When a Vigilmon alert fires for a Kube-VIP monitor, here's the runbook:

# 1. Check which node currently holds the leader role
kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-vip -o wide

# 2. Check Kube-VIP logs on each node
kubectl logs -n kube-system -l app.kubernetes.io/name=kube-vip --since=5m

# 3. Check if the VIP is bound on any node
# Run on each control plane node:
ip addr show | grep <your-vip>

# 4. Verify ARP tables on nodes
arp -n | grep <your-vip>

# 5. Check recent Kubernetes events for Kube-VIP
kubectl get events -n kube-system --field-selector reason=LeaderElection --sort-by=.lastTimestamp

If the VIP is bound on a node but still unreachable from external networks, the problem is likely ARP propagation. Force a gratuitous ARP announcement:

# SSH into the leader node and send a gratuitous ARP
arping -U -I <interface> <your-vip> -c 3

Monitoring summary

| Monitor | Type | Interval | What it catches | |---|---|---|---| | Kube-VIP API Server VIP /healthz | HTTP | 30s | VIP unreachable, API server down | | Kube-VIP VIP TCP 6443 | TCP | 30s | Port-level reachability loss | | Kube-VIP Node 1 Metrics | HTTP | 60s | Pod crash on node 1 | | Kube-VIP Node 2 Metrics | HTTP | 60s | Pod crash on node 2 | | Kube-VIP Node 3 Metrics | HTTP | 60s | Pod crash on node 3 |

What's next

SSL certificate monitoring — if your Kube-VIP API server VIP uses TLS, Vigilmon monitors certificate expiry and alerts you before your cluster becomes inaccessible due to a lapsed cert
Status page — publish a Vigilmon status page for your cluster so that platform engineering teams can check cluster availability without needing kubectl access
Heartbeat monitoring — if you run automated Kube-VIP configuration scripts or leader health scripts as CronJobs, Vigilmon's heartbeat feature alerts you when they stop checking in

Start monitoring your Kube-VIP infrastructure for free at vigilmon.online — setup takes under two minutes.