How to Monitor MetalLB with Vigilmon (Speaker Health, BGP Peers, Address Pools, IP Reachability)

MetalLB brings native LoadBalancer service support to bare-metal Kubernetes clusters — the feature that cloud providers deliver automatically through AWS ELB, GCP Cloud Load Balancing, or Azure Load Balancer. Without MetalLB, kubectl expose --type=LoadBalancer on bare metal produces services that sit in <pending> forever. With MetalLB, they get real IPs that external traffic can reach.

That IP allocation and announcement machinery is surprisingly easy to break — BGP peer sessions drop, address pools exhaust, speaker pods crash silently — and when it breaks, all new LoadBalancer services stop working while existing ones may lose their announcements. In this tutorial you'll set up thorough monitoring for MetalLB using Vigilmon to catch failures before they affect production traffic.

Why MetalLB needs external monitoring

MetalLB exposes Prometheus metrics, but scraping those metrics requires the monitoring stack to already be reachable — and if MetalLB fails, you may lose access to your monitoring cluster itself if it lives on the same infrastructure.

External monitoring from outside the cluster catches the failure modes that internal observability misses:

Speaker pod crash — a MetalLB speaker pod on one node crashes; that node stops announcing IPs for any services whose backend pods live on it. Services appear healthy; packets are dropped silently.
BGP session flap — a BGP peer session to your top-of-rack switch drops. MetalLB withdraws route announcements. Traffic destined for your service IPs has no route and is dropped at the router.
Address pool exhaustion — your configured IP pool runs out of addresses. New LoadBalancer services stay <pending>. Existing services are unaffected but scaling fails.
ARP mode announcement failure — in Layer 2 mode, gratuitous ARP announcements stop reaching the gateway because of a switch or VLAN change. Allocated IPs are unreachable even though MetalLB believes it is announcing them.
Webhook server failure — the MetalLB controller's webhook is unavailable. New service creation fails with admission webhook errors, breaking all kubectl apply operations that touch LoadBalancer services.

What you'll need

A Kubernetes cluster running MetalLB (v0.13+ with the new CRD-based configuration, or v0.12 with ConfigMap)
MetalLB's speaker DaemonSet and controller Deployment running
At least one LoadBalancer service with an assigned external IP
A free Vigilmon account

Step 1: Verify MetalLB is healthy and note your endpoints

Check speaker and controller status:

# Check all MetalLB pods
kubectl get pods -n metallb-system -o wide

# Expected output:
# controller-xxx   Running   metallb-system
# speaker-xxx      Running   node-1
# speaker-xxx      Running   node-2
# speaker-xxx      Running   node-3

Check speaker metrics availability (default port 7472):

# Get a speaker pod name
SPEAKER=$(kubectl get pods -n metallb-system -l component=speaker -o name | head -1)

# Port-forward to test
kubectl port-forward -n metallb-system $SPEAKER 7472:7472 &
curl http://localhost:7472/metrics | grep metallb
kill %1

List currently allocated LoadBalancer IPs:

kubectl get svc -A --field-selector spec.type=LoadBalancer

Note the external IPs — you'll monitor their reachability directly.

Step 2: Monitor allocated IP reachability

The most direct MetalLB health check is whether the IPs MetalLB has allocated are actually reachable. If BGP announcements or ARP fail, the IPs will be allocated but unreachable:

For each critical LoadBalancer service, add an HTTP monitor:

Log in to vigilmon.online and go to Monitors → New Monitor
Choose HTTP / HTTPS
URL: http://<allocated-external-ip>/health (or the relevant health path)
Check interval: 1 minute
Expected status code: 200
Save as "MetalLB Service — <service-name>"

For services without HTTP (e.g., a database exposed via LoadBalancer), use a TCP check instead:

Choose TCP Port
Enter the external IP and port
Save as "MetalLB TCP — <service-name>:<port>"

This catches the most common MetalLB failure: the IP is allocated, Kubernetes thinks the service is available, but network-level announcement has failed.

Step 3: Monitor the MetalLB speaker metrics endpoint

MetalLB speakers expose per-node metrics at port 7472. Monitoring each speaker's metrics endpoint tells you if a speaker pod has crashed, without waiting for an allocated IP to become unreachable.

First, expose the speaker metrics via a NodePort service so Vigilmon can reach them:

# metallb-speaker-metrics-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: metallb-speaker-metrics
  namespace: metallb-system
spec:
  type: NodePort
  selector:
    component: speaker
  ports:
    - name: metrics
      port: 7472
      targetPort: 7472
      nodePort: 30472
      protocol: TCP

kubectl apply -f metallb-speaker-metrics-nodeport.yaml

Then for each node, add a Vigilmon monitor:

Go to Monitors → New Monitor
Choose HTTP / HTTPS
URL: http://<node-ip>:30472/metrics
Check interval: 1 minute
Expected status code: 200
Response body contains: metallb_
Save as "MetalLB Speaker — node-1", "MetalLB Speaker — node-2", etc.

Step 4: Monitor the MetalLB controller webhook

The MetalLB controller runs an admission webhook that validates IPAddressPool and L2Advertisement/BGPAdvertisement resources. If the webhook is unavailable, all MetalLB CRD operations fail.

Check the webhook service:

kubectl get svc -n metallb-system webhook-service
# NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
# webhook-service   ClusterIP   10.96.50.100    <none>        443/TCP   14d

Expose it for external monitoring:

# metallb-controller-metrics-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: metallb-controller-metrics
  namespace: metallb-system
spec:
  type: NodePort
  selector:
    component: controller
  ports:
    - name: metrics
      port: 7472
      targetPort: 7472
      nodePort: 30473
      protocol: TCP

kubectl apply -f metallb-controller-metrics-nodeport.yaml

Add the monitor in Vigilmon:

Go to Monitors → New Monitor
Choose HTTP / HTTPS
URL: http://<any-node-ip>:30473/metrics
Check interval: 1 minute
Expected status code: 200
Save as "MetalLB Controller Metrics"

Step 5: Configure BGP peer status alerting

In BGP mode, MetalLB maintains sessions with your network's BGP routers. When a BGP session drops, MetalLB withdraws route announcements for all IPs it was advertising to that peer.

MetalLB's metrics expose BGP session state:

metallb_bgp_session_up{peer="<peer-ip>"} 1

A value of 0 means the BGP session is down. While you can't query Prometheus from Vigilmon directly, you can create a small health endpoint that wraps this check. Alternatively, monitor the BGP peer's router management interface if it's accessible, or set up a simple script that exposes a health endpoint based on MetalLB BGP metrics:

# Simple BGP health shim — run as a sidecar or external script
# Returns 200 if all BGP sessions are up, 503 if any are down
curl -s http://localhost:7472/metrics | \
  grep 'metallb_bgp_session_up' | \
  awk '{if ($NF == 0) exit 1}'

The most reliable approach for BGP monitoring is to directly test that allocated IPs are reachable from outside the network — which is exactly what the HTTP and TCP monitors from Step 2 do. If BGP sessions drop, Vigilmon will catch the IP becoming unreachable within one check interval.

Step 6: Configure alert channels and thresholds

MetalLB failures can be gradual (one speaker crashes, capacity is reduced) or sudden (BGP session drops, IPs become unreachable immediately). Configure alerts to reflect both types:

For IP reachability monitors (critical path)

Go to Alert Channels → Add Channel → Webhook
Enter your PagerDuty or Opsgenie URL
Assign to all IP reachability monitors
Set Confirmation to 1 consecutive failure — IP reachability failure should alert immediately

For speaker metrics monitors (degraded capacity)

Set Confirmation to 2 consecutive failures — speaker metrics may briefly fail during pod restarts
Route to your team Slack channel rather than pager — one speaker being down is urgent but not P1 if other speakers are healthy

Example alert routing

IP Reachability Down → PagerDuty P1
Speaker Pod Down → Slack #platform-alerts
Controller Metrics Down → Slack #platform-alerts

Monitoring summary

| Monitor | Type | Interval | Confirmation | What it catches | |---|---|---|---|---| | http://<lb-ip>/health | HTTP | 60s | 1 failure | BGP/ARP announcement failure | | http://<lb-ip2>/health | HTTP | 60s | 1 failure | Per-service IP reachability | | MetalLB Speaker — node-1 | HTTP | 60s | 2 failures | Speaker pod crash on node 1 | | MetalLB Speaker — node-2 | HTTP | 60s | 2 failures | Speaker pod crash on node 2 | | MetalLB Controller | HTTP | 60s | 2 failures | Controller pod crash, webhook unavailable |

Correlating Vigilmon alerts with MetalLB state

When an IP reachability alert fires:

# 1. Check all MetalLB pods
kubectl get pods -n metallb-system -o wide

# 2. Check speaker logs for ARP/BGP errors
kubectl logs -n metallb-system -l component=speaker --since=5m | grep -E "error|BGP|ARP"

# 3. Check controller logs
kubectl logs -n metallb-system -l component=controller --since=5m

# 4. Verify address pool status
kubectl get ipaddresspools -n metallb-system

# 5. Check advertisement status
kubectl get l2advertisements -n metallb-system
# or for BGP:
kubectl get bgpadvertisements -n metallb-system

# 6. Verify the service still has an external IP
kubectl get svc <service-name> -n <namespace>

What's next

SSL monitoring — if MetalLB-assigned IPs serve HTTPS, Vigilmon monitors TLS certificate expiry automatically
Status page — expose a Vigilmon status page showing the health of your MetalLB-backed services so platform and application teams have visibility without cluster access
Heartbeat monitors — monitor any automation scripts (IP pool management, BGP config sync) as heartbeats that must check in periodically

Set up MetalLB monitoring for free at vigilmon.online and catch networking failures before they reach your users.