How to Monitor Coil CNI with Vigilmon (HTTP + TCP + Alerts)

Coil is a Kubernetes CNI plugin built around BIRD (the BGP Internet Routing Daemon) that provides on-demand NAT and flexible IPAM for pods. Unlike CNI plugins that use a flat overlay network, Coil uses BGP route propagation between nodes to advertise pod CIDRs, then optionally applies SNAT via egress gateways for traffic leaving the cluster. It is designed for environments where pods need routable IPs that are exported via BGP to upstream network infrastructure, or where fine-grained egress NAT control is required — for example, multi-tenant platforms where each tenant namespace must appear to external systems as a different source IP. When Coil is working correctly, BIRD propagates pod routes between nodes, egress gateways translate outbound traffic to the correct external IP, and pods communicate with external services with predictable source addresses. When Coil fails — because the BIRD daemon crashes, a BGP session drops, or an egress gateway pod is unscheduled — pods silently lose egress connectivity or become unreachable while Kubernetes reports everything as healthy. Vigilmon provides the external view that Coil cannot self-report: continuous uptime and reachability checks from outside your cluster that detect BGP routing and egress NAT failures the moment they occur. You can add your first monitor for free, no credit card required.

Why Coil CNI needs external monitoring

BIRD daemon crash stops BGP route propagation and isolates pods on the affected node. Coil runs a BIRD BGP daemon on each node to advertise that node's pod CIDR to the rest of the cluster and to receive routes for other nodes' pod CIDRs. If BIRD crashes — due to a configuration syntax error after a ConfigMap update, a memory exhaustion event, or a kernel routing table conflict — the affected node stops advertising its pod routes. Pods on other nodes can no longer reach pods on the affected node, and pods on the affected node can no longer reach pods on other nodes. Kubernetes does not generate a node condition for a failed BGP session, and in-cluster health probes that run on the same node continue to pass. An external HTTP monitor probing a service backed by pods on the affected node detects this BGP isolation failure.
Egress gateway pod eviction silently breaks all outbound traffic that relies on SNAT. Coil's on-demand NAT feature routes egress traffic from pods through dedicated egress gateway pods, which apply SNAT so that external systems see a predictable source IP rather than the pod's internal IP. If an egress gateway pod is evicted — due to node pressure, an eviction policy, or a DaemonSet rollout — all pods that are configured to use that gateway lose their outbound NAT path. Outbound connections from those pods either fail with EHOSTUNREACH or appear to the external system from an unexpected source IP that breaks IP-based access controls. Kubernetes reports all pods as ready, but outbound traffic is silently broken. An external HTTP monitor with a known expected response from an IP-allowlisted endpoint detects this NAT failure.
BGP peer session drop after node network reconfiguration causes routing convergence delay. Coil's BIRD sessions peer with other nodes over the Kubernetes node network. If a node's management IP changes — due to a cloud provider maintenance event, an IPAM reconfiguration, or a NIC failover — BIRD's peering sessions drop and must re-establish. During the BGP convergence window (typically 30–120 seconds, but potentially longer if BIRD's hold time is high), routes to pods on the reconfigured node are withdrawn from all peers. Pod-to-pod traffic is dropped for the duration of the convergence delay. An external HTTP monitor with a check interval shorter than the convergence window detects this transient outage.
IPv4/IPv6 address pool exhaustion in IPAM prevents new pods from being scheduled. Coil manages pod IP assignment through its own IPAM backed by Kubernetes Custom Resources (AddressPool and AddressBlock). If all addresses in a pool are allocated — because pool blocks are too small for the workload, because a pod IP was leaked and not returned on deletion, or because rapid scale-out consumed the pool faster than new blocks could be allocated — new pods remain in ContainerCreating with a failed to allocate IP CNI error. Existing pods continue to run, masking the IPAM exhaustion from any check that probes existing pods. An external HTTP monitor targeting a canary deployment that is periodically redeployed detects this IPAM exhaustion.
Coil controller crash stops address block allocation and prevents IPAM from scaling. The Coil controller runs as a Deployment in the kube-system namespace and is responsible for carving address blocks out of the address pool and assigning them to nodes. If the controller crashes and enters CrashLoopBackOff — due to a CRD version mismatch after a Coil upgrade, a webhook TLS failure, or an unhandled exception — nodes cannot request new address blocks. Nodes with existing blocks continue to schedule pods, but once a node's current block is exhausted, all new pods on that node fail to get IPs. The cluster continues to appear healthy for existing workloads while silently losing the ability to scale. An external HTTP monitor with a synthetic scaling test catches the point where no additional replicas can be provisioned.

What you'll need

A running Kubernetes cluster with Coil CNI installed and BIRD running on nodes
The hostname or IP for a Coil-networked service endpoint, for example http://coil-workload.example.com
A free Vigilmon account — monitors start running in under a minute with no credit card required

Step 1: Expose Coil CNI workload health endpoints

Verify Coil is routing correctly and that service endpoints are reachable before configuring Vigilmon monitors.

# Check Coil controller is running
kubectl get pods -n kube-system -l app.kubernetes.io/name=coil
# Expected: 1 controller pod running

# Check Coil agent DaemonSet on all nodes
kubectl get daemonset -n kube-system coil-agent
kubectl get pods -n kube-system -l app.kubernetes.io/component=coil-agent -o wide

# Check BIRD is running on a node (via coil-agent pod)
kubectl exec -n kube-system <coil-agent-pod> -- birdc show status
# Expected:
# BIRD 2.x.x ready.
# Last reboot on ...
# Current server time is ...

# Check BGP session status
kubectl exec -n kube-system <coil-agent-pod> -- birdc show protocols
# Expected: BGP sessions to all peer nodes should be in "Established" state

# Check pod CIDR routes are installed
kubectl exec -n kube-system <coil-agent-pod> -- birdc show route
# Expected: routes for all node pod CIDRs

# Check address pools
kubectl get addresspools -A
# Expected: pools showing available and used counts

# Verify pod IP assignment
kubectl exec -n <namespace> <coil-pod> -- ip addr show eth0
# Expected: pod IP from the assigned block

# Test HTTP endpoint
curl -I http://coil-workload.example.com/health
# Expected:
# HTTP/1.1 200 OK

# Check egress NAT configuration (if using coil-egress)
kubectl get egressnats -A
kubectl get pods -n kube-system -l app.kubernetes.io/component=coil-egress -o wide

For clusters using Coil egress NAT, verify the egress gateway is working:

# Check what external IP egress pods appear with
kubectl exec -n <namespace> <coil-pod> -- curl -s https://api.ipify.org
# Expected: the SNAT IP configured in the EgressNAT resource, not the pod's internal IP

# Verify the EgressNAT resource
kubectl describe egressnat <egress-nat-name> -n <namespace>

Step 2: Set up HTTP monitoring in Vigilmon

Add an HTTP monitor for the Coil-networked service endpoint:

Log in to vigilmon.online and click Add Monitor.
Set Monitor type to HTTP.
Enter the URL: http://coil-workload.example.com/health
Set Check interval to 60 seconds and Expected status code to 200.
Click Save Monitor — Vigilmon will run the first check within a minute.

Add a monitor for egress NAT verification

If your deployment uses Coil egress NAT, monitor an endpoint that validates the source IP to detect egress gateway failures:

Click Add Monitor again.
Set Monitor type to HTTP.
Enter the URL of an external endpoint that reflects the source IP in its response (for example, a service that only accepts requests from the configured egress SNAT IP).
Set Expected status code to 200 — a 403 or connection refusal indicates the pod is reaching out from the wrong source IP, meaning egress NAT has failed.
Click Save Monitor.

Why external monitoring catches what internal checks miss

Coil failures are structurally invisible to Kubernetes. When a BGP session drops, routes are withdrawn and pod-to-pod traffic stops — but readiness probes running on the same node bypass the BGP routing path and continue to pass. When an egress gateway pod is evicted, outbound SNAT breaks — but in-cluster health probes do not make outbound connections and cannot detect the NAT failure. Vigilmon monitors from outside your infrastructure, using the same network paths and source IPs that external systems use. It detects BGP convergence outages, egress NAT failures, and IPAM-induced scaling failures that the entire in-cluster observability stack is architecturally incapable of detecting.

Step 3: Monitor Coil CNI's TCP port

A TCP monitor verifies connectivity to the Coil-networked workload at the transport layer, independent of BGP convergence state or BIRD routing status.

In Vigilmon, click Add Monitor.
Set Monitor type to TCP.
Enter Host: coil-workload.example.com and Port: 80 (or the application port).
Set Check interval to 60 seconds.
Click Save Monitor.

Find exposed ports for Coil-networked workloads:

# List services backed by Coil-networked pods
kubectl get svc -A -o wide

# Check pod CIDR routing on a node
ip route show | grep <pod-cidr>
# Expected: routes for each node's pod CIDR via BIRD-managed paths

# Verify TCP connectivity
nc -zv coil-workload.example.com 80

# Check what ports the Coil-networked pod is listening on
kubectl exec -n <namespace> <coil-pod> -- ss -tlnp
# Sample output:
# State  Recv-Q  Send-Q  Local Address:Port
# LISTEN 0       128     0.0.0.0:80         0.0.0.0:*   users:(("server",pid=1))

Step 4: Configure alert channels

Email alerts

In Vigilmon, go to Alert Channels and click Add Channel.
Choose Email and enter the address that should receive notifications (for example, platform-team@example.com or an on-call distribution list).
Assign this channel to all Coil CNI monitors created in Steps 2 and 3.
Click Save — Vigilmon will send a test email to confirm delivery.

Webhook alerts

For Slack, PagerDuty, or an incident management platform, add a webhook channel:

In Alert Channels, click Add Channel and choose Webhook.
Paste your endpoint URL.
Vigilmon will POST the following JSON payload when a monitor transitions to down:

{
  "monitor_id": "mon_coil_cni_health",
  "monitor_name": "Coil CNI workload /health",
  "status": "down",
  "previous_status": "up",
  "checked_at": "2026-07-12T09:14:22Z",
  "response_time_ms": null,
  "status_code": null,
  "error": "Connection refused",
  "url": "http://coil-workload.example.com/health"
}

Assign all Coil CNI monitors to this channel and click Save.

Runbook: diagnostics to run when an alert fires

#!/bin/bash
# Coil CNI alert runbook — run these in order when Vigilmon fires

echo "=== Pod status ==="
kubectl get pods -A | grep <coil-workload>

echo "=== Coil controller status ==="
kubectl get pods -n kube-system -l app.kubernetes.io/name=coil

echo "=== Coil agent DaemonSet ==="
kubectl get daemonset -n kube-system coil-agent
kubectl get pods -n kube-system -l app.kubernetes.io/component=coil-agent -o wide

echo "=== BIRD session status on coil-agent ==="
for POD in $(kubectl get pods -n kube-system -l app.kubernetes.io/component=coil-agent \
    -o jsonpath='{.items[*].metadata.name}'); do
  echo "--- $POD ---"
  kubectl exec -n kube-system $POD -- birdc show protocols 2>/dev/null | grep BGP
done

echo "=== Address pool availability ==="
kubectl get addresspools -A

echo "=== HTTP endpoint response ==="
curl -sv http://coil-workload.example.com/health 2>&1 | tail -20

echo "=== Egress gateway status ==="
kubectl get pods -n kube-system -l app.kubernetes.io/component=coil-egress -o wide 2>/dev/null

echo "=== EgressNAT resources ==="
kubectl get egressnats -A 2>/dev/null

echo "=== Service endpoints ==="
kubectl get endpoints -A

echo "=== Pod CIDR routes on node ==="
ip route show | grep "via" | grep -E "10\.|172\." | head -20

echo "=== Recent Coil events ==="
kubectl get events -A --sort-by='.lastTimestamp' | \
  grep -i "coil\|bird\|cni\|egress\|ipam" | tail -20

echo "=== Port connectivity ==="
nc -zv coil-workload.example.com 80

echo "=== Coil controller logs ==="
kubectl logs -n kube-system -l app.kubernetes.io/name=coil --tail=30 | \
  grep -i "error\|warn\|fail" | tail -20

Step 5: Create a public status page

A Vigilmon status page gives your platform engineering team and application operators a single URL to check Coil CNI network health during incidents — no cluster access or BIRD console required.

In Vigilmon, navigate to Status Pages and click New Status Page.
Give it a name such as Coil CNI Platform Network and choose a subdomain like coil-network-status.vigilmon.page.
Add all monitors created in this tutorial: the /health HTTP monitor, the egress NAT HTTP monitor (if applicable), and the TCP port monitor.
Optionally add a description explaining that this cluster uses Coil CNI with BGP route propagation via BIRD and that a monitor alert may indicate a BGP session failure or egress NAT gateway eviction.
Click Publish — the page is immediately accessible and updates in real time.

Share this URL in your platform engineering runbooks and incident response procedures. When pod-to-pod connectivity or outbound NAT issues are reported, the status page confirms whether the problem is at the Coil CNI layer before engineers begin BGP or IPAM investigation.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | http://coil-workload.example.com/health | HTTP | BGP route withdrawal, pod unreachable, CNI failure, IPAM exhaustion | | Egress IP validation endpoint | HTTP | Egress gateway eviction, SNAT failure, wrong source IP | | coil-workload.example.com:80 | TCP | Transport-level failure, port not reachable after BGP convergence |

# Quick-reference diagnostic one-liners when a Vigilmon alert fires

# 1. Is the Coil controller running?
kubectl get pods -n kube-system -l app.kubernetes.io/name=coil | grep -v Running

# 2. Are all Coil agent pods running?
kubectl get pods -n kube-system -l app.kubernetes.io/component=coil-agent | grep -v Running

# 3. Are BIRD BGP sessions established?
kubectl exec -n kube-system <coil-agent-pod> -- birdc show protocols | grep -E "BGP|Established"

# 4. Is the health endpoint responding?
curl -sf http://coil-workload.example.com/health || echo "Health check FAILED"

# 5. Are address pools still have available IPs?
kubectl get addresspools -A | grep -v "0 free"

# 6. Is any egress gateway pod missing?
kubectl get pods -n kube-system -l app.kubernetes.io/component=coil-egress

# 7. Is port 80 open?
nc -zv coil-workload.example.com 80 && echo "Port open" || echo "Port CLOSED"

# 8. Are pod CIDR routes present?
ip route show | grep "10.244" | wc -l

What's next

Heartbeat monitoring for BGP session health — Configure a Kubernetes CronJob that runs on each node, queries birdc show protocols via the Coil agent, verifies that all expected BGP sessions are in Established state, and POSTs a heartbeat to a Vigilmon dead-man's-switch endpoint only when all sessions are healthy. If any BGP session drops, the heartbeat stops and Vigilmon fires an alert before pod-to-pod connectivity failures become visible to users.
SSL certificate monitoring — If your Coil-networked workloads expose HTTPS endpoints, add a Vigilmon SSL monitor to alert before TLS certificate expiry causes connection rejections that can be confused with BGP routing failures during incident response.
Egress NAT source IP monitoring — Use Vigilmon to monitor an external endpoint that records and validates the source IP of incoming requests. Configure the external endpoint to return a non-200 status when the source IP does not match the expected Coil egress SNAT address. This creates a continuous, automated check that your egress gateway is active and NATing traffic to the correct external IP — the earliest possible signal of egress gateway eviction or misconfiguration.

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.