How to Monitor Whereabouts with Vigilmon (HTTP + TCP + Alerts)

Whereabouts is a Kubernetes IPAM (IP Address Management) plugin that allocates IP addresses for pods across the entire cluster without requiring a central DHCP server — using a distributed datastore-backed approach that works with any CNI plugin. Every pod scheduled anywhere in the cluster depends on Whereabouts to receive a routable IP address; if the Whereabouts IPAM daemon becomes unavailable or its datastore backend becomes unreachable, new pods cannot start and rescheduled pods cannot obtain IPs, causing scheduling failures cluster-wide. Vigilmon adds the external perspective that Whereabouts cannot provide for itself: continuous uptime checks from outside your infrastructure that keep watching even when the IPAM component is the failure point. You can add your first monitor for free, no credit card required.

Why Whereabouts needs external monitoring

IPAM daemon unavailability blocks all new pod scheduling. Whereabouts runs as a binary invoked by the kubelet via the CNI interface on each node. If the Whereabouts binary is missing, has wrong permissions, or if the node-local daemon (when running in daemonset mode) is crashed or unresponsive, every pod scheduled on that node fails at the network allocation phase with a Failed to allocate address error. The pod enters ContainerCreating state indefinitely. Kubernetes does not surface a clear node-level alert for this — only individual pod events indicate the failure. An external HTTP monitor targeting the Whereabouts health endpoint detects the daemon unavailability before a queue of unschedulable pods builds up.
Datastore connectivity loss causes IP allocation to freeze. Whereabouts stores its IP allocation state in either a Kubernetes CRD (the IPPool resource) or an etcd cluster. If the Whereabouts daemon loses connectivity to the Kubernetes API server or etcd — due to a network partition, expired service account credentials, or an RBAC change — it cannot write new allocations or release old ones. New pods hang indefinitely waiting for IP allocation, and released IPs from terminated pods are never reclaimed. The IP pool for the affected subnets gradually exhausts. An external monitor that checks Whereabouts' health endpoint detects the datastore connectivity failure because the health check probes the backend connection.
IP pool exhaustion silently blocks pod creation at capacity limits. Whereabouts allocates IPs from a configured CIDR range per subnet. As the cluster grows or as pods are recycled during rolling deployments, the pool can become temporarily exhausted — especially if IP release is delayed by a crash loop or a stale network namespace left behind after an ungraceful node shutdown. When the pool is exhausted, new pod creation fails silently from an operator perspective: Kubernetes marks the pod as Pending with a CNI error in the events, but there is no cluster-level alert. The Whereabouts health and metrics endpoints expose pool utilization, and an external Vigilmon monitor targeting these endpoints can surface high utilization before scheduling failures begin.
Overlapping allocation reconciliation failures cause IP conflicts. Whereabouts includes a reconciler that periodically scans for stale IP allocations — IPs assigned to pods that no longer exist — and reclaims them. If the reconciler fails to run due to a permission error, a missing CRD, or a bug in the reconciliation logic, stale allocations accumulate over time. Eventually, Whereabouts attempts to allocate an IP that is still reserved for a terminated pod but is now also running in a new pod — creating an IP conflict. IP conflicts cause routing failures and can cause both pods to lose network connectivity intermittently. An external monitor that tracks the Whereabouts reconciler health metric detects reconciliation failures before IP conflicts begin manifesting as application errors.
Node-local cache corruption causes IP leaks after restarts. Whereabouts maintains a node-local cache of IP allocations to reduce API server calls during normal operation. If the cache is corrupted by an unclean shutdown, a storage failure, or an abrupt container restart, Whereabouts may allocate IPs from its in-memory state that conflict with allocations recorded in the shared datastore. These conflicts are typically detected only when a second pod attempts to claim the same IP, at which point both pods may experience connectivity issues. An external monitor on the Whereabouts health endpoint that fires after daemon restarts gives you an early indicator that cache re-sync has completed successfully.

What you'll need

A running Whereabouts installation with its health or metrics endpoint accessible
The public hostname or IP for your Whereabouts deployment, for example https://whereabouts.example.com
A free Vigilmon account — monitors start running in under a minute with no credit card required

Step 1: Expose Whereabouts's health endpoint

Whereabouts (when running as a DaemonSet with its reconciler) exposes a health endpoint. Verify it is reachable before configuring Vigilmon monitors.

# Check Whereabouts DaemonSet status
kubectl get daemonset -n kube-system whereabouts

# Check pod status across nodes
kubectl get pods -n kube-system -l app=whereabouts -o wide

# View Whereabouts logs for IPAM activity
kubectl logs -n kube-system -l app=whereabouts --tail=50

# Check the IPPool CRDs to verify the datastore is working
kubectl get ippools.whereabouts.cni.cncf.io -A

# Sample output:
# NAMESPACE   NAME                     RANGE             ALLOCATIONS
# default     whereabouts-pool-1       10.244.0.0/24     42

# Check for overlapping IP reservations (a key failure mode)
kubectl get overlappingrangeipreservations.whereabouts.cni.cncf.io -A

# Verify Whereabouts binary is present on a node
kubectl debug node/<node-name> -it --image=busybox -- ls /opt/cni/bin/whereabouts

If Whereabouts exposes a health endpoint via a Kubernetes Service:

# Find Whereabouts services
kubectl get svc -n kube-system | grep whereabouts

# Test the health endpoint
curl http://<whereabouts-service-ip>:8080/healthz

# Sample output:
# {"status": "ok", "datastore": "connected", "reconciler": "running"}

# Check metrics endpoint
curl http://<whereabouts-service-ip>:8080/metrics

# Verify IP pool utilization
kubectl get ippools.whereabouts.cni.cncf.io -A \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.range}{"\n"}{end}'

Step 2: Set up HTTP monitoring in Vigilmon

Add an HTTP monitor for the Whereabouts health endpoint:

Log in to vigilmon.online and click Add Monitor.
Set Monitor type to HTTP.
Enter the URL: http://whereabouts.example.com:8080/healthz
Set Check interval to 60 seconds and Expected status code to 200.
Click Save Monitor — Vigilmon will run the first check within a minute.

Add a monitor for the Whereabouts metrics endpoint

Whereabouts exports Prometheus metrics that reflect IP pool utilization, reconciliation status, and allocation rates:

Click Add Monitor again.
Set Monitor type to HTTP.
Enter the URL: http://whereabouts.example.com:8080/metrics
Set Expected status code to 200 and Timeout to 10 seconds.
Set a Response time threshold of 5000ms to detect datastore connectivity slowdowns before they cascade to pod scheduling failures.
Click Save Monitor.

Why external monitoring catches what internal checks miss

Whereabouts is an invisible prerequisite for pod scheduling. When Whereabouts fails, pods stall in ContainerCreating and engineers diagnose Kubernetes scheduling issues rather than IPAM failures — the root cause is buried in CNI error messages that are not surfaced as cluster-level alerts. Vigilmon operates entirely outside your infrastructure, probing Whereabouts health endpoints from external network vantage points on a fixed interval. It surfaces IPAM failures at the infrastructure layer before they cascade into scheduling failures across every workload on the affected nodes.

Step 3: Monitor Whereabouts's TCP port

A TCP monitor confirms that the Whereabouts health HTTP server is accepting connections at the network layer before any HTTP processing begins. This catches scenarios where the Whereabouts reconciler pod is stuck in initialization or has lost its binding to the health port.

In Vigilmon, click Add Monitor.
Set Monitor type to TCP.
Enter Host: whereabouts.example.com and Port: 8080.
Set Check interval to 60 seconds.
Click Save Monitor.

If Whereabouts exposes its metrics on a separate port:

# Find Whereabouts container ports
kubectl get daemonset -n kube-system whereabouts \
  -o jsonpath='{.spec.template.spec.containers[0].ports}' | jq .

# Sample output:
# [
#   {"containerPort": 8080, "name": "health",   "protocol": "TCP"},
#   {"containerPort": 9090, "name": "metrics",  "protocol": "TCP"}
# ]

Add a TCP monitor for port 9090 to independently confirm the metrics scrape port is open, which is used by Prometheus to collect IP pool utilization data.

Step 4: Configure alert channels

Email alerts

In Vigilmon, go to Alert Channels and click Add Channel.
Choose Email and enter the address that should receive notifications (for example, platform-team@example.com or an on-call distribution list).
Assign this channel to all Whereabouts monitors created in Steps 2 and 3.
Click Save — Vigilmon will send a test email to confirm delivery.

Webhook alerts

For Slack, PagerDuty, or an internal incident management system, add a webhook channel:

In Alert Channels, click Add Channel and choose Webhook.
Paste your endpoint URL (a Slack incoming webhook, PagerDuty Events API v2 endpoint, or custom URL).
Vigilmon will POST the following JSON payload when a monitor transitions to down:

{
  "monitor_id": "mon_whereabouts_healthz",
  "monitor_name": "Whereabouts /healthz",
  "status": "down",
  "previous_status": "up",
  "checked_at": "2026-07-12T09:14:22Z",
  "response_time_ms": null,
  "status_code": null,
  "error": "Connection refused",
  "url": "http://whereabouts.example.com:8080/healthz"
}

Assign all Whereabouts monitors to this channel and click Save.

Runbook: diagnostics to run when an alert fires

#!/bin/bash
# Whereabouts alert runbook — run these in order when Vigilmon fires

echo "=== DaemonSet status ==="
kubectl get daemonset -n kube-system whereabouts

echo "=== Pod status per node ==="
kubectl get pods -n kube-system -l app=whereabouts -o wide

echo "=== Recent events ==="
kubectl get events -n kube-system --sort-by='.lastTimestamp' | grep -i whereabouts | tail -20

echo "=== Health endpoint response ==="
curl -sv http://whereabouts.example.com:8080/healthz 2>&1 | tail -20

echo "=== IP pool status ==="
kubectl get ippools.whereabouts.cni.cncf.io -A -o wide

echo "=== Overlapping reservations ==="
kubectl get overlappingrangeipreservations.whereabouts.cni.cncf.io -A

echo "=== Pods stuck in ContainerCreating ==="
kubectl get pods -A | grep ContainerCreating | head -20

echo "=== CNI errors in recent pod events ==="
kubectl get events -A | grep -i "cni\|ipam\|whereabouts\|allocat" | tail -20

echo "=== Port connectivity ==="
nc -zv whereabouts.example.com 8080

echo "=== Whereabouts logs (last 100 lines) ==="
kubectl logs -n kube-system -l app=whereabouts --tail=100

echo "=== Test IP allocation manually ==="
kubectl run ip-test --image=busybox --rm --restart=Never \
  -- ip addr show eth0 2>&1 | head -10

Step 5: Create a public status page

A Vigilmon status page gives your platform engineering team and application developers a single URL to check Whereabouts health during incidents — no cluster access required.

In Vigilmon, navigate to Status Pages and click New Status Page.
Give it a name such as Whereabouts IPAM and choose a subdomain like whereabouts-status.vigilmon.page.
Add all monitors created in this tutorial: the /healthz HTTP monitor, the /metrics HTTP monitor, and the TCP port monitor.
Optionally add a description explaining that Whereabouts is the cluster IPAM plugin and that downtime causes pod scheduling failures.
Click Publish — the page is immediately accessible and updates in real time.

Share this URL in your team's incident response runbook and link it from your developer platform documentation. When engineers see pods stuck in ContainerCreating with CNI errors, the status page is the first place to check whether Whereabouts IPAM is healthy.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | http://whereabouts.example.com:8080/healthz | HTTP | Daemon crash, datastore connectivity loss, reconciler failure | | http://whereabouts.example.com:8080/metrics | HTTP | Metrics pipeline failure, pool exhaustion signals, allocation rate anomalies | | whereabouts.example.com:8080 | TCP | Network-level connectivity loss, health server crash, port binding failure |

# Quick-reference diagnostic one-liners when a Vigilmon alert fires

# 1. Are Whereabouts pods running on all nodes?
kubectl get pods -n kube-system -l app=whereabouts -o wide

# 2. Is the health endpoint responding?
curl -sf http://whereabouts.example.com:8080/healthz || echo "Health check FAILED"

# 3. Are any pods stuck in ContainerCreating?
kubectl get pods -A | grep ContainerCreating

# 4. What is the IP pool state?
kubectl get ippools.whereabouts.cni.cncf.io -A

# 5. Are there overlapping IP reservations?
kubectl get overlappingrangeipreservations.whereabouts.cni.cncf.io -A

# 6. Is port 8080 open?
nc -zv whereabouts.example.com 8080 && echo "Port open" || echo "Port CLOSED"

# 7. Check Whereabouts logs for errors
kubectl logs -n kube-system -l app=whereabouts --tail=30 | grep -i "error\|fail\|panic"

# 8. Test a pod IP allocation
kubectl run alloc-test --image=busybox --rm --restart=Never \
  -- ip addr show eth0 2>&1

What's next

Heartbeat monitoring for IP pool reconciliation — Configure a Kubernetes CronJob that creates and immediately deletes a test pod, verifying that Whereabouts allocates and releases an IP successfully, then POSTs a heartbeat to a Vigilmon dead-man's-switch endpoint. If the IPAM allocation pipeline stops processing, the heartbeat stops and Vigilmon fires an alert before pod scheduling failures cascade across the cluster.
Latency trend monitoring — Use Vigilmon's response time history to establish a latency baseline for the Whereabouts health endpoint. Sudden latency increases often indicate that the Whereabouts daemon is experiencing high load from many concurrent IP allocation requests during a large-scale deployment or cluster-wide rollout.
IP pool exhaustion alerting — Configure a Vigilmon HTTP monitor that checks the Whereabouts metrics endpoint and use a response body keyword check to alert when the IP pool utilization metric exceeds a configured threshold, giving you advance warning before new pod scheduling begins failing across affected nodes.

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.