tutorial

How to Monitor SpiderPool IPAM and Kubernetes Services with Vigilmon

SpiderPool is a Kubernetes IPAM (IP Address Management) plugin designed for underlay and overlay network scenarios. It manages IP address allocation for CNI ...

SpiderPool is a Kubernetes IPAM (IP Address Management) plugin designed for underlay and overlay network scenarios. It manages IP address allocation for CNI plugins like Macvlan, IPVlan, SR-IOV, and OVS, giving pods direct access to underlay networks with real IP addresses rather than cluster-internal addresses. This makes pods reachable from the physical network — but it also introduces a new category of failure: IPAM pool exhaustion, IP conflicts, and binding failures that leave pods with no IP at all.

This tutorial covers how to monitor SpiderPool-managed services with Vigilmon so you catch IPAM and connectivity failures before they impact users.

Why SpiderPool environments need external monitoring

SpiderPool's underlay networking gives pods real network addresses, which means their failure modes are fundamentally different from overlay-network clusters:

  • IP pool exhaustion — when a SpiderPool IPPool runs out of available addresses, new pods fail to start with a CreateContainerError; existing pods appear healthy while the cluster can't scale
  • IP binding failure — if the CNI plugin (Macvlan, IPVlan, SR-IOV) fails to bind the allocated IP to the pod's network interface, the pod starts but is unreachable even though it has an assigned IP
  • Gateway misconfiguration — SpiderPool allows per-pool gateway configuration; a wrong gateway means pods in that pool can communicate locally but not reach the default route
  • IP conflict with physical network — if SpiderPool allocates an IP that's already in use on the physical network, both the pod and the physical host experience connectivity failures
  • SpiderCoordinator webhook failure — SpiderPool's coordinator mutating webhook modifies pod specs; if the webhook is unavailable, pod creation fails cluster-wide
  • Namespace binding violation — SpiderPool supports namespace-scoped pools; incorrect RBAC means a pod binds an IP from the wrong pool, causing routing anomalies

External monitoring from Vigilmon catches these as reachability failures even when Kubernetes reports healthy pods.

What you'll need

  • A Kubernetes cluster with SpiderPool installed and at least one IPPool configured
  • Pods/services assigned IPs from SpiderPool pools
  • A free Vigilmon account

Step 1: Verify SpiderPool allocation and pod reachability

# Check SpiderPool system pods
kubectl get pods -n kube-system -l app=spiderpool

# List IP pools and their utilization
kubectl get spiderippool -A
kubectl describe spiderippool <pool-name>

# Check IP endpoint allocations
kubectl get spiderendpoint -A

# Verify a pod received an IP from SpiderPool
POD_NAME=$(kubectl get pods -n production -l app=my-api -o jsonpath='{.items[0].metadata.name}')
kubectl describe pod $POD_NAME -n production | grep -E "IP:|SpiderPool"

# Test pod reachability from outside the cluster (underlay IP is directly routable)
POD_IP=$(kubectl get pod $POD_NAME -n production -o jsonpath='{.status.podIP}')
curl http://$POD_IP:8080/health

If the pod has an IP but curl fails, you have an IP binding failure — the IP was allocated but not successfully configured on the network interface.

Expose a service for monitoring

SpiderPool pods with underlay IPs can be reached directly via pod IPs from the physical network, but for production monitoring you typically want a Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: my-api
  namespace: production
spec:
  type: LoadBalancer
  selector:
    app: my-api
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
        app: my-api
      annotations:
        # SpiderPool annotation to select the IP pool
        ipam.spidernet.io/ippool: '{"ipv4": ["production-pool"]}'
    spec:
      containers:
        - name: api
          image: your-registry/my-api:latest
          ports:
            - containerPort: 8080

Step 2: Set up HTTP monitoring

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your service endpoint:
    • http://<SERVICE_EXTERNAL_IP>/health (for LoadBalancer services)
    • http://<POD_IP>:8080/health (for direct pod monitoring on underlay networks)
  4. Check interval: 1 minute
  5. Expected status: 200
  6. Save

For SpiderPool environments with direct pod access from the physical network, you can monitor individual pod IPs. This gives you finer-grained visibility into which pods are reachable and which have binding issues.

Monitor multiple pod IPs directly

# Get all pod IPs for a deployment
kubectl get pods -n production -l app=my-api \
  -o jsonpath='{range .items[*]}{.metadata.name}: {.status.podIP}{"\n"}{end}'

Create one HTTP monitor per pod IP. A single pod going unreachable while others stay up points to a SpiderPool IP binding failure on that specific pod.

Step 3: Monitor TCP connectivity

TCP monitoring validates that the port is open at the network layer, independent of the HTTP application:

  1. Go to Monitors → New Monitor → TCP Port
  2. Enter the service IP or pod IP
  3. Port: your application port
  4. Interval: 1 minute
  5. Save

For SR-IOV-accelerated pods where the kernel bypasses the standard TCP stack, TCP monitoring catches failures that HTTP-level checks might not surface immediately.

Step 4: Monitor SpiderPool component health

SpiderPool has several critical components. A webhook failure is particularly impactful because it blocks all pod creation:

# Check the SpiderPool IPAM plugin pod
kubectl get pods -n kube-system -l app.kubernetes.io/component=spiderpool-agent

# Check the controller
kubectl get pods -n kube-system -l app.kubernetes.io/component=spiderpool-controller

# Check for webhook availability (this is critical — if down, no pods can start)
kubectl get mutatingwebhookconfigurations | grep spiderpool
kubectl get validatingwebhookconfigurations | grep spiderpool

# Check IP pool utilization to catch exhaustion before it happens
kubectl get spiderippool -A -o jsonpath='{range .items[*]}{.metadata.name}: {.status.allocatedIPCount}/{.spec.ipVersion} total={.status.totalIPCount}{"\n"}{end}'

# Look for allocation failures in the controller logs
kubectl logs -n kube-system -l app.kubernetes.io/component=spiderpool-controller \
  --tail=100 | grep -i "error\|fail\|exhaust"

Set up a Vigilmon webhook alert that runs these checks automatically:

#!/bin/bash
# spiderpool-check.sh — triggered by Vigilmon downtime alert

echo "=== SpiderPool agent pods ==="
kubectl get pods -n kube-system -l app.kubernetes.io/component=spiderpool-agent

echo "=== IP Pool utilization ==="
kubectl get spiderippool -A

echo "=== Recent allocation errors ==="
kubectl logs -n kube-system \
  -l app.kubernetes.io/component=spiderpool-controller \
  --tail=50 | grep -i "error\|fail"

echo "=== Pods without IPs ==="
kubectl get pods -A --field-selector=status.podIP="" | grep -v "Completed\|Terminating"

Step 5: Heartbeat monitoring for IPAM-dependent workloads

Batch jobs that allocate IPs from SpiderPool pools on launch are particularly vulnerable to pool exhaustion. If the pool is full when a CronJob fires, the pod never starts — no Kubernetes event reaches your monitoring.

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name: batch-ipam-job
  3. Expected interval: match your CronJob schedule
  4. Copy the heartbeat URL

In your job:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: data-processor
spec:
  schedule: "0 */2 * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          annotations:
            ipam.spidernet.io/ippool: '{"ipv4": ["batch-pool"]}'
        spec:
          containers:
            - name: processor
              image: your-registry/processor:latest
              command:
                - /bin/sh
                - -c
                - |
                  ./process --source http://api.production.svc.cluster.local/data
                  # Signal success — if SpiderPool pool was exhausted, this line is never reached
                  curl -sf https://vigilmon.online/heartbeat/YOUR_TOKEN
          restartPolicy: OnFailure

Step 6: Configure alert channels

  1. Email — go to Alert Channels → Add Channel → Email
  2. Webhook — go to Alert Channels → Add Channel → Webhook

Pair your alerts with an auto-check that surfaces pool utilization in the alert message:

# webhook_handler.py — receives Vigilmon alerts and enriches them
import subprocess
import requests
import json

def handle_alert(alert_data):
    # Get pool status
    pool_status = subprocess.run(
        ['kubectl', 'get', 'spiderippool', '-A', '-o', 'json'],
        capture_output=True, text=True
    )
    pools = json.loads(pool_status.stdout)
    
    # Build enriched alert
    pool_info = []
    for item in pools.get('items', []):
        name = item['metadata']['name']
        allocated = item['status'].get('allocatedIPCount', 0)
        total = item['status'].get('totalIPCount', 0)
        pool_info.append(f"{name}: {allocated}/{total}")
    
    enriched = {
        **alert_data,
        'pool_utilization': pool_info
    }
    
    # Forward to Slack
    requests.post(SLACK_WEBHOOK, json={'text': json.dumps(enriched, indent=2)})

Step 7: Create a status page

  1. Go to Status Pages → New Status Page
  2. Name: Underlay Network Services
  3. Add monitors: service HTTP checks, individual pod TCP checks, heartbeats
  4. Group by: Production API, Batch Workers, Network Validation
  5. Publish

Monitoring coverage map

| Monitor | Type | Catches | |---------|------|---------| | http://<SERVICE_IP>/health | HTTP | App errors, service routing failure | | http://<POD_IP>:8080/health | HTTP | Individual pod IP binding failure | | <SERVICE_IP>:80 | TCP | Port-level connectivity, SR-IOV path | | batch-ipam-job | Heartbeat | IP pool exhaustion blocking job pod launch |

Diagnosing a SpiderPool connectivity failure

When Vigilmon alerts:

# 1. Are pods running and have IPs?
kubectl get pods -n production -l app=my-api -o wide

# 2. Check SpiderEndpoint for the failing pod
kubectl get spiderendpoint -n production

# 3. Is the IP pool exhausted?
kubectl describe spiderippool production-pool | grep -E "Allocated|Total|Available"

# 4. Is the webhook running? (critical — blocks all pod creation if down)
kubectl get pods -n kube-system | grep spiderpool

# 5. Check if IP is conflicting on the physical network
# (ping from a physical host on the same subnet)
ping -c 3 <POD_IP>

# 6. Check CNI plugin logs for binding errors
journalctl -u kubelet -n 100 | grep -i "CNI\|spiderpool\|macvlan\|ipvlan"

What's next

  • Pool utilization alerting — set up a Kubernetes CronJob that checks SpiderPool pool utilization percentage and pings a Vigilmon heartbeat; if utilization exceeds 80%, stop sending the ping to trigger an alert
  • Multi-pool monitoring — create separate Vigilmon monitor groups for each SpiderPool IP pool to correlate reachability issues with specific pool exhaustion
  • IPv6 dual-stack — SpiderPool supports dual-stack; create parallel IPv4 and IPv6 monitors to validate both stacks are reachable

Get started free at vigilmon.online — no credit card required, monitors are live in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles