How to Monitor kube-capacity with Vigilmon

kube-capacity is a kubectl plugin that provides a concise, human-readable table view of node and pod resource requests and limits across your entire Kubernetes cluster. Instead of piecing together information from kubectl describe node and kubectl top pod, kube-capacity shows CPU and memory requests, limits, and utilization in a single output, making it the go-to tool for capacity planning, cost optimization, and diagnosing why pods are pending due to insufficient resources.

In production environments, kube-capacity is commonly embedded in automated capacity reporting pipelines, CI gates that enforce resource limit policies, and on-call runbooks for diagnosing cluster saturation. When kube-capacity's dependencies fail — the Kubernetes metrics-server, the API server, or RBAC permissions — those pipelines produce empty reports or errors that are easy to miss if no one is actively watching. In this tutorial you'll set up uptime monitoring for the infrastructure kube-capacity depends on using Vigilmon — free tier, no credit card required.

Why kube-capacity's dependencies need external monitoring

kube-capacity aggregates data from two sources: the Kubernetes API server (for resource requests and limits) and metrics-server (for live utilization). Failures in either produce silent gaps:

• metrics-server unavailability — when metrics-server is down or not installed, kube-capacity returns resource requests and limits but no utilization data; reports that rely on utilization percentages produce zeros without indicating a data gap
• Kubernetes API server degradation — API server slowness causes kube-capacity calls in automated pipelines to time out, producing empty capacity reports at exactly the moments when capacity data is most needed
• RBAC permission drift — kube-capacity needs get and list on nodes, pods, and node metrics; when these permissions are removed, the plugin fails with a cryptic error that is swallowed by CI scripts checking only exit code zero
• metrics-server RBAC misconfiguration — if metrics-server's own RBAC bindings are accidentally removed, it continues running but returns 403 errors on all metric requests, causing kube-capacity to report utilization as zero
• Node not ready — capacity data for a not-ready node is absent from kube-capacity output; an automated report that compares reported capacity against expected capacity won't notice a node has silently dropped out

External monitoring from Vigilmon watches the metrics-server and Kubernetes API health endpoints kube-capacity depends on and alerts you when capacity data becomes unavailable.

What you'll need

• A Kubernetes cluster with kube-capacity installed (kubectl krew install resource-capacity)
• metrics-server deployed in your cluster
• An HTTP health endpoint that validates kube-capacity can retrieve complete data (see Step 1)
• A free Vigilmon account

Step 1: Deploy a kube-capacity health probe and status server

kube-capacity does not expose HTTP endpoints natively. Create a Deployment that continuously validates the data sources kube-capacity depends on and exposes the result as an HTTP status endpoint:

# kube-capacity-health.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-capacity-health
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-capacity-health-reader
rules:
  - apiGroups: [""]
    resources: ["nodes", "pods", "namespaces"]
    verbs: ["get", "list"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["nodes", "pods"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-capacity-health-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-capacity-health-reader
subjects:
  - kind: ServiceAccount
    name: kube-capacity-health
    namespace: observability
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-capacity-health
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-capacity-health
  template:
    metadata:
      labels:
        app: kube-capacity-health
    spec:
      serviceAccountName: kube-capacity-health
      containers:
        - name: checker
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Check 1: API server can list nodes
                NODE_LIST=$(kubectl get nodes -o name 2>&1)
                API_OK=$?
                NODE_COUNT=$(echo "$NODE_LIST" | grep -c "^node/" || true)

                # Check 2: metrics-server is available and returning data
                METRICS=$(kubectl top nodes 2>&1)
                METRICS_OK=$?

                # Check 3: pods can be listed (required for pod-level capacity)
                POD_COUNT=$(kubectl get pods --all-namespaces -o name 2>/dev/null | wc -l)

                if [ $API_OK -eq 0 ] && [ $METRICS_OK -eq 0 ] && [ "$NODE_COUNT" -gt 0 ]; then
                  echo "healthy nodes=$NODE_COUNT pods=$POD_COUNT metrics=ok" > /tmp/status
                elif [ $API_OK -eq 0 ] && [ $METRICS_OK -ne 0 ]; then
                  METRICS_ERR=$(echo "$METRICS" | head -1)
                  echo "degraded api=ok metrics=unavailable error=$METRICS_ERR" > /tmp/status
                else
                  API_ERR=$(echo "$NODE_LIST" | head -1)
                  echo "unhealthy api=unavailable error=$API_ERR" > /tmp/status
                fi
                sleep 30
              done
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if echo "$STATUS" | grep -q "^healthy"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                elif echo "$STATUS" | grep -q "^degraded"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: kube-capacity-health
  namespace: observability
spec:
  type: NodePort
  selector:
    app: kube-capacity-health
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30084

kubectl apply -f kube-capacity-health.yaml

# Verify the health endpoint
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:30084/
# healthy nodes=3 pods=47 metrics=ok

Step 2: Verify kube-capacity output and metrics-server health

Before enabling monitoring, validate that kube-capacity is returning complete data from all nodes:

#!/bin/bash
# validate-kube-capacity.sh

# Install kube-capacity if not present
kubectl krew install resource-capacity 2>/dev/null || true

echo "=== Node capacity overview ==="
kubectl resource-capacity --sort cpu.util 2>/dev/null || \
  echo "WARNING: metrics-server may be unavailable — utilization data absent"

echo ""
echo "=== Pod resource limits (top consumers) ==="
kubectl resource-capacity --pods --sort cpu.request | head -20

echo ""
echo "=== metrics-server health check ==="
kubectl get deployment metrics-server -n kube-system
kubectl top nodes 2>/dev/null || echo "FAIL: kubectl top nodes failed — metrics-server unavailable"

echo ""
echo "=== Nodes not ready ==="
kubectl get nodes | grep -v "Ready" | grep -v "NAME" | \
  awk '{print "NOT READY:", $1, $2}' || echo "All nodes ready"

echo ""
echo "=== Namespaces with no resource limits set ==="
for NS in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  PODS_NO_LIMITS=$(kubectl get pods -n "$NS" -o jsonpath='{range .items[*]}{.spec.containers[*].resources.limits}{"\n"}{end}' 2>/dev/null | grep -c "^$" || true)
  if [ "$PODS_NO_LIMITS" -gt 0 ]; then
    echo "  $NS: $PODS_NO_LIMITS pods without limits"
  fi
done

chmod +x validate-kube-capacity.sh
./validate-kube-capacity.sh

This confirms both the API data path and the metrics-server data path are working before you rely on them in automated capacity reports.

Step 3: Set up HTTP monitoring in Vigilmon

With the health endpoint running, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add monitors for your endpoints:

| Monitor name | URL | Expected status | |---|---|---| | kube-capacity health | http://your-node-ip:30084/ | 200 | | metrics-server health | https://your-api-server:6443/apis/metrics.k8s.io/v1beta1 | 200 | | Kubernetes API server livez | https://your-api-server:6443/livez | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match healthy in the response body for the kube-capacity health endpoint
6. For Kubernetes API endpoints, configure the CA certificate under TLS settings if you use a private CA
7. Save each monitor

The metrics-server API endpoint (/apis/metrics.k8s.io/v1beta1) returns a list of resources when healthy. A 503 here means kube-capacity utilization data will be unavailable for all automated reports.

Step 4: Monitor the kube-capacity health TCP port

Add a TCP monitor to detect networking issues that affect capacity data access:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your node IP or Ingress hostname and port 30084
4. Save the monitor

A TCP failure while the pod appears Running typically indicates a NodePort rule was removed or a network policy was applied that blocks health check access.

Step 5: Configure alert channels

When capacity data is unavailable, on-call engineers cannot diagnose why pods are pending, capacity planning reports are silently wrong, and resource limit enforcement CI gates may fail or produce false negatives. Alert immediately.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your platform engineering or SRE on-call address
3. Assign the channel to all kube-capacity monitors

Webhook alerts for incident routing

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. The payload Vigilmon sends:

{
  "monitor_name": "kube-capacity health",
  "status": "down",
  "url": "http://your-node-ip:30084/",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Wire this alert into a runbook that checks metrics-server availability and verifies API server reachability, since a kube-capacity failure always points to one of these two sources.

Step 6: Correlate Vigilmon alerts with kube-capacity diagnostics

When you receive a kube-capacity health downtime alert, run these checks:

# 1. Check metrics-server deployment health
kubectl get deployment metrics-server -n kube-system
kubectl rollout status deployment/metrics-server -n kube-system

# 2. Test metrics-server directly
kubectl top nodes
kubectl top pods --all-namespaces | sort -k3 -rn | head -20

# 3. Check metrics-server logs for errors
kubectl logs -n kube-system -l k8s-app=metrics-server --tail=30

# 4. Verify metrics-server RBAC is intact
kubectl get clusterrolebinding metrics-server:system:auth-delegator
kubectl get rolebinding -n kube-system metrics-server-auth-reader

# 5. Check API aggregation layer for the metrics API
kubectl get apiservice v1beta1.metrics.k8s.io -o yaml | grep -A5 status

# 6. Verify node listing still works (API server check)
kubectl get nodes -o wide

# 7. Check RBAC permissions for the health service account
kubectl auth can-i list nodes --as=system:serviceaccount:observability:kube-capacity-health
kubectl auth can-i list pods --as=system:serviceaccount:observability:kube-capacity-health -A
kubectl auth can-i get nodes.metrics.k8s.io --as=system:serviceaccount:observability:kube-capacity-health -A

# 8. Check the health checker pod logs for specific errors
kubectl logs -n observability -l app=kube-capacity-health -c checker --tail=20

# 9. Run kube-capacity directly to reproduce the failure
kubectl resource-capacity 2>&1
kubectl resource-capacity --pods 2>&1 | head -20

# 10. Check if specific nodes are missing from capacity output
kubectl resource-capacity | awk '{print $1}' > /tmp/capacity-nodes.txt
kubectl get nodes -o name | cut -d/ -f2 > /tmp/all-nodes.txt
diff /tmp/capacity-nodes.txt /tmp/all-nodes.txt

If Vigilmon shows degraded (HTTP 200 with metrics=unavailable), metrics-server is the problem but the API server is fine — restart metrics-server first (kubectl rollout restart deployment/metrics-server -n kube-system) before investigating RBAC.

If Vigilmon shows a 503 (unhealthy api=unavailable), the API server itself is unreachable from the observability namespace — check network policies and API server health directly.

Step 7: Create a status page for cluster capacity data

Capacity data is used by engineers, product managers, and finance teams for resource planning. Make its availability visible to all stakeholders:

1. Go to Status Pages → New Status Page
2. Name it: "Cluster Capacity Data"
3. Add your monitors: kube-capacity health, metrics-server health, Kubernetes API server livez
4. Share the URL with the platform engineering, finance, and product teams

This lets any stakeholder confirm whether reported capacity numbers are live or stale before making scaling decisions.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on health endpoint | API server issues, metrics-server failures, RBAC permission loss | | HTTP monitor on metrics-server API | metrics-server unavailability (utilization data gone) | | HTTP monitor on Kubernetes livez | API server degradation before capacity retrieval fails | | TCP monitor on health port | Networking issues blocking capacity data access | | Email + webhook alert channels | Immediate notification when capacity data becomes unavailable | | Status page | Stakeholder visibility into whether capacity data is live or stale |

kube-capacity turns complex resource accounting into a single table — but only when its underlying data sources are healthy. External monitoring is the only reliable way to know when your capacity data has silently become empty or stale.

Get started at vigilmon.online — free tier, no credit card required.