tutorial

How to Monitor Kubernetes Cluster Autoscaler with Vigilmon

Kubernetes Cluster Autoscaler (CA) is invisible when it's working — pods schedule, nodes appear, everything hums. It becomes very visible when it's broken. A...

Kubernetes Cluster Autoscaler (CA) is invisible when it's working — pods schedule, nodes appear, everything hums. It becomes very visible when it's broken. A stuck scale-up leaves pods in Pending indefinitely. A runaway scale-down evicts workloads mid-request. A CA pod crash means no autoscaling at all.

This tutorial walks through monitoring the Cluster Autoscaler with Vigilmon so you catch failures in your autoscaling infrastructure before they become capacity crises.

Why CA needs external monitoring

Cluster Autoscaler exposes Prometheus metrics and logs decisions, but those signals share fate with the control plane. External monitoring adds coverage that internal observability misses:

  • CA pod is down — no autoscaling is happening; Pending pods pile up silently
  • Scale-up stuck — CA triggered a node group expansion that the cloud provider never fulfilled; pods stay Pending
  • Node group exhausted — all node groups hit maxSize; CA cannot schedule new pods
  • Scale-down too aggressive — CA is evicting nodes faster than your PodDisruptionBudgets intended
  • Health endpoint unreachable — the CA metrics/status endpoint itself is down; you have no visibility

What you'll need

  • Kubernetes cluster with Cluster Autoscaler deployed (v1.26+)
  • kubectl access to the kube-system namespace
  • A free Vigilmon account

Step 1: Expose the CA health and metrics endpoint

Cluster Autoscaler exposes an HTTP server on port 8085. By default it's not accessible outside the cluster. Create a Service and optionally an Ingress to expose the status endpoint:

# Confirm CA is running
kubectl get pods -n kube-system -l app=cluster-autoscaler

# Check current metrics (in-cluster)
kubectl exec -n kube-system \
  $(kubectl get pod -n kube-system -l app=cluster-autoscaler -o name | head -1) \
  -- wget -qO- http://localhost:8085/metrics | grep -E "cluster_autoscaler_(nodes|unschedulable)"

Expose the health endpoint externally:

# ca-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: cluster-autoscaler-health
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  selector:
    app: cluster-autoscaler
  ports:
    - name: health
      port: 8085
      targetPort: 8085
  type: LoadBalancer

kubectl apply -f ca-health-service.yaml
kubectl get svc cluster-autoscaler-health -n kube-system
# Note the EXTERNAL-IP — this is your Vigilmon target

The CA exposes a /health-check endpoint that returns 200 when the main loop is running:

curl http://<EXTERNAL-IP>:8085/health-check
# {"lastActivity":"2026-07-01T12:00:00Z"}

Step 2: Build a pending-pod depth monitor

CA's most critical failure mode is failing to scale up. Write a lightweight HTTP service that returns 503 when unschedulable pods have been pending for more than a threshold:

# pending_pod_monitor.py
import subprocess, json, time, http.server, threading

PENDING_THRESHOLD_SECONDS = 120  # alert if pods pending > 2 min
CHECK_INTERVAL = 30

state = {"pending_too_long": [], "last_check": 0}

def check_pending():
    while True:
        try:
            result = subprocess.run(
                ["kubectl", "get", "pods", "--all-namespaces",
                 "--field-selector=status.phase=Pending",
                 "-o", "json"],
                capture_output=True, text=True, timeout=15
            )
            pods = json.loads(result.stdout).get("items", [])
            now = time.time()
            long_pending = []
            for pod in pods:
                conditions = pod.get("status", {}).get("conditions", [])
                scheduled = next((c for c in conditions if c["type"] == "PodScheduled"), None)
                if scheduled and scheduled.get("status") == "False":
                    # Use creationTimestamp as proxy for pending start
                    from datetime import datetime, timezone
                    created = pod["metadata"]["creationTimestamp"]
                    ts = datetime.fromisoformat(created.replace("Z", "+00:00"))
                    age = (datetime.now(timezone.utc) - ts).total_seconds()
                    if age > PENDING_THRESHOLD_SECONDS:
                        long_pending.append({
                            "name": pod["metadata"]["name"],
                            "namespace": pod["metadata"]["namespace"],
                            "age_seconds": int(age)
                        })
            state["pending_too_long"] = long_pending
            state["last_check"] = now
        except Exception as e:
            state["pending_too_long"] = [{"error": str(e)}]
        time.sleep(CHECK_INTERVAL)

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/healthz":
            self.send_response(404); self.end_headers(); return
        body = json.dumps({
            "pending_too_long": state["pending_too_long"],
            "last_check": state["last_check"]
        }).encode()
        if state["pending_too_long"]:
            self.send_response(503)
        else:
            self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args): pass

threading.Thread(target=check_pending, daemon=True).start()
http.server.HTTPServer(("", 9091), Handler).serve_forever()

Deploy it in-cluster with appropriate RBAC:

# rbac-pending-monitor.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pending-monitor
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pending-pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: pending-monitor-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pending-pod-reader
subjects:
  - kind: ServiceAccount
    name: pending-monitor
    namespace: kube-system

Step 3: Monitor node group health

Check that CA's configured node groups are responsive:

# List node groups CA is managing (EKS example)
kubectl describe configmap cluster-autoscaler-status -n kube-system \
  | grep -A3 "NodeGroup"

Create a CA status parser that reads the cluster-autoscaler-status ConfigMap and exposes node group health:

kubectl get configmap cluster-autoscaler-status \
  -n kube-system \
  -o jsonpath='{.data.status}' | grep -E "(Health|ScaleUp|ScaleDown)"

A healthy output looks like:

Health:      Healthy (ready=3 unready=0 notStarted=0 longNotStarted=0 registered=3 ...)
ScaleUp:     NoActivity (ready=3 registered=3)
ScaleDown:   NoCandidates (candidates=0)

Any Unhealthy or stuck InProgress lasting more than 5 minutes warrants an alert.

Step 4: Set up Vigilmon monitors

In Vigilmon, create these monitors:

Monitor 1 — CA pod liveness

| Field | Value | |---|---| | Type | HTTP | | URL | http://<CA-EXTERNAL-IP>:8085/health-check | | Expected status | 200 | | Interval | 30 seconds | | Alert after | 1 failure |

Monitor 2 — Pending pod depth

| Field | Value | |---|---| | Type | HTTP | | URL | http://<pending-monitor-IP>:9091/healthz | | Expected status | 200 | | Keyword check | "pending_too_long":[] | | Interval | 60 seconds |

Monitor 3 — CA metrics endpoint reachability

| Field | Value | |---|---| | Type | HTTP | | URL | http://<CA-EXTERNAL-IP>:8085/metrics | | Expected status | 200 | | Keyword check | cluster_autoscaler | | Interval | 60 seconds |

Step 5: Key Prometheus metrics to watch alongside Vigilmon

If you have Prometheus in-cluster, these metrics complement Vigilmon's external view:

# Unschedulable pods right now
cluster_autoscaler_unschedulable_pods_count

# Nodes in each state
cluster_autoscaler_nodes_count{state="ready"}
cluster_autoscaler_nodes_count{state="notStarted"}

# Scale-up errors in the last 5 minutes
increase(cluster_autoscaler_scaled_up_nodes_total[5m])

# CA main loop duration (should be < 10s normally)
cluster_autoscaler_function_duration_seconds{function="main"}

Set Vigilmon's response-time alerting on the /metrics endpoint at > 5 seconds — a slow main loop is a leading indicator of CA getting stuck.

Step 6: Configure escalation policies

CA pod down            → P1 immediately (no autoscaling)
Pending pods > 2 min   → P1 (workloads waiting for capacity)
Metrics endpoint slow  → P2 (CA may be struggling)
Node group unhealthy   → P1 (cloud provider issue)

In Vigilmon, set 3 check locations minimum so a single region outage doesn't false-alert on CA health.

What to monitor — quick reference

| Signal | Vigilmon monitor | Alert | |---|---|---| | CA pod alive | HTTP /health-check 200 | Immediate P1 | | Pending pod queue | HTTP /healthz 200 | Immediate P1 | | Metrics endpoint | HTTP /metrics 200 + keyword | 2 failures | | Scale-up latency | Response time baseline | >2× baseline | | Node group capacity | External API + Vigilmon TCP | Cloud API unreachable |

Vigilmon for autoscaling infrastructure

Cluster Autoscaler is a control-plane component — when it fails, the failure is often silent until pod scheduling starts degrading. Vigilmon's external probes give you the first signal because they don't share fate with the cluster. A 30-second CA health-check means you know about autoscaling problems before your on-call engineer notices pods stuck in Pending.

Get started at vigilmon.online — free plan, no credit card, monitors live in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles