tutorial

How to Monitor Karpenter with Vigilmon (Controller Health, Webhook Availability, Node Provisioning)

Karpenter is the node autoscaler that replaces the Kubernetes Cluster Autoscaler for many AWS (and increasingly Azure) environments. Where the Cluster Autosc...

Karpenter is the node autoscaler that replaces the Kubernetes Cluster Autoscaler for many AWS (and increasingly Azure) environments. Where the Cluster Autoscaler works at the node group level — scaling pre-defined groups up and down — Karpenter provisions individual EC2 instances directly, selecting the optimal instance type and capacity for each pending workload in real time.

This architectural difference makes Karpenter extremely powerful but also introduces new failure modes. When Karpenter stops working, pods can't be scheduled onto new nodes. Existing workloads keep running, but scale-out events fail silently: pending pods sit in Scheduling state while Karpenter's controller is broken or its webhook is refusing connections. In this tutorial you'll set up comprehensive Karpenter monitoring using Vigilmon to surface these failures immediately.

Why Karpenter needs external monitoring

Karpenter's failures are particularly dangerous because they're invisible under normal load. The system only breaks visibly when:

  1. A scale-out event is triggered (a new deployment, a traffic spike, a node failure)
  2. A NodeClaim is attempted but fails

Here are the failure categories that external monitoring catches before they become incidents:

  • Controller crash — the Karpenter controller pod fails. No new nodes are provisioned. Pods pile up in Pending. If you don't have alerts on pod-scheduling latency, this can go undetected for hours during low-traffic periods.
  • Webhook unavailable — Karpenter's admission webhook validates NodePool and EC2NodeClass resources. If the webhook pod is down, any kubectl apply that touches these resources fails. Infrastructure-as-code pipelines break. Operators can't change node configuration.
  • AWS API connectivity lost — Karpenter calls EC2 APIs to launch and terminate instances. If IAM permissions expire, API rate limits are hit, or network policies block egress to AWS APIs, node provisioning silently fails.
  • Metrics endpoint down — while less directly impactful, a failed metrics endpoint often signals a partially broken controller that's struggling but not fully crashed.
  • NodePool misconfiguration — a bad NodePool or EC2NodeClass update causes Karpenter to reject all provisioning requests. The controller runs but produces errors on every scheduling attempt.

What you'll need

  • A Kubernetes cluster with Karpenter installed (v0.33+ / v1.0+ recommended)
  • Karpenter running in its own namespace (karpenter by default)
  • AWS permissions for EC2 instance management (if using AWS)
  • A free Vigilmon account

Step 1: Verify Karpenter's health and metrics endpoints

Karpenter exposes health and metrics on port 8080 (metrics) and 8081 (health probes) by default:

# Verify Karpenter pods are running
kubectl get pods -n karpenter -o wide

# Check the deployment
kubectl describe deployment karpenter -n karpenter | grep -A10 "Ports"

Port-forward to test the endpoints:

KARPENTER_POD=$(kubectl get pods -n karpenter -l app.kubernetes.io/name=karpenter -o name | head -1)

# Test health endpoint
kubectl port-forward -n karpenter $KARPENTER_POD 8081:8081 &
curl http://localhost:8081/healthz
# Response: OK
kill %1

# Test metrics endpoint
kubectl port-forward -n karpenter $KARPENTER_POD 8080:8080 &
curl http://localhost:8080/metrics | grep karpenter | head -20
kill %1

Note the exact ports — they can differ if you've customized the Karpenter Helm chart. Check the chart values:

helm get values karpenter -n karpenter | grep -E "metricsPort|healthzPort"

Step 2: Expose health and metrics endpoints for external monitoring

Create NodePort services to make the Karpenter endpoints reachable from outside the cluster:

# karpenter-health-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: karpenter-health-external
  namespace: karpenter
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: karpenter
  ports:
    - name: healthz
      port: 8081
      targetPort: 8081
      nodePort: 30081
      protocol: TCP
    - name: metrics
      port: 8080
      targetPort: 8080
      nodePort: 30080
      protocol: TCP

kubectl apply -f karpenter-health-nodeport.yaml

Verify from outside:

curl http://<node-ip>:30081/healthz
# Expected: OK

curl http://<node-ip>:30080/metrics | grep karpenter_nodes

Step 3: Monitor the Karpenter controller health endpoint

The controller health endpoint is your primary signal that Karpenter is functioning:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://<node-ip>:30081/healthz
  4. Check interval: 1 minute
  5. Under Expected response:
    • Status code: 200
    • Response body contains: OK
  6. Save as "Karpenter Controller Health"

This catches: controller pod crash, OOMKilled, CrashLoopBackOff, failed liveness probes, and any state where the process can't serve health checks.

Step 4: Monitor the Karpenter webhook server

The admission webhook is a separate concern from the controller — it's what validates changes to your Karpenter NodePools and EC2NodeClasses. Karpenter's webhook runs on port 8443:

# Check webhook service
kubectl get svc -n karpenter | grep webhook

# Port-forward and test
kubectl port-forward -n karpenter $KARPENTER_POD 8443:8443 &
# The webhook uses TLS, so this will return a TLS error without a cert, but
# confirming the port is open is still a useful signal
curl -k https://localhost:8443/
kill %1

Expose the webhook port for TCP monitoring:

# karpenter-webhook-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: karpenter-webhook-external
  namespace: karpenter
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: karpenter
  ports:
    - name: webhook
      port: 8443
      targetPort: 8443
      nodePort: 30443
      protocol: TCP

kubectl apply -f karpenter-webhook-nodeport.yaml

Add a TCP monitor in Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Host: <node-ip>, Port: 30443
  4. Check interval: 1 minute
  5. Save as "Karpenter Webhook Server"

TCP monitoring confirms the webhook is accepting connections, even without validating the TLS certificate. A failed TCP check means the webhook process is down and Karpenter CRD operations will fail with admission errors.

Step 5: Monitor the metrics endpoint for provisioning signals

Karpenter's metrics endpoint exposes detailed signals about node provisioning activity. Use it as an application-layer health check:

  1. Go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://<node-ip>:30080/metrics
  4. Check interval: 2 minutes
  5. Expected status code: 200
  6. Response body contains: karpenter
  7. Save as "Karpenter Metrics Endpoint"

Key Karpenter metrics to watch for (via your internal Prometheus stack if available):

# Node provisioning attempts
karpenter_nodes_created_total

# Provisioning errors
karpenter_provisioner_scheduling_simulation_results_total

# NodeClaim creation latency
karpenter_nodeclaims_lifecycle_duration_seconds_bucket

# Disruption events
karpenter_disruption_actions_performed_total

While Vigilmon doesn't scrape Prometheus metrics directly, monitoring the metrics endpoint confirms the metrics subsystem is healthy — a broken metrics server often precedes or accompanies controller instability.

Step 6: Validate node provisioning with a canary workload

To confirm Karpenter is actually provisioning nodes (not just healthy), run a scheduled canary deployment that requires a new node:

# karpenter-canary-job.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: karpenter-provisioning-canary
  namespace: karpenter
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          nodeSelector:
            # Force scheduling on a Karpenter-managed node
            karpenter.sh/registered: "true"
          tolerations:
            - key: karpenter.sh/disruption
              operator: Exists
          containers:
            - name: canary
              image: busybox:1.36
              command: ["wget", "-qO-", "http://karpenter-health-external.karpenter:8081/healthz"]
          resources:
            requests:
              cpu: 100m
              memory: 64Mi

Pair this with a Vigilmon heartbeat monitor:

  1. Go to Monitors → New Monitor
  2. Choose Heartbeat
  3. Set expected interval: 20 minutes (the job runs every 15 minutes; alert if no heartbeat in 20)
  4. Save as "Karpenter Provisioning Canary"

Have the CronJob ping the Vigilmon heartbeat URL at the end of each successful run:

          containers:
            - name: canary
              image: curlimages/curl:8.4.0
              command:
                - sh
                - -c
                - |
                  # Check Karpenter health
                  curl -sf http://karpenter-health-external.karpenter:8081/healthz && \
                  # Report to Vigilmon heartbeat
                  curl -sf https://vigilmon.online/heartbeat/<your-heartbeat-id>

If the CronJob stops checking in, either Karpenter can't provision the node for the job, or the job itself is broken.

Step 7: Configure alert channels and escalation

Karpenter failures can block cluster scaling — during a traffic spike, a broken Karpenter means no new nodes and eventual pod evictions as resources are exhausted. Configure alerts accordingly:

Alert routing

  1. Go to Alert Channels → Add Channel → Webhook
  2. Add your PagerDuty or Opsgenie webhook for critical alerts
  3. Add your Slack webhook for #platform-alerts

Controller health → PagerDuty P1 after 2 consecutive failures (broken provisioning)

Webhook server → Slack + email, P2 (CRD operations fail but existing workloads run)

Metrics endpoint → Slack only, P3 (monitoring gap, not direct user impact)

Provisioning canary → PagerDuty P1 (confirms Karpenter can't actually provision nodes)

Set confirmation thresholds:

  • Controller health: 2 consecutive failures (~2 minutes)
  • Webhook TCP: 3 consecutive failures (accounts for pod restarts)
  • Provisioning canary: 1 missed heartbeat

Correlating Vigilmon alerts with Karpenter state

When a Karpenter health alert fires:

# 1. Check controller pod status
kubectl get pods -n karpenter -o wide

# 2. Check for recent restarts
kubectl describe pod -n karpenter -l app.kubernetes.io/name=karpenter | \
  grep -A5 "Last State"

# 3. Review controller logs
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --since=10m | \
  grep -E "error|Error|failed|Failed"

# 4. Check for AWS API errors
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --since=10m | \
  grep -E "RequestError|ThrottlingException|AccessDenied|NoCredentialProviders"

# 5. Check NodeClaim status — are any stuck?
kubectl get nodeclaims -A
kubectl describe nodeclaim <name>

# 6. Check NodePool configuration
kubectl get nodepools
kubectl describe nodepool <name>

# 7. Check pending pods that Karpenter should be scheduling
kubectl get pods -A --field-selector=status.phase=Pending

For provisioning canary failures specifically:

# Check if the canary job ran
kubectl get jobs -n karpenter -l app=karpenter-provisioning-canary

# Check canary pod status
kubectl get pods -n karpenter -l app=karpenter-provisioning-canary --sort-by=.metadata.creationTimestamp

# Look at the pod events
kubectl describe pod -n karpenter -l app=karpenter-provisioning-canary | tail -20

Monitoring summary

| Monitor | Type | Interval | Alert Tier | What it catches | |---|---|---|---|---| | Karpenter Controller Health | HTTP | 60s | P1 | Controller crash, OOMKill | | Karpenter Webhook Server | TCP | 60s | P2 | Webhook unavailable | | Karpenter Metrics Endpoint | HTTP | 2min | P3 | Process instability | | Provisioning Canary | Heartbeat | 20min | P1 | Node provisioning failure end-to-end |

What's next

  • SSL certificate monitoring — Karpenter's webhook uses TLS. Vigilmon monitors certificate expiry and alerts you before the webhook cert lapses, which would break all CRD operations.
  • Multi-cluster monitoring — if you run Karpenter across multiple clusters (dev, staging, prod), create monitors for each cluster and use Vigilmon's status page to give platform teams a unified view.
  • Cluster Autoscaler migration — if you're migrating from Cluster Autoscaler to Karpenter, run both monitoring sets simultaneously to catch regressions during the transition.

Start monitoring Karpenter for free at vigilmon.online — set up your first monitor in under two minutes and catch provisioning failures before they affect your cluster's ability to scale.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles