tutorial

Monitoring Kubernetes Vertical Pod Autoscaler (VPA): Recommender Health, Updater Evictions, and Resource Optimization Observability

The Kubernetes Vertical Pod Autoscaler (VPA) automatically adjusts the CPU and memory requests of pods based on historical usage. When it works correctly, it...

The Kubernetes Vertical Pod Autoscaler (VPA) automatically adjusts the CPU and memory requests of pods based on historical usage. When it works correctly, it eliminates the guesswork from resource sizing — over-provisioned pods get trimmed, starved pods get more headroom, and OOM kills drop. When VPA silently fails, pods continue running with stale or incorrect resource requests, and the OOM events and throttling that VPA was supposed to prevent return unnoticed.

This guide covers how to monitor VPA's three core components — the Recommender, Updater, and Admission Controller — and how to use Vigilmon for external visibility into your resource optimization infrastructure.

VPA Architecture: What Can Go Wrong

VPA has three distinct processes, each of which can fail independently:

  • Recommender — reads historical metrics from the Metrics Server or Prometheus and produces LowerBound/UpperBound/Target resource recommendations. If this fails, VPA recommendations go stale.
  • Updater — checks running pods against current recommendations and evicts pods that are outside recommendation bounds so they restart with updated requests. If this fails, VPA recommendations exist but are never applied.
  • Admission Controller — a Mutating Admission Webhook that rewrites resource requests at pod creation time. If this fails, new pods start with their original (uncorrected) requests.

A monitoring gap in any one of these lets incorrect resource sizing persist indefinitely.

Monitoring the VPA Recommender

Recommender Pod Health

# Check Recommender pod status
kubectl get deployment vpa-recommender -n kube-system

# View recent Recommender logs
kubectl logs -n kube-system -l app=vpa-recommender --since=1h | tail -30

# Check for Recommender errors
kubectl logs -n kube-system -l app=vpa-recommender --since=1h | grep -i "error\|warn\|fail"

Recommendation Freshness

VPA stores recommendations in the VerticalPodAutoscaler custom resource status. Check whether recommendations are being updated:

# View current recommendations across all VPAs
kubectl get vpa -A -o json | jq -r \
  '.items[] | "\(.metadata.namespace)/\(.metadata.name): lastUpdated=\(.status.recommendation.containerRecommendations[0].uncappedTarget // "none")"'

# Check if any VPA has no recommendation yet (Recommender may be broken)
kubectl get vpa -A -o json | jq -r \
  '.items[] | select(.status.recommendation == null) | "\(.metadata.namespace)/\(.metadata.name): NO RECOMMENDATION"'

A script to alert on stale recommendations:

#!/bin/sh
# Alert if any VPA recommendation hasn't been updated in 2 hours
kubectl get vpa -A -o json | jq -r '.items[] | .metadata.name' | while read VPA_NAME; do
  LAST_TRANSITION=$(kubectl get vpa "$VPA_NAME" -o jsonpath='{.status.conditions[?(@.type=="RecommendationProvided")].lastTransitionTime}' 2>/dev/null)
  if [ -z "$LAST_TRANSITION" ]; then
    echo "WARNING: VPA $VPA_NAME has no RecommendationProvided condition"
  fi
done

Recommender Prometheus Metrics

The VPA Recommender exposes metrics on port 8942:

# Recommendations currently provided
vpa_recommender_recommendations_provided

# How many container specs have recommendations
vpa_recommender_containers_covered

# Metric fetch failures (Recommender can't get usage data)
vpa_recommender_recommendation_process_duration_seconds_count

# Port-forward to check Recommender metrics
kubectl port-forward -n kube-system deployment/vpa-recommender 8942:8942 &
curl -s http://localhost:8942/metrics | grep vpa_recommender

Monitoring the VPA Updater

Updater Pod Health and Eviction Activity

The Updater is responsible for evicting pods so they restart with corrected resource requests:

# Check Updater pod status
kubectl get deployment vpa-updater -n kube-system

# View Updater logs — look for eviction events
kubectl logs -n kube-system -l app=vpa-updater --since=1h | grep -E "(evict|Evict|error|Error)"

Expected healthy output includes lines like:

I0601 10:00:05 updater.go:160] "Successfully evicted pod" pod="production/api-6c8d9f-xz1q" vpa="production/api-vpa"

Eviction Rate Tracking

Monitor eviction rates to distinguish healthy recalibration from runaway churn:

# Count VPA-driven evictions in the last hour
kubectl get events -A \
  --field-selector reason=EvictedByVPA \
  --sort-by='.lastTimestamp' | wc -l

# Watch evictions in real time
kubectl get events -A \
  --field-selector reason=EvictedByVPA \
  -w

OOM Event Reduction Validation

VPA's core promise is to eliminate OOM kills. Track whether OOM events are decreasing after VPA is applied:

# Count OOMKilled events cluster-wide
kubectl get events -A \
  --field-selector reason=OOMKilling \
  --sort-by='.lastTimestamp' | tail -20

# Check containers that have been OOMKilled recently
kubectl get pods -A -o json | jq -r \
  '.items[] | .metadata.namespace as $ns | .metadata.name as $pod |
   .status.containerStatuses[]? |
   select(.lastState.terminated.reason == "OOMKilled") |
   "\($ns)/\($pod)/\(.name): last OOM at \(.lastState.terminated.finishedAt)"'

Alert if OOM events increase after VPA is deployed — this indicates the Recommender is underestimating resource needs:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: vpa-alerts
  namespace: kube-system
spec:
  groups:
  - name: vpa
    rules:
    - alert: VPARecommenderDown
      expr: up{job="vpa-recommender"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "VPA Recommender is down"
        description: "The VPA Recommender has been unreachable for 5 minutes. Recommendations are no longer being updated."

    - alert: VPAUpdaterDown
      expr: up{job="vpa-updater"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "VPA Updater is down"
        description: "The VPA Updater has been unreachable for 5 minutes. Existing recommendations are not being applied."

    - alert: VPAHighEvictionRate
      expr: increase(vpa_updater_evictions_total[30m]) > 20
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "VPA is evicting pods at an unusually high rate"
        description: "More than 20 VPA-driven evictions in 30 minutes may indicate unstable recommendations or misconfigured min/max bounds."

    - alert: VPANoRecommendations
      expr: vpa_recommender_recommendations_provided == 0
      for: 30m
      labels:
        severity: warning
      annotations:
        summary: "VPA Recommender is producing no recommendations"
        description: "No VPA recommendations have been generated in 30 minutes. Check Metrics Server connectivity."

Monitoring the VPA Admission Controller

The Admission Controller is a Mutating Webhook — if it becomes unavailable, new pod creation fails or falls back to original resource requests depending on your failurePolicy setting.

Webhook Health

# Check Admission Controller pod status
kubectl get deployment vpa-admission-controller -n kube-system

# Verify the MutatingWebhookConfiguration is registered
kubectl get mutatingwebhookconfigurations | grep vpa

# Describe the webhook to check its endpoint and failure policy
kubectl describe mutatingwebhookconfiguration vpa-webhook-config

Webhook Response Time

A slow Admission Controller adds latency to pod scheduling. Monitor it:

# Check recent webhook-related events
kubectl get events -A \
  --field-selector reason=FailedCreate \
  --sort-by='.lastTimestamp' | grep -i vpa | tail -10

# Look for admission webhook timeout errors in API server logs
# (requires access to control plane logs)
kubectl logs -n kube-system kube-apiserver-$(hostname) 2>/dev/null | \
  grep "vpa" | grep -i "timeout\|error" | tail -20

End-to-End Admission Validation

Deploy a test pod and confirm VPA mutated its resource requests:

# Create a test pod in a namespace with VPA
kubectl run vpa-test --image=nginx --restart=Never -n production -- sleep 3600

# Check if VPA mutated the resource requests
kubectl get pod vpa-test -n production \
  -o jsonpath='{.spec.containers[0].resources}' | jq .

# Expected: resources will include requests set by VPA, not the original spec
kubectl delete pod vpa-test -n production

Integrating Vigilmon for Resource Optimization Infrastructure Monitoring

VPA is internal cluster infrastructure — but you can monitor its health signals externally with Vigilmon:

Monitor your VPA status API or health endpoint. If you expose a simple health endpoint that queries VPA status, Vigilmon checks it from outside the cluster every minute:

  1. Log in to Vigilmon and go to Monitors → New Monitor.
  2. Add an HTTP / HTTPS monitor targeting your VPA health endpoint.
  3. Set check interval to 1 minute.
  4. Configure a keyword assertion for "recommender":"healthy" or similar JSON field from your status API.
  5. Set up alert channels (email, Slack, PagerDuty) so your platform team is notified immediately when VPA infrastructure degrades.

A minimal VPA health API deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vpa-health-exporter
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vpa-health-exporter
  template:
    metadata:
      labels:
        app: vpa-health-exporter
    spec:
      serviceAccountName: vpa-health-reader
      containers:
      - name: exporter
        image: bitnami/kubectl:latest
        command:
        - /bin/sh
        - -c
        - |
          while true; do
            RECOMMENDER=$(kubectl get deployment vpa-recommender -n kube-system \
              -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo 0)
            UPDATER=$(kubectl get deployment vpa-updater -n kube-system \
              -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo 0)
            ADMISSION=$(kubectl get deployment vpa-admission-controller -n kube-system \
              -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo 0)
            echo "{\"recommender\":${RECOMMENDER},\"updater\":${UPDATER},\"admission\":${ADMISSION}}" > /tmp/status.json
            sleep 30
          done &
          python3 -m http.server 8080 --directory /tmp
        ports:
        - containerPort: 8080

Vigilmon will alert you if this endpoint becomes unreachable — catching node failures, namespace deletions, or RBAC regressions that kill VPA infrastructure before they affect workload sizing.

Key Metrics Summary

| Component | What to Monitor | Alert Threshold | |-----------|----------------|-----------------| | Recommender pod | up{job="vpa-recommender"} | Down for > 5m | | Updater pod | up{job="vpa-updater"} | Down for > 5m | | Admission Controller | Pod readiness | 0 ready for > 5m | | Recommendations provided | vpa_recommender_recommendations_provided | 0 for > 30m | | Eviction rate | vpa_updater_evictions_total | > 20 in 30m | | OOM events | kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} | Any increase post-VPA | | VPA objects with no recommendation | kubectl get vpa | Any VPA with null status |

Conclusion

VPA's three-component architecture means a failure in any one layer silently degrades your resource optimization. The Recommender going stale means recommendations don't update; the Updater going down means updates aren't applied; the Admission Controller failing means new pods ignore VPA entirely. By monitoring each component with targeted health checks, eviction rate tracking, and OOM event reduction validation, you catch VPA failures before they manifest as throttling, OOM kills, or resource waste. Vigilmon's external monitoring rounds out the picture by giving you an out-of-cluster perspective on whether your VPA infrastructure is alive and reachable.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles