tutorial

Monitoring the Kubernetes Descheduler: Evictions, Policy Health, and Cluster Rebalancing Observability

The Kubernetes Descheduler is the missing counterpart to the scheduler. While `kube-scheduler` places pods optimally at the time they start, the cluster stat...

The Kubernetes Descheduler is the missing counterpart to the scheduler. While kube-scheduler places pods optimally at the time they start, the cluster state drifts: nodes become overloaded after new workloads arrive, tainted nodes accumulate pods that shouldn't be there, and anti-affinity rules can be violated by rolling restarts. The Descheduler periodically evicts pods that no longer fit their placement rules so the scheduler can re-place them more optimally.

When the Descheduler stops working — silently, without surfacing an error — your cluster rebalancing strategy fails without anyone noticing. Pods pile up on overloaded nodes, Pod Disruption Budgets stop being respected, and your bin-packing improvements never materialise. This guide covers how to monitor the Descheduler's health, track eviction activity, and use Vigilmon for external visibility into the infrastructure running it.

Why Descheduler Monitoring Matters

The Descheduler typically runs as a CronJob or Deployment inside the cluster. Both failure modes are invisible without dedicated monitoring:

  • CronJob silently stops firing — a misconfigured schedule or resource exhaustion causes the CronJob to fail; kubectl get pods shows the last successful run, not the absence of new ones
  • Plugin execution failures — individual eviction plugins (e.g., LowNodeUtilization, RemovePodsViolatingNodeAffinity) can fail without crashing the Descheduler process
  • PodDisruptionBudget violations — the Descheduler evicts more pods than PDBs allow, causing service disruption
  • Eviction rate anomalies — zero evictions over a long period may indicate the policy is misconfigured; too many evictions indicate a thrashing cluster

Core Components to Monitor

1. Descheduler Job Execution Health

If you deploy the Descheduler as a CronJob (the most common production pattern), verify runs are completing successfully:

# Check recent Descheduler job history
kubectl get jobs -n kube-system -l app=descheduler --sort-by=.metadata.creationTimestamp

# Check the last job's pod status
kubectl get pods -n kube-system -l app=descheduler --sort-by=.metadata.creationTimestamp | tail -5

# Inspect the most recent run's logs
kubectl logs -n kube-system -l app=descheduler --since=1h | tail -50

Look for lines like "Evicted pod" or "no evictable pods" — either is healthy. An absence of recent log output means the job is not running.

For a Deployment-based Descheduler, check readiness directly:

kubectl rollout status deployment/descheduler -n kube-system
kubectl get deployment descheduler -n kube-system -o jsonpath='{.status.readyReplicas}'

2. Eviction Event Rate

Kubernetes records eviction events. Track them to detect both complete silence (policy misconfiguration) and runaway eviction (cluster instability):

# Count eviction events in the last hour
kubectl get events -A --field-selector reason=Evicted \
  --sort-by='.lastTimestamp' | grep "$(date +%Y-%m-%d)" | wc -l

# View eviction events with source context
kubectl get events -A \
  --field-selector reason=Evicted \
  -o custom-columns='TIME:.lastTimestamp,NS:.involvedObject.namespace,POD:.involvedObject.name,MSG:.message' \
  --sort-by='.lastTimestamp' | tail -20

You can also query eviction events via the Kubernetes API and push counts to a monitoring system:

# Export eviction count as a metric for Prometheus pushgateway
EVICTIONS=$(kubectl get events -A --field-selector reason=Evicted \
  --no-headers 2>/dev/null | wc -l)
echo "descheduler_evictions_total ${EVICTIONS}" | \
  curl --data-binary @- http://pushgateway:9091/metrics/job/descheduler

3. Plugin-Level Health via Descheduler Logs

The Descheduler logs which plugins ran and how many pods each evicted. Parse these logs to detect plugin failures:

# Check which plugins ran and their eviction counts
kubectl logs -n kube-system -l app=descheduler --since=2h | \
  grep -E "(plugin|Evicted|error|Error|failed)" | \
  grep -v "^#"

A healthy run looks like:

I0601 10:00:01 descheduler.go:214] "Running descheduler with policies" policies=["LowNodeUtilization","RemovePodsViolatingNodeAffinity"]
I0601 10:00:02 lownodeutilization.go:201] "Evicted pod" pod="production/api-7d9f8b-xk2p" node="node-3"
I0601 10:00:02 descheduler.go:221] "Number of evicted pods" totalEvicted=3

A plugin failure looks like:

E0601 10:00:02 node.go:83] "Error getting node utilization" err="context deadline exceeded"

Set up a CronJob to check for error patterns and alert:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: descheduler-log-health-check
  namespace: kube-system
spec:
  schedule: "*/30 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: descheduler-log-reader
          containers:
          - name: checker
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - |
              ERRORS=$(kubectl logs -n kube-system -l app=descheduler --since=35m 2>/dev/null | \
                grep -c "^E" || true)
              echo "Descheduler error lines in last 35m: $ERRORS"
              [ "$ERRORS" -lt 5 ] || exit 1
          restartPolicy: Never

4. PodDisruptionBudget Respect

If the Descheduler violates PDBs, you'll see eviction failures logged — or worse, service disruption without logging. Monitor PDB status to detect when the Descheduler is pushing against them:

# Check all PDBs for disruptions allowed
kubectl get pdb -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,MIN:.spec.minAvailable,CURRENT:.status.currentHealthy,DISRUPTIONS:.status.disruptionsAllowed'

# Watch for PDB violations in events
kubectl get events -A --field-selector reason=EvictionThrottled --sort-by='.lastTimestamp' | tail -10

Alert if any PDB reports zero disruptions allowed during Descheduler run windows — this means the Descheduler is blocked from evicting, and workloads may be stuck on suboptimal nodes:

# Find PDBs that are blocking evictions (0 disruptions allowed, <100% healthy)
kubectl get pdb -A -o json | jq -r \
  '.items[] | select(.status.disruptionsAllowed == 0) |
   "\(.metadata.namespace)/\(.metadata.name): currentHealthy=\(.status.currentHealthy) expected=\(.status.expectedPods)"'

5. Prometheus Alerting Rules

Deploy these rules via PrometheusRule to catch Descheduler failures:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: descheduler-alerts
  namespace: kube-system
spec:
  groups:
  - name: descheduler
    rules:
    - alert: DeschedulerJobNotRunning
      expr: time() - kube_cronjob_status_last_successful_time{cronjob="descheduler", namespace="kube-system"} > 7200
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Descheduler CronJob has not run in 2 hours"
        description: "The Descheduler last ran more than 2 hours ago — cluster rebalancing may be stalled."

    - alert: DeschedulerJobFailed
      expr: kube_job_status_failed{job_name=~"descheduler-.*", namespace="kube-system"} > 0
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: "Descheduler job failed"
        description: "A Descheduler execution job has failed. Check pod logs in kube-system."

    - alert: DeschedulerHighEvictionRate
      expr: increase(descheduler_pods_evicted_total[30m]) > 50
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Unusually high Descheduler eviction rate"
        description: "More than 50 pods evicted in the last 30 minutes — check for cluster instability or misconfigured policies."

    - alert: PodDisruptionBudgetBlocking
      expr: kube_poddisruptionbudget_status_disruptions_allowed == 0
      for: 30m
      labels:
        severity: warning
      annotations:
        summary: "PDB {{ $labels.namespace }}/{{ $labels.poddisruptionbudget }} is blocking evictions"

Integrating Vigilmon for External Observability

The Descheduler itself doesn't expose an HTTP endpoint — but the infrastructure it manages does. Use Vigilmon to monitor the external health signals that prove rebalancing is working:

Monitor your node utilization endpoint if you expose cluster metrics via a custom dashboard or Prometheus proxy:

  1. Log in to Vigilmon and go to Monitors → New Monitor.
  2. Add an HTTP monitor targeting your cluster metrics endpoint or a synthetic health API that reports Descheduler last-run status.
  3. Set check interval to 5 minutes.
  4. Configure keyword assertion for "descheduler" or "last_eviction" to confirm your metrics pipeline is alive.

For teams running a metrics exporter or health API, a simple endpoint exposes Descheduler status:

# Health endpoint wrapper script — deploy as a sidecar or standalone pod
#!/bin/sh
LAST_RUN=$(kubectl logs -n kube-system -l app=descheduler --since=2h 2>/dev/null | \
  grep "Number of evicted pods" | tail -1)
if [ -z "$LAST_RUN" ]; then
  echo '{"status":"stale","message":"No Descheduler run in 2 hours"}'
  exit 1
fi
echo "{\"status\":\"ok\",\"last_run\":\"${LAST_RUN}\"}"

Wrap this in a minimal HTTP server and add it to Vigilmon:

apiVersion: v1
kind: Service
metadata:
  name: descheduler-health
  namespace: kube-system
spec:
  selector:
    app: descheduler-health-exporter
  ports:
  - port: 8080
    targetPort: 8080
  type: LoadBalancer

Vigilmon will then alert you from outside the cluster if the Descheduler health API stops responding — catching CronJob failures, namespace issues, or RBAC problems that internal Prometheus rules may miss.

Key Metrics Summary

| Signal | What to Monitor | Alert Threshold | |--------|----------------|-----------------| | CronJob last success | kube_cronjob_status_last_successful_time | > 2h without a run | | Job failure count | kube_job_status_failed | Any failure | | Eviction event rate | descheduler_pods_evicted_total | > 50 in 30m or 0 in 24h | | PDB disruptions allowed | kube_poddisruptionbudget_status_disruptions_allowed | 0 for > 30m | | Error log lines | Grep for ^E in Descheduler logs | > 5 errors per run | | External health API | HTTP 200 on health endpoint | Any failure for > 10m |

Conclusion

The Descheduler is a set-and-forget component that quietly degrades when it stops working. By tracking job execution history, parsing plugin logs, monitoring eviction event rates, and watching PDB status, you catch rebalancing failures before they result in node overload or placement drift. Layering Vigilmon's external HTTP monitoring on top gives you a fast, out-of-band signal when the Descheduler infrastructure itself is unreachable — critical for catching RBAC misconfigurations or namespace deletions that silence the CronJob without any in-cluster alert.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles