Monitoring kube-downscaler: Ensure Your Kubernetes Downscaling Is Actually Working

kube-downscaler (also known as py-kube-downscaler) automatically scales Kubernetes Deployments, StatefulSets, and CronJobs to zero or minimum replicas during off-hours. For staging and development clusters, this can cut cloud costs by 60–80% overnight. But if kube-downscaler stops running, your off-hours savings evaporate silently. And if it runs when it shouldn't — wrong timezone, misconfigured schedule, or a bug — it can downscale production.

This tutorial shows you how to monitor kube-downscaler's operation with Vigilmon so you know immediately when it's not doing its job, and just as importantly, when it's doing it at the wrong time.

What Is kube-downscaler?

kube-downscaler runs as a Kubernetes Deployment in your cluster. It watches namespaces for the downscaler/uptime and downscaler/downtime annotations and scales workloads according to time-based rules:

# Annotation on a Deployment to downscale outside business hours
annotations:
  downscaler/uptime: "Mon-Fri 08:00-20:00 Europe/Berlin"
  downscaler/downtime: "always"

The controller reconciles on a configurable interval (default 30 seconds), reads current time, and scales workloads up or down based on whether the current time falls within the uptime window.

Why Monitoring kube-downscaler Matters

Silent controller crashes — if the kube-downscaler pod crashes and isn't restarted quickly (due to CrashLoopBackOff or a resource limit), workloads stay at whatever replica count they had when it died. Savings stop accruing, but nothing alerts.

Wrong timezone configuration — the --default-uptime-timezone flag controls how schedule strings are interpreted. A misconfigured timezone causes downscaling to fire at the wrong time. UTC vs CET is a 1–2 hour offset that can downscale staging during working hours.

Namespace annotation drift — engineers removing or modifying downscaler/uptime annotations without coordination can silently disable downscaling for specific workloads, or worse, accidentally enable it on production namespaces.

Scale-up failures — kube-downscaler scales workloads back up at the start of each uptime window. If it crashes just before scale-up, your team arrives in the morning to find all development environments at zero replicas with no automatic recovery.

RBAC permission removal — kube-downscaler needs ClusterRole permissions to list and patch Deployments across namespaces. An inadvertent RBAC change makes the controller silently skip workloads it can no longer patch.

What You'll Set Up

• Heartbeat monitor to verify kube-downscaler is reconciling on schedule
• HTTP monitor for the controller's health endpoint
• Heartbeat monitors for scale-up verification (environments should be up during business hours)
• Alert routing for schedule violations

You'll need a free Vigilmon account — no credit card required.

Step 1: Monitor the kube-downscaler Pod Health

kube-downscaler exposes a metrics endpoint on port 9090 by default (configurable with --metrics-port). The /health path returns a simple status:

kubectl port-forward -n kube-downscaler deploy/kube-downscaler 9090:9090
curl http://localhost:9090/health
# Expected: healthy

Expose this endpoint via an Ingress for external monitoring:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kube-downscaler-health
  namespace: kube-downscaler
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /health
spec:
  rules:
    - host: kube-downscaler-health.internal.yourdomain.com
      http:
        paths:
          - path: /health
            pathType: Prefix
            backend:
              service:
                name: kube-downscaler
                port:
                  number: 9090

Then add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL: https://kube-downscaler-health.internal.yourdomain.com/health
4. Check interval: 1 minute
5. Expected response:
   • Status code: 200
   • Response body contains: healthy
6. Save the monitor

Step 2: Heartbeat Monitor for Reconciliation Activity

The best proof that kube-downscaler is working is that it logs reconciliation events. A wrapper script that verifies recent reconciliation activity and pings Vigilmon:

#!/bin/bash
# kube-downscaler-heartbeat.sh
set -e

NAMESPACE="${DOWNSCALER_NAMESPACE:-kube-downscaler}"
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"
MAX_LOG_AGE_SECONDS=120  # alert if no reconciliation logged in 2 minutes

# Get the downscaler pod name
POD=$(kubectl get pods -n "$NAMESPACE" -l app=kube-downscaler \
  -o jsonpath='{.items[0].metadata.name}')

if [ -z "$POD" ]; then
  echo "ERROR: No kube-downscaler pod found in namespace $NAMESPACE"
  exit 1
fi

# Check for recent log activity (reconciliation logs contain "scaling" or "already at")
RECENT_LOGS=$(kubectl logs "$POD" -n "$NAMESPACE" --since="${MAX_LOG_AGE_SECONDS}s" 2>/dev/null | \
  grep -c -E "(scaling|already at|no change)" || true)

if [ "$RECENT_LOGS" -eq "0" ]; then
  echo "WARNING: No reconciliation activity in last ${MAX_LOG_AGE_SECONDS}s"
  exit 1
fi

echo "kube-downscaler active: $RECENT_LOGS reconciliation log lines in last ${MAX_LOG_AGE_SECONDS}s"
curl -fsS "$HEARTBEAT_URL" > /dev/null
echo "Heartbeat sent."

Deploy as a CronJob running every 3 minutes:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-downscaler-heartbeat
  namespace: monitoring
spec:
  schedule: "*/3 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: downscaler-monitor-sa
          containers:
            - name: check
              image: bitnami/kubectl:latest
              env:
                - name: DOWNSCALER_NAMESPACE
                  value: "kube-downscaler"
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: downscaler-heartbeat-url
              command: ["/bin/sh", "/scripts/kube-downscaler-heartbeat.sh"]
              volumeMounts:
                - name: scripts
                  mountPath: /scripts
          volumes:
            - name: scripts
              configMap:
                name: downscaler-check-scripts
                defaultMode: 0755

Create the RBAC for the monitoring service account:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: downscaler-monitor
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: downscaler-monitor
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: downscaler-monitor
subjects:
  - kind: ServiceAccount
    name: downscaler-monitor-sa
    namespace: monitoring

Set up the heartbeat in Vigilmon:

1. Go to Monitors → New Monitor → Heartbeat
2. Name it kube-downscaler reconciliation
3. Expected interval: 3 minutes
4. Grace period: 2 minutes
5. Copy the ping URL into your Kubernetes Secret

Step 3: Scale-Up Verification Heartbeat

The most critical kube-downscaler event is the morning scale-up — workloads must be running before your team starts work. Add a business-hours verification heartbeat that checks environments are up during working hours:

#!/bin/bash
# scale-up-verification.sh — run at start of business hours
set -e

NAMESPACE="${TARGET_NAMESPACE:-staging}"
HEARTBEAT_URL="${VIGILMON_SCALEUP_HEARTBEAT_URL}"
MIN_READY_PODS=3  # adjust to match your environment

# Count ready pods across all deployments in the namespace
READY=$(kubectl get pods -n "$NAMESPACE" \
  --field-selector=status.phase=Running \
  -o json | jq '[.items[] | 
    select(.status.containerStatuses[0].ready == true)] | length')

if [ "$READY" -lt "$MIN_READY_PODS" ]; then
  echo "ERROR: Only $READY ready pods in $NAMESPACE — expected at least $MIN_READY_PODS"
  echo "kube-downscaler may have failed to scale up"
  exit 1
fi

echo "Scale-up verified: $READY ready pods in $NAMESPACE"
curl -fsS "$HEARTBEAT_URL" > /dev/null
echo "Scale-up heartbeat sent."

Schedule this to run at the start of your uptime window — for a Mon-Fri 08:00-20:00 Europe/Berlin schedule, run at 08:10 to give kube-downscaler 10 minutes to complete scale-up:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: staging-scaleup-verify
  namespace: monitoring
spec:
  schedule: "10 6 * * 1-5"  # 08:10 Berlin = 06:10 UTC
  timeZone: "UTC"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: downscaler-monitor-sa
          containers:
            - name: verify
              image: bitnami/kubectl:latest
              env:
                - name: TARGET_NAMESPACE
                  value: "staging"
                - name: VIGILMON_SCALEUP_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: staging-scaleup-heartbeat-url
              command: ["/bin/sh", "/scripts/scale-up-verification.sh"]
              volumeMounts:
                - name: scripts
                  mountPath: /scripts
          volumes:
            - name: scripts
              configMap:
                name: downscaler-check-scripts
                defaultMode: 0755

Set up the Vigilmon heartbeat:

1. Monitors → New Monitor → Heartbeat
2. Name: staging scale-up (Mon-Fri 08:10 Berlin)
3. Expected interval: 1 day (it only fires on weekdays; set grace to 4 hours for weekends)
4. Grace period: 30 minutes

If kube-downscaler fails to scale staging back up, the heartbeat goes silent and your team gets an alert before they discover it themselves.

Step 4: Alert Routing

Configure separate alert channels for different severity levels:

1. Controller health → page on-call immediately (kube-downscaler pod down)
2. Scale-up failure → alert engineering team in #staging-alerts (no cost impact but team productivity blocked)
3. Reconciliation gap → alert SRE in #infrastructure-alerts (savings at risk)

In Vigilmon:

1. Go to Alert Channels → New Channel
2. Create a Slack webhook channel for #infrastructure-alerts
3. Create a separate Slack webhook channel for #staging-alerts
4. Assign the controller health and reconciliation monitors to #infrastructure-alerts
5. Assign the scale-up verification monitor to #staging-alerts

Recommended Alert Thresholds

| Monitor | Check interval | Grace period | Alert channel | |---|---|---|---| | Controller /health endpoint | 1 min | — | #infrastructure-alerts (page) | | Reconciliation activity heartbeat | 3 min | 2 min | #infrastructure-alerts | | Staging scale-up verification | Daily (Mon-Fri) | 30 min | #staging-alerts | | Dev scale-up verification | Daily (Mon-Fri) | 30 min | #dev-alerts |

Common Issues and What Monitoring Catches

| Issue | Symptom | Which monitor fires | |---|---|---| | Controller pod CrashLoopBackOff | No scale-up/down occurs | Health endpoint + reconciliation heartbeat | | Wrong timezone configured | Downscaling at wrong time | Scale-up verification at wrong time | | RBAC permission removed | Workloads silently skipped | Reconciliation heartbeat (no patch activity) | | Scale-up missed before business hours | Team finds zero replicas at 9 AM | Scale-up verification heartbeat | | Controller running but frozen | Health shows OK, no reconciliation | Reconciliation activity heartbeat |

Summary

kube-downscaler saves money while you sleep — but only if it's running correctly. Monitoring it with Vigilmon gives you confidence that your cost savings are real and your environments come back up on time:

| What to monitor | Vigilmon monitor type | |---|---| | Controller pod health | HTTP /health monitor | | Active reconciliation | Heartbeat (3-minute interval) | | Business-hours scale-up | Heartbeat (daily, morning CronJob) |

Start free at vigilmon.online — your first kube-downscaler monitor is running in under two minutes.