Monitoring KEDA: Kubernetes Event-Driven Autoscaling Observability

title: "Monitoring KEDA: Kubernetes Event-Driven Autoscaling Observability" description: "How to monitor KEDA scaler health, scaling event latency, queue depth metrics, and set up Vigilmon heartbeat checks for event-driven workloads in Kubernetes." date: 2026-07-01 tags: [kubernetes, keda, autoscaling, monitoring, vigilmon]

KEDA (Kubernetes Event-Driven Autoscaling) scales deployments based on external event sources — Kafka lag, SQS queue depth, Redis list length, or any custom metric. When KEDA silently stops scaling, workloads fall behind and queues back up long before any replica count alarm fires. This guide covers what to monitor in KEDA, how to set up meaningful alerts, and how Vigilmon heartbeats keep your scaling pipeline honest.

KEDA Architecture and Failure Modes

KEDA runs two main components:

• keda-operator — watches ScaledObject/ScaledJob resources, polls scalers, and drives HPA objects
• keda-metrics-apiserver — exposes custom and external metrics to the Kubernetes metrics API

Key failure modes:

• Scaler poll failures — KEDA cannot reach the event source (Kafka broker down, SQS endpoint unreachable)
• Scaling event latency — lag accumulates but HPA replicas do not increase fast enough
• Zero-to-one cold start delays — scale-from-zero takes longer than expected, causing request timeouts
• Metrics API unavailability — keda-metrics-apiserver crashing blocks all HPA scaling decisions
• ScaledObject reconciliation errors — misconfigured triggers or changed secret references stall the controller

Prometheus Metrics

KEDA exposes metrics on the keda-operator pod at :8080/metrics and the metrics server at :9022/metrics. Scrape both:

# ServiceMonitor for KEDA operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: keda-operator
  namespace: keda
spec:
  selector:
    matchLabels:
      app: keda-operator
  endpoints:
    - port: metrics
      path: /metrics

Scaler Health

# Active scalers by type
keda_scaler_active{namespace=~"production"}

# Scaler errors (non-zero means the event source is unreachable)
rate(keda_scaler_errors_total[5m]) > 0

# Errors per scaler type broken down
rate(keda_scaler_errors_total[5m]) by (namespace, scaledObject, scaler)

Scaling Event Latency

# Time from metric threshold breach to scale-up completion
histogram_quantile(0.95, rate(keda_scaler_metrics_latency_bucket[5m]))

Queue Depth / Lag Metrics

KEDA exposes the raw metric value it reads from each scaler:

# Current metric value for each scaler (e.g., Kafka consumer lag)
keda_scaler_metrics_value{namespace="production"}

# Compare against the target threshold configured in ScaledObject
keda_scaled_object_paused

Replica Count Tracking

# Desired replicas as decided by KEDA
keda_resource_totals{namespace="production", type="deployment"}

# Detect when KEDA wants more replicas than Kubernetes can schedule
kube_deployment_status_replicas_unavailable{namespace="production"} > 0
and on(deployment)
keda_resource_totals{type="deployment"} > 0

Alerting Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: keda-alerts
  namespace: keda
spec:
  groups:
    - name: keda
      rules:
        - alert: KEDAScalerErrors
          expr: rate(keda_scaler_errors_total[5m]) > 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "KEDA scaler {{ $labels.scaler }} errors in {{ $labels.namespace }}/{{ $labels.scaledObject }}"

        - alert: KEDAHighQueueDepth
          expr: keda_scaler_metrics_value > 10000
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "KEDA queue depth {{ $value }} for {{ $labels.scaledObject }} in {{ $labels.namespace }}"

        - alert: KEDAMetricsAPIDown
          expr: up{job="keda-metrics-apiserver"} == 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "KEDA metrics API server is down — HPA scaling halted"

        - alert: KEDAScaledObjectPaused
          expr: keda_scaled_object_paused == 1
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "ScaledObject {{ $labels.scaledObject }} is paused in {{ $labels.namespace }}"

Debugging Scaling Issues with kubectl

# Check ScaledObject status
kubectl get scaledobject -n production
kubectl describe scaledobject my-worker -n production
# Look for: "Ready" condition and "Active" state

# Inspect HPA that KEDA manages
kubectl get hpa -n production
kubectl describe hpa keda-hpa-my-worker -n production

# Check KEDA operator logs for scaler errors
kubectl logs -n keda deploy/keda-operator --tail=100 | grep -E "error|ERR|scaler"

# Check metrics API server
kubectl logs -n keda deploy/keda-metrics-apiserver --tail=50

# Query the external metric value directly
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/production/kafka-consumer-lag"

For Kafka-based scalers, verify the consumer group lag independently:

# Kafka consumer group lag
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --group my-consumer-group --describe | grep -E "LAG|TOPIC"

ScaledJob Monitoring

For batch workloads using ScaledJob, track job completion and failure rates:

# Job failures in the scaled namespace
rate(kube_job_failed[5m]) > 0

# Jobs stuck pending (not scheduled)
kube_job_status_active > 0 and kube_job_status_start_time == 0

Vigilmon Integration

Heartbeat Checks for Consumer Workers

For event-driven workers that must process messages continuously, register a Vigilmon heartbeat that the worker sends after each successful processing batch:

import requests

def process_batch(messages):
    # process messages...
    for msg in messages:
        handle(msg)

    # Signal health to Vigilmon
    requests.get("https://vigilmon.online/heartbeat/YOUR-HEARTBEAT-ID", timeout=5)

Configure the heartbeat with a period matching your expected throughput — if your worker processes a batch every 30 seconds, set the grace period to 120 seconds. A silent worker triggers an alert.

Monitor Downstream Endpoints

For KEDA scalers triggered by HTTP requests (using the HTTP add-on), add Vigilmon HTTPS monitors:

1. Go to Monitors → Add Monitor → HTTPS
2. Set the URL to your HTTP-scaled service endpoint
3. Set check interval to 30 seconds
4. Add an expected response code (200) and keyword assertion

Scaler Health Dashboard

Create a status page in Vigilmon grouping your event-driven services by pipeline stage — ingest, process, egress — so on-call engineers immediately see which stage is backed up when an alert fires.

Summary

KEDA monitoring must span both the autoscaler infrastructure and the event sources it reads:

| Layer | Tool | What It Catches | |---|---|---| | Scaler metrics | Prometheus | Poll failures, lag buildup, paused objects | | Scaling events | kube-state-metrics + HPA | Replica drift, unavailable pods | | Consumer health | Application heartbeats → Vigilmon | Silent worker failures | | External endpoints | Vigilmon HTTPS monitors | HTTP-trigger endpoint availability |

The most dangerous KEDA failure is one where the scaler stops polling the event source and no replica changes are triggered — queues back up silently. The keda_scaler_errors_total metric and consumer heartbeats together close that gap.