Monitoring Kueue with Vigilmon

Kueue is a Kubernetes-native job queuing and quota management system that sits on top of standard batch APIs like Job, JobSet, and RayJob. It enforces resource quotas, priorities, and borrowing across namespaces — but it ships no built-in uptime dashboard. When the Kueue controller stops admitting workloads or its webhook goes dark, batch pipelines stall silently. Vigilmon closes that gap with external health checks, webhook probes, and dead-man's-switch heartbeats for your batch workloads.

What You'll Set Up

• HTTP health probe for the Kueue controller-manager
• Webhook availability check so admission never silently breaks
• Cron heartbeat from a periodic workload to confirm end-to-end admission works
• Alerting thresholds tuned for batch scheduling latency

Prerequisites

• Kubernetes 1.27+ cluster with Kueue installed (v0.7+)
• kubectl access with permission to expose internal services
• A free Vigilmon account

Step 1: Expose the Kueue Health Endpoint

Kueue's controller-manager exposes a /healthz endpoint on port 8081 inside the cluster. Expose it for external probing via a NodePort or an Ingress.

Option A — NodePort (quick)

kubectl patch svc kueue-controller-manager-metrics-service \
  -n kueue-system \
  --type='json' \
  -p='[{"op":"add","path":"/spec/ports/-","value":{"name":"healthz","port":8081,"targetPort":8081,"nodePort":30081}}]'
kubectl patch svc kueue-controller-manager-metrics-service \
  -n kueue-system \
  --type='merge' \
  -p='{"spec":{"type":"NodePort"}}'

Then add a monitor in Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter http://<node-ip>:30081/healthz.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Option B — Ingress (recommended for production)

Create an Ingress rule that routes kueue-health.internal.example.com/healthz to the controller service. Add the resulting URL as an HTTP monitor in Vigilmon with the same settings.

Step 2: Monitor the Webhook Server

Kueue runs an admission webhook on port 9443. If the webhook pod becomes unreachable, all new workload submissions are blocked (depending on your failurePolicy setting). Add a second Vigilmon monitor pointed at the webhook readiness path:

kubectl port-forward -n kueue-system deploy/kueue-controller-manager 9443:9443 &
curl -k https://localhost:9443/readyz

If it returns {"status":"ok"}, you can wire a similar external probe:

1. Expose the webhook readiness port through an internal LoadBalancer or NodePort.
2. Add a second Vigilmon monitor for https://<host>:<port>/readyz with TLS verification enabled.
3. Set Alert after to 2 consecutive failures — a single timeout is common during rolling upgrades.

Step 3: Heartbeat from a Periodic Batch Job

The most valuable Kueue monitor is an end-to-end admission check: a lightweight Job that Kueue must admit, run, and complete — then pings Vigilmon's heartbeat URL. If the ping doesn't arrive on schedule, Vigilmon alerts.

Create the heartbeat CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vigilmon-kueue-heartbeat
  namespace: batch-workloads
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            kueue.x-k8s.io/queue-name: default-queue
        spec:
          restartPolicy: Never
          containers:
          - name: heartbeat
            image: curlimages/curl:latest
            command:
            - sh
            - -c
            - |
              echo "Heartbeat job running"
              curl -fsS https://vigilmon.online/api/heartbeat/<YOUR_HEARTBEAT_ID>

Configure the LocalQueue label

Replace default-queue with the name of your Kueue LocalQueue. Kueue reads the kueue.x-k8s.io/queue-name label on the Pod template to route the workload.

Set up the Vigilmon heartbeat monitor

1. In Vigilmon, click Add Monitor → Heartbeat.
2. Set Expected interval to 5 minutes.
3. Set Grace period to 2 minutes.
4. Copy the heartbeat URL and paste it into the CronJob manifest above.
5. Save both the monitor and the CronJob (kubectl apply -f heartbeat-cronjob.yaml).

If Kueue stops admitting workloads — quota exhausted, webhook down, controller crashed — the Job never completes, the heartbeat stops, and Vigilmon fires an alert within 7 minutes.

Step 4: Alert on Queue Saturation

Kueue exposes Prometheus metrics at :8080/metrics. Key signals to watch:

| Metric | Alert when | |---|---| | kueue_admission_attempts_total{result="inadmissible"} | Rate > 0 for 5 min | | kueue_pending_workloads | > threshold for your cluster size | | kueue_admitted_active_workloads | Drops to 0 unexpectedly | | kueue_evicted_workloads_total | Spike in a 5-minute window |

While Vigilmon focuses on availability rather than Prometheus scraping, you can expose a small sidecar service that queries these metrics and returns HTTP 200 / 503 based on a threshold — then point a Vigilmon HTTP monitor at it for a blended availability + saturation alert.

Step 5: Configure Notifications

1. In Vigilmon, go to Settings → Notifications.
2. Add your Slack webhook, PagerDuty integration key, or email address.
3. On the Kueue health monitor, set the notification channel.
4. Recommended: use a 15-minute escalation delay for the batch heartbeat (minor queue depth spikes resolve quickly) and a 2-minute delay for the controller health check (a crashed controller needs immediate attention).

What You've Built

| Monitor | Type | Checks | |---|---|---| | Kueue controller /healthz | HTTP | Controller process alive | | Webhook /readyz | HTTP | Admission webhook reachable | | Batch job heartbeat | Heartbeat | End-to-end admission + completion |

This three-layer setup catches controller crashes, webhook outages, and silent admission freezes — the three failure modes most likely to stall your batch infrastructure without obvious error messages. With Vigilmon alerting, you'll know about Kueue problems before your data pipelines miss their SLAs.