tutorial

Monitoring Zuul CI with Vigilmon

Zuul CI's project-gating model ensures only tested, mergeable code lands in your branches — but when Zuul's scheduler stalls, executors go offline, or Zookeeper connectivity breaks, pipelines silently drain and pull requests accumulate without merging. Here's how to monitor Zuul CI availability, executor health, pipeline queue depth, and gating latency with Vigilmon.

Zuul CI is an open-source project gating and CI/CD system originally developed for the OpenDev (OpenStack) infrastructure. Its speculative merge queue ensures that every commit to a protected branch passes all required tests in the exact configuration it will have after merging — preventing the "works on my branch" failures that plague simpler CI systems. When Zuul's scheduler loses its Zookeeper quorum, an executor becomes unavailable, or a pipeline stall causes the gate queue to pile up, pull requests stop merging silently. Developers see jobs in a permanent "queued" or "waiting" state with no indication that the gating infrastructure itself is unhealthy. Vigilmon provides external monitoring for Zuul's REST API, executor availability, pipeline queue depth, and job execution latency so you detect gating failures before your development velocity collapses.

What You'll Set Up

  • Zuul web API availability and scheduler health
  • Executor availability and task queue monitoring
  • Pipeline queue depth per project
  • Job execution latency tracking
  • Zookeeper connectivity monitoring

Prerequisites

  • Zuul CI 5.x+ with the REST API enabled
  • Access to the Zuul web endpoint (e.g., https://zuul.your-org.internal)
  • Zookeeper ensemble accessible for health checks
  • A free Vigilmon account

Step 1: Monitor the Zuul Web API

Zuul exposes a REST API at /api/ that reflects scheduler health. When the scheduler loses its Zookeeper connection or enters an error state, the API returns degraded responses or stops responding entirely. Add Vigilmon's HTTP monitor as your first line of defense:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://zuul.your-org.internal/api/info.
  3. Set Check interval to 1 minute.
  4. Under Expected response, add keyword capabilities — the info endpoint always includes this in a healthy response.
  5. Set Timeout to 15 seconds.

The /api/info endpoint returns scheduler metadata including tenant configuration and API version. If this endpoint becomes unavailable, Zuul's entire web UI and API are down — developers cannot see build results, check pipeline status, or trigger manual builds.

For deeper scheduler health, monitor the tenants endpoint too:

# Verify Zuul scheduler is serving tenant data
ZUUL_URL="https://zuul.your-org.internal"
EXPECTED_TENANT="your-tenant-name"

TENANTS=$(curl -sf --max-time 15 "${ZUUL_URL}/api/tenants" 2>/dev/null)

if echo "$TENANTS" | python3 -c "
import sys, json
tenants = json.load(sys.stdin)
names = [t.get('name') for t in tenants]
sys.exit(0 if '${EXPECTED_TENANT}' in names else 1)
" 2>/dev/null; then
  echo "Zuul scheduler healthy: tenant '${EXPECTED_TENANT}' present"
  curl -s "https://vigilmon.online/heartbeat/REPLACE_WITH_YOUR_TOKEN"
else
  echo "Zuul scheduler degraded: tenant '${EXPECTED_TENANT}' missing from API"
  exit 1
fi

Step 2: Monitor Executor Availability

Zuul executors are the workers that actually run jobs by connecting to remote nodes via Ansible. A Zuul deployment can have multiple executors, and if all executors go offline (due to OOM, disk pressure, or failed Zookeeper session renewal), no jobs execute. Monitor executor availability via the API:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 5 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the executor health check script:

#!/bin/bash
# /usr/local/bin/zuul-executor-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
ZUUL_URL="https://zuul.your-org.internal"
MIN_EXECUTORS=1   # Alert if fewer than this many executors are online
TIMEOUT=15

# Query Zuul executor stats via components endpoint
EXECUTOR_DATA=$(curl -sf \
  --max-time "$TIMEOUT" \
  "${ZUUL_URL}/api/components" 2>/dev/null)

if [ -z "$EXECUTOR_DATA" ]; then
  echo "Zuul API unreachable"
  exit 1
fi

ONLINE_EXECUTORS=$(echo "$EXECUTOR_DATA" | python3 -c "
import sys, json
data = json.load(sys.stdin)
executors = data.get('executor', [])
online = [e for e in executors if e.get('state') == 'running']
print(len(online))
" 2>/dev/null || echo 0)

if [ "${ONLINE_EXECUTORS:-0}" -ge "$MIN_EXECUTORS" ]; then
  echo "Zuul executors OK: ${ONLINE_EXECUTORS} online"
  curl -s "$HEARTBEAT_URL"
else
  echo "Zuul executors down: ${ONLINE_EXECUTORS} online (minimum: ${MIN_EXECUTORS})"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/zuul-executor-check.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/zuul-executor-check.sh >> /var/log/zuul-health.log 2>&1") | crontab -

If all executors go offline, Zuul continues to accept job triggers and enqueue them — but nothing ever runs. Without monitoring, this looks identical to a slow build queue until developers notice their jobs have been "waiting for a node" for hours.

Step 3: Monitor Pipeline Queue Depth

Zuul's pipeline model — check, gate, post — each maintain their own queues. A healthy gate pipeline queue grows during peak commit periods and drains as jobs pass. A stuck gate queue (one that only grows, never drains) indicates either executor starvation, a consistently failing gate job that blocks all changes behind it, or a speculative execution cycle that never completes.

#!/bin/bash
# /usr/local/bin/zuul-pipeline-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
ZUUL_URL="https://zuul.your-org.internal"
TENANT="your-tenant-name"
PIPELINE="gate"              # Check your most critical pipeline
MAX_QUEUE_DEPTH=25           # Alert if more than 25 changes queued in gate
TIMEOUT=15

# Get pipeline status
PIPELINE_STATUS=$(curl -sf \
  --max-time "$TIMEOUT" \
  "${ZUUL_URL}/api/tenant/${TENANT}/status" 2>/dev/null)

QUEUE_DEPTH=$(echo "$PIPELINE_STATUS" | python3 -c "
import sys, json
data = json.load(sys.stdin)
pipelines = data.get('pipelines', [])
for p in pipelines:
    if p.get('name') == '${PIPELINE}':
        total = sum(len(q.get('heads', [])) for q in p.get('change_queues', []))
        print(total)
        sys.exit(0)
print(0)
" 2>/dev/null || echo 0)

QUEUE_DEPTH="${QUEUE_DEPTH:-0}"

if [ "$QUEUE_DEPTH" -le "$MAX_QUEUE_DEPTH" ]; then
  echo "Zuul ${PIPELINE} pipeline queue OK: ${QUEUE_DEPTH} changes"
  curl -s "$HEARTBEAT_URL"
else
  echo "Zuul ${PIPELINE} pipeline queue backed up: ${QUEUE_DEPTH} changes (max: ${MAX_QUEUE_DEPTH})"
  exit 1
fi

Run this every 5 minutes. A gate queue that never drains during off-peak hours is a strong signal that a blocking change is causing all subsequent changes to fail speculative execution — check the oldest change in the gate queue first.

Step 4: Track Job Execution Latency

Zuul's speculative execution runs multiple jobs in parallel, but each job still consumes time waiting for a node from the nodepool. When nodepool has no available nodes — because the cloud quota is exhausted, instance launch failures are accumulating, or the nodepool launcher itself is stuck — jobs wait indefinitely in "waiting for node" state. Monitor build latency to detect node provisioning problems:

#!/bin/bash
# /usr/local/bin/zuul-latency-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
ZUUL_URL="https://zuul.your-org.internal"
TENANT="your-tenant-name"
MAX_WAIT_MINUTES=30   # Alert if jobs are waiting longer than 30 minutes
TIMEOUT=15

# Get current builds and their wait times
BUILDS=$(curl -sf \
  --max-time "$TIMEOUT" \
  "${ZUUL_URL}/api/tenant/${TENANT}/builds?complete=false&limit=50" 2>/dev/null)

if [ -z "$BUILDS" ]; then
  echo "Could not fetch build data from Zuul API"
  curl -s "$HEARTBEAT_URL"  # Don't alert on API fetch failure — use the API monitor for that
  exit 0
fi

MAX_WAIT=$(echo "$BUILDS" | python3 -c "
import sys, json
from datetime import datetime, timezone

builds = json.load(sys.stdin)
now = datetime.now(timezone.utc)
max_wait = 0

for b in builds:
    if b.get('start_time') is None and b.get('event_timestamp'):
        try:
            queued_at = datetime.fromisoformat(b['event_timestamp'].replace('Z', '+00:00'))
            wait_min = (now - queued_at).total_seconds() / 60
            max_wait = max(max_wait, wait_min)
        except:
            pass

print(int(max_wait))
" 2>/dev/null || echo 0)

MAX_WAIT="${MAX_WAIT:-0}"

if [ "$MAX_WAIT" -le "$MAX_WAIT_MINUTES" ]; then
  echo "Zuul job wait time OK: max ${MAX_WAIT} min"
  curl -s "$HEARTBEAT_URL"
else
  echo "Zuul jobs stalled: longest wait ${MAX_WAIT} min (threshold: ${MAX_WAIT_MINUTES} min)"
  exit 1
fi

Set the Vigilmon heartbeat to 35 minutes. Jobs waiting over 30 minutes for a node nearly always indicate nodepool exhaustion — check your cloud provider's quota and the nodepool launcher logs.

Step 5: Monitor Zookeeper Connectivity

Zuul relies entirely on Apache Zookeeper for distributed state — pipeline queues, executor assignments, build results, and scheduler elections are all stored in Zookeeper. If Zuul loses its Zookeeper session (network partition, Zookeeper node failure, session timeout), the scheduler stops processing new triggers and executors abandon in-flight builds. Monitor Zookeeper health directly:

#!/bin/bash
# /usr/local/bin/zuul-zookeeper-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
ZK_HOST="zookeeper.your-org.internal"
ZK_PORT="2181"
TIMEOUT=10

# Check Zookeeper using 'ruok' four-letter command — returns 'imok' if healthy
ZK_RESPONSE=$(echo "ruok" | timeout "$TIMEOUT" nc -q 1 "$ZK_HOST" "$ZK_PORT" 2>/dev/null)

if [ "$ZK_RESPONSE" = "imok" ]; then
  echo "Zookeeper healthy on ${ZK_HOST}:${ZK_PORT}"

  # Also check the mode to ensure quorum is established
  ZK_STAT=$(echo "stat" | timeout "$TIMEOUT" nc -q 1 "$ZK_HOST" "$ZK_PORT" 2>/dev/null)
  ZK_MODE=$(echo "$ZK_STAT" | grep "Mode:" | awk '{print $2}')

  if [ "$ZK_MODE" = "leader" ] || [ "$ZK_MODE" = "follower" ] || [ "$ZK_MODE" = "standalone" ]; then
    echo "Zookeeper mode: ${ZK_MODE}"
    curl -s "$HEARTBEAT_URL"
  else
    echo "Zookeeper in unexpected mode: ${ZK_MODE}"
    exit 1
  fi
else
  echo "Zookeeper not responding with 'imok': got '${ZK_RESPONSE}'"
  exit 1
fi

For Zookeeper with AdminServer enabled:

# For Zookeeper AdminServer HTTP endpoint (available since ZK 3.5+)
ZK_ADMIN_URL="http://zookeeper.your-org.internal:8080/commands/stat"
ZK_STATUS=$(curl -sf --max-time 10 "$ZK_ADMIN_URL" 2>/dev/null)

if echo "$ZK_STATUS" | python3 -c "
import sys, json
data = json.load(sys.stdin)
sys.exit(0 if data.get('error') is None else 1)
" 2>/dev/null; then
  echo "Zookeeper AdminServer healthy"
  curl -s "$HEARTBEAT_URL"
else
  echo "Zookeeper AdminServer reporting errors"
  exit 1
fi

Run this check every 2 minutes — Zookeeper session loss is the fastest path from "healthy Zuul" to "completely stalled CI."

Step 6: Set Up Alert Channels

Configure Vigilmon to route Zuul alerts to the right teams:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #ci-ops for all Zuul monitors — gating failures affect every engineer submitting code.
  3. Add PagerDuty for the Zookeeper monitor and executor monitor — these cause complete gating stoppage.
  4. Add email for pipeline queue depth — a backed-up queue is serious but not immediately catastrophic.
  5. Set Consecutive failures before alert to 2 for the queue depth monitor — transient spikes during mass-merge windows are normal.

Include pipeline context in your Slack notifications:

PIPELINE_URL="https://zuul.your-org.internal/t/your-tenant-name/status"
ALERT_MSG=":stop_sign: Zuul CI gate pipeline is stalled — *${QUEUE_DEPTH}* changes waiting. Check: ${PIPELINE_URL}"

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-type: application/json' \
  --data "{\"text\": \"$ALERT_MSG\", \"channel\": \"#ci-ops\"}"

Add maintenance windows during Zuul scheduler restarts:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "ZUUL_API_MONITOR_ID",
    "duration_minutes": 15,
    "reason": "Zuul scheduler restart for config reload"
  }'

sudo systemctl restart zuul-scheduler

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (API) | /api/info keyword check | Scheduler crash, web API outage | | Cron heartbeat (executors) | /api/components executor state | All executors offline, Ansible executor failures | | Cron heartbeat (queue depth) | Pipeline status endpoint | Gate queue stall, blocking change, executor starvation | | Cron heartbeat (latency) | In-flight builds without start time | Nodepool exhaustion, instance launch failures | | Cron heartbeat (Zookeeper) | ruok / AdminServer stat | Zookeeper quorum loss, session timeout |

Zuul's gating model is only as reliable as its infrastructure stack. When Zookeeper, executors, or nodepool fail, the system silently stops merging code — there are no automated alerts and no fallback path. Vigilmon gives you external visibility into every layer of the Zuul stack so you know about infrastructure failures within minutes, not after your team notices that no PRs have merged in hours.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles