tutorial

How to Monitor Buildkite CI/CD Pipelines with Vigilmon

Buildkite is a hybrid CI/CD platform: the control plane (pipelines, builds, job scheduling) runs in Buildkite's cloud, while the actual build execution happe...

Buildkite is a hybrid CI/CD platform: the control plane (pipelines, builds, job scheduling) runs in Buildkite's cloud, while the actual build execution happens on your own infrastructure via self-hosted agents. This split architecture gives you control and flexibility — but it also means there are two distinct failure surfaces to watch.

Your Buildkite agents run on your servers, your cloud instances, or your Kubernetes cluster. When they go down, builds queue silently. When the machines hosting them run out of disk space or memory, agents crash. When network connectivity drops, agents disconnect without triggering any Buildkite notification. Meanwhile, your pipelines appear "queued" indefinitely, and the first signal your team gets is an engineer wondering why their PR hasn't gotten a build result in two hours.

This tutorial covers setting up external monitoring for your Buildkite infrastructure using Vigilmon to catch agent failures, infrastructure problems, and pipeline stalls before they derail your team.

Why Buildkite needs external monitoring

Buildkite's own dashboard and notifications cover pipeline-level events well — build failures, step timeouts, and deploy results. What they don't cover:

  • Agent host is unreachable — the VM or container running your Buildkite agent dies; Buildkite sees the agent as "lost" but doesn't page your ops team
  • All agents disconnected — if every agent for a queue goes offline, new builds queue without any alert; depending on your organization's settings, this might not generate any notification at all
  • Agent host is up but agent process crashed — the host is responding to health checks but the buildkite-agent process isn't running; builds queue silently
  • Artifact/cache storage unreachable — your S3 bucket or cache backend becomes unavailable; builds start but fail or hang on artifact upload
  • Webhook delivery fails — outgoing webhooks to your deployment systems stop working, silently breaking your CD pipeline

External monitoring fills these gaps by watching the infrastructure your Buildkite agents run on — independently of Buildkite itself.

What you'll need

  • One or more Buildkite agent hosts (VMs, bare metal, or Kubernetes pods)
  • Agents configured to run buildkite-agent start
  • A free Vigilmon account

Step 1: Add a health endpoint to your agent hosts

Buildkite agents don't expose a built-in HTTP health endpoint, but you can add one to each agent host easily. The simplest approach is a minimal HTTP server that checks whether the buildkite-agent process is running.

Option A: systemd + simple health check script

If you run Buildkite agents via systemd, add a lightweight health endpoint:

# /usr/local/bin/buildkite-health.sh
#!/bin/bash
# Returns 200 if buildkite-agent is running, 503 if not

AGENT_COUNT=$(pgrep -c buildkite-agent 2>/dev/null || echo 0)

if [ "$AGENT_COUNT" -gt 0 ]; then
    echo -e "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\r\n{\"status\":\"ok\",\"agents\":$AGENT_COUNT}"
else
    echo -e "HTTP/1.1 503 Service Unavailable\r\nContent-Type: application/json\r\n\r\n{\"status\":\"no_agents\",\"agents\":0}"
fi

Serve it with socat for a quick solution:

# Run this as a systemd service
socat TCP-LISTEN:8080,fork,reuseaddr EXEC:/usr/local/bin/buildkite-health.sh

Option B: Python one-liner health server

# /usr/local/bin/buildkite_health.py
import subprocess, json
from http.server import HTTPServer, BaseHTTPRequestHandler

class Health(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != '/health':
            self.send_response(404); self.end_headers(); return
        try:
            result = subprocess.run(['pgrep', '-c', 'buildkite-agent'],
                                    capture_output=True, text=True)
            count = int(result.stdout.strip() or 0)
            status = 200 if count > 0 else 503
        except Exception:
            count, status = 0, 503
        body = json.dumps({"status": "ok" if status == 200 else "no_agents",
                           "agents": count}).encode()
        self.send_response(status)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args): pass  # suppress access logs

HTTPServer(('0.0.0.0', 8080), Health).serve_forever()

# /etc/systemd/system/buildkite-health.service
[Unit]
Description=Buildkite Agent Health Endpoint
After=network.target

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/buildkite_health.py
Restart=always
User=nobody

[Install]
WantedBy=multi-user.target

systemctl enable --now buildkite-health

Verify it's working:

curl http://your-agent-host:8080/health
# {"status":"ok","agents":4}

Step 2: Set up HTTP monitoring in Vigilmon

With the health endpoint running, add each agent host to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://your-agent-host:8080/health
  4. Check interval: 1 minute
  5. Expected response:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

If you have multiple agent hosts (e.g., a pool of spot instances), add a monitor for each. Vigilmon's multi-region consensus means a single probe blip won't wake anyone up at 3 AM — a real outage is confirmed from multiple locations before alerting.

For agent hosts in private subnets (not directly reachable from the internet), either:

  • Place the health endpoint behind a reverse proxy or load balancer with a public address
  • Use a site monitoring agent running on your network that reports to Vigilmon

Step 3: Monitor agent host TCP connectivity

Beyond the application-level health endpoint, monitor the SSH port on your agent hosts. If a host goes completely unreachable (kernel panic, power failure, cloud instance termination), the SSH port will be the first thing to drop.

  1. In Vigilmon, go to Monitors → New Monitor
  2. Choose TCP Port
  3. Host: your-agent-host, Port: 22
  4. Save the monitor

This gives you a second signal: if the HTTP health check fails but the TCP check on port 22 also fails, the host itself is down (not just the agent process). If port 22 is reachable but the health check fails, the agent process crashed on an otherwise healthy host — a completely different remediation path.

Step 4: Monitor your artifact storage

Buildkite pipelines typically upload artifacts (test results, build outputs, logs) to S3 or another object store. If that storage becomes unreachable, builds will hang or fail on upload.

If you have a pre-signed URL or a public S3 health endpoint:

# Example: check that S3 bucket endpoint is reachable
curl -I https://your-bucket.s3.amazonaws.com/

Add an HTTP monitor in Vigilmon pointing to a known-good artifact URL or a canary file you upload as part of a scheduled pipeline. This catches S3 outages, IAM permission changes, and bucket policy issues before they start failing real builds.

Step 5: Configure alert channels

Buildkite infrastructure failures block every engineer waiting for CI results, so fast alerting is essential.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Add your platform/ops team email
  3. Assign to all Buildkite agent monitors

Slack webhook

  1. Go to Alert Channels → Add Channel → Webhook
  2. Paste your #ci-alerts Slack channel webhook
  3. Assign to your monitors

Example Vigilmon alert payload:

{
  "monitor_name": "Buildkite Agent Host - build-agent-01",
  "status": "down",
  "url": "http://build-agent-01.internal:8080/health",
  "started_at": "2026-07-02T14:22:00Z",
  "duration_seconds": 180
}

Use this payload to drive automated responses:

# Example: restart buildkite-agent service on alert via webhook handler
curl -X POST https://your-automation-host/restart \
  -H "Content-Type: application/json" \
  -d '{"host": "build-agent-01", "service": "buildkite-agent"}'

Step 6: Diagnose Buildkite agent failures

When Vigilmon fires an alert on an agent host, follow this runbook:

# 1. Check if the buildkite-agent process is running
systemctl status buildkite-agent
ps aux | grep buildkite-agent

# 2. Check agent logs for disconnect or error messages
journalctl -u buildkite-agent -n 100 --no-pager | grep -E "error|disconnect|panic"

# 3. Check system resources — agents die from disk full and OOM
df -h          # disk usage
free -h        # memory
top -bn1 | head -20

# 4. Check network connectivity to Buildkite
curl -I https://agent.buildkite.com/v3

# 5. Re-register the agent if the token is still valid
buildkite-agent start --token "$BUILDKITE_AGENT_TOKEN"

Common failure patterns and fixes

| Vigilmon alert | Probable cause | First action | |----------------|----------------|-------------| | HTTP 503 (agents: 0), TCP 22 up | Agent process crashed | systemctl restart buildkite-agent | | HTTP down, TCP 22 down | Host unreachable | Check cloud console, replace instance | | HTTP timeout, TCP 22 timeout | Network partition | Check VPC/security groups, cloud provider status | | HTTP 200 (agents: 0), builds queuing | All agents idle but not running jobs | Check queue assignment in Buildkite dashboard |

Step 7: Create a CI infrastructure status page

If your team runs multiple build pools (e.g., a general pool and a macOS pool for mobile builds), a shared status page lets anyone check infrastructure health quickly.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name: "CI/CD Infrastructure"
  3. Add monitors:
    • All agent host health endpoints
    • Artifact storage endpoint
  4. Group by pool type (Linux, macOS, GPU, etc.)
  5. Publish and share the URL in your engineering wiki

When CI is slow or builds are queuing, engineers can check the status page themselves rather than pinging the platform team.

Recommended monitoring setup for Buildkite

| Monitor | Type | What it detects | |---------|------|-----------------| | http://agent-host-01:8080/health | HTTP | Agent process alive and running | | agent-host-01:22 | TCP | Host reachable at all | | http://agent-host-02:8080/health | HTTP | Second agent host health | | https://artifact-bucket.s3.amazonaws.com/canary | HTTP | Artifact storage reachable |

What's next

With external monitoring covering your Buildkite agent infrastructure, you'll know about failures within a minute — not when your team's build queue shows "0 agents available" two hours later.

Next steps to harden your Buildkite reliability:

  • Auto-scaling alerts: if you use spot instances for agents, wire Vigilmon alerts into an auto-scaling group policy that replaces terminated instances automatically
  • Queue depth monitoring: Buildkite's API exposes scheduled_jobs_count; a custom script can poll this and push a synthetic metric to a Vigilmon HTTP endpoint — if the queue depth exceeds a threshold with zero builds running, something is wrong
  • Dead letter builds: periodically check for builds stuck in "scheduled" state for more than your typical build time; this is a leading indicator that agents are disconnected without Buildkite knowing

Start monitoring your Buildkite infrastructure in under 2 minutes at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles