Monitoring Buildbot with Vigilmon: Web UI, Builder Health, Worker Connectivity, Build Heartbeats & SSL Alerts

Buildbot is a Python-based open-source framework for automating builds, tests, and releases. Unlike opinionated CI systems, Buildbot is a programmable build master with workers that execute steps on any platform — Linux, macOS, Windows, or embedded systems. When you self-host Buildbot, you own the master's uptime and worker connectivity. If the buildmaster crashes, workers disconnect, or the database becomes unavailable, builds queue silently and your team is left guessing. Vigilmon adds external monitoring: web UI checks, builder health validation, worker connectivity heartbeats, build completion signals, and SSL certificate alerts.

What You'll Build

• An HTTP monitor for the Buildbot web UI
• A health check via the Buildbot REST API
• A builder status check confirming builders are active
• A worker connectivity heartbeat to catch disconnected workers
• A build completion heartbeat for end-to-end automation verification
• SSL certificate expiry alerts

Prerequisites

• Buildbot master installed and running (buildbot start)
• At least one Buildbot worker connected
• A domain configured for the Buildbot web interface with HTTPS
• A free account at vigilmon.online

Step 1: Monitor the Buildbot Web UI

The Buildbot waterfall or grid view is the primary interface for viewing build status, step logs, and build history. A down web UI leaves your team without visibility:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://buildbot.yourdomain.com/#/
3. Check interval: 120 seconds.
4. Expected status: 200.
5. Label: Buildbot Web UI
6. Click Save.

Buildbot's web interface is served by the www plugin on the master process. A 200 response confirms the buildmaster is alive and the Twisted web server is accepting connections.

Step 2: Check the Buildbot API Health

Buildbot's REST API at /api/v2/ is the backbone of the web UI and all external integrations. Monitoring it directly catches master failures that the frontend may mask:

1. Add Monitor → HTTP.
2. URL: https://buildbot.yourdomain.com/api/v2/
3. Check interval: 60 seconds.
4. Expected status: 200.
5. Keyword: meta (confirms the API response includes the metadata field).
6. Label: Buildbot REST API
7. Click Save.

# Verify the API manually
curl -s https://buildbot.yourdomain.com/api/v2/ | python3 -m json.tool
# Returns: {"meta": {"total": N, "limit": 100, ...}}

The /api/v2/ root endpoint exercises the Twisted web server, the database layer, and the REST framework simultaneously. A 500 or empty response signals a subsystem failure requiring immediate investigation.

Step 3: Monitor Builder Status

Buildbot builders define what gets built and how. A builder in a failing or offline state means that category of builds will never run. Monitor builder health with a periodic script:

#!/bin/bash
# /etc/cron.d/buildbot-builders — runs every 5 minutes

ACTIVE=$(curl -s \
  https://buildbot.yourdomain.com/api/v2/builders \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
builders = data.get('builders', [])
# Count builders that are not in an inactive/offline state
active = [b for b in builders if b.get('description') or b.get('name')]
print(len(active))
" 2>/dev/null || echo "0")

if [ "${ACTIVE:-0}" -ge 1 ] 2>/dev/null; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-BUILDERS-HEARTBEAT-ID
fi

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: 5 minutes.
3. Grace period: 15 minutes.
4. Label: Buildbot Builders Active

This confirms that builders are configured and reachable. If a master reconfiguration (buildbot reconfig) removes all builders or enters an error state, the heartbeat stops.

Step 4: Worker Connectivity Heartbeat

Buildbot workers execute the actual build steps. A disconnected worker means all builds dispatched to it queue indefinitely. Monitor worker connectivity via the API:

#!/bin/bash
# /etc/cron.d/buildbot-workers — runs every 5 minutes

CONNECTED=$(curl -s \
  https://buildbot.yourdomain.com/api/v2/workers \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
workers = data.get('workers', [])
# connected_to is a list; non-empty means the worker is connected
connected = [w for w in workers if w.get('connected_to')]
print(len(connected))
" 2>/dev/null || echo "0")

if [ "${CONNECTED:-0}" -ge 1 ] 2>/dev/null; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-WORKER-HEARTBEAT-ID
fi

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: 5 minutes.
3. Grace period: 15 minutes.
4. Label: Buildbot Worker Connectivity

The connected_to field in the Buildbot API indicates which masters each worker is currently connected to. An empty list means the worker has disconnected — its builds will queue and never execute.

Step 5: SSL Certificate Alerts

Buildbot's web interface and worker-to-master communication both rely on TLS. An expired certificate breaks browser access and may cause worker reconnection failures depending on your TLS configuration:

1. Add Monitor → SSL Certificate.
2. Domain: buildbot.yourdomain.com
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days.
5. Click Save.

If Buildbot uses a self-signed certificate for worker connections (configured in buildbot.tac), also monitor that certificate's expiry through your certificate management tool. The Vigilmon SSL monitor checks the certificate your domain serves to browsers.

Step 6: Build Completion Heartbeat

Add a Vigilmon heartbeat step at the end of a critical Buildbot build to confirm end-to-end automation — not just that the master is running, but that it's completing successful builds:

# master.cfg

from buildbot.plugins import steps, util

factory = util.BuildFactory()

# Your regular build steps
factory.addStep(steps.Git(repourl='https://github.com/yourorg/yourrepo.git', mode='incremental'))
factory.addStep(steps.ShellCommand(command=["npm", "ci"]))
factory.addStep(steps.ShellCommand(command=["npm", "test"]))

# Vigilmon heartbeat — only runs when all prior steps succeed
factory.addStep(steps.ShellCommand(
    command=[
        "curl", "-fsS", "-X", "POST",
        "https://vigilmon.online/api/heartbeat/YOUR-BUILD-HEARTBEAT-ID"
    ],
    name="Vigilmon Heartbeat",
    description="Notify Vigilmon of successful build",
    haltOnFailure=False,   # Don't fail the build if the notification fails
    flunkOnFailure=False,
))

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: Match your trigger frequency (e.g., 60 minutes for hourly builds).
3. Grace period: 30 minutes.
4. Label: Buildbot Build Completion

Because Buildbot executes steps sequentially and stops on failure by default, the heartbeat step only runs when all build and test steps succeed. Consistent failures in preceding steps will silence the heartbeat and trigger a Vigilmon alert.

Step 7: Master Process Heartbeat

The Buildbot master runs as a long-lived Twisted process. Monitor the process directly from the host to catch crashes the web monitor might miss (e.g., if the reverse proxy continues serving cached responses):

#!/bin/bash
# /etc/cron.d/buildbot-master — runs every 5 minutes

MASTER_DIR="/path/to/your/buildbot/master"

if buildbot status "$MASTER_DIR" 2>/dev/null | grep -q "master is running"; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-MASTER-HEARTBEAT-ID
fi

Or using the PID file:

#!/bin/bash
# /etc/cron.d/buildbot-master — runs every 5 minutes

PID_FILE="/path/to/buildbot/master/twistd.pid"

if [ -f "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-MASTER-HEARTBEAT-ID
fi

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: 5 minutes.
3. Grace period: 15 minutes.
4. Label: Buildbot Master Process

The PID check confirms the master process is alive at the OS level — separate from whether the web interface is responding. This catches scenarios where the Twisted reactor enters an unresponsive state while the process technically remains running.

Common Buildbot Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Buildmaster process crash | Master process heartbeat and web UI monitor fire | | Database connection failure | REST API returns 500; API heartbeat stops | | All workers disconnect | Worker connectivity heartbeat stops | | Builder reconfiguration error | Builder status heartbeat stops | | Build queue stalls | Build completion heartbeat goes silent | | SSL certificate expires | SSL monitor alerts at 30 days | | Twisted reactor hangs | Master process heartbeat stops (PID alive, not responsive) | | Worker-to-master PB port blocked | Worker heartbeat stops; builds queue | | Builds consistently failing | Build completion heartbeat stops | | Reverse proxy misconfiguration | Web UI HTTP monitor fires |

Buildbot's flexibility makes it the right tool for complex, multi-platform build automation that opinionated CI systems can't handle. Vigilmon ensures that automation stays running — watching the master process, API layer, worker pool, build completion, and SSL certificate simultaneously so you know the moment your build infrastructure needs attention.

Start monitoring Buildbot in under 5 minutes — register free at vigilmon.online.