tutorial

Monitoring Bazel Builds with Vigilmon

Bazel's remote execution backend and build cache are what make polyglot monorepos build at scale — but when RBE workers are unavailable, cache hit rates fall, or action graph evaluation stalls, your CI throughput collapses silently. Here's how to monitor Bazel build health with Vigilmon.

Bazel is Google's open-source, polyglot build system capable of building and testing Java, C++, Go, Python, TypeScript, and more in a single hermetic monorepo. Its correctness guarantees and remote execution backend (RBE) make it the build system of choice for organizations with thousands of targets — but that same distributed architecture introduces multiple failure surfaces. Remote execution workers go unhealthy, build cache servers become unreachable, and action graph evaluation can stall under load. Vigilmon gives you the monitoring layer Bazel itself doesn't provide: HTTP health checks on your RBE backend, cron heartbeats for scheduled builds, and SSL certificate alerts for your cache infrastructure.

What You'll Set Up

  • HTTP health monitor for the Bazel remote execution backend
  • Build cache server health monitoring
  • Cron heartbeat for scheduled full-workspace builds and CI cache warm-ups
  • SSL certificate monitoring for RBE and cache servers
  • Maintenance windows for cache server upgrades and RBE worker deployments

Prerequisites

  • Bazel 6+ monorepo with a MODULE.bazel or WORKSPACE file
  • Remote execution backend (Buildbarn, BuildBuddy, Google RBE, or self-hosted)
  • Remote build cache (HTTP cache server or CAS backend)
  • CI pipeline running bazel build //... or bazel test //... on pull requests
  • A free Vigilmon account

Step 1: Monitor the Remote Execution Backend

Bazel's remote execution backend (RBE) is what makes distributed builds fast. When the backend is unavailable, Bazel falls back to local execution and your 10-minute build becomes a 2-hour build.

Common RBE providers expose health endpoints:

  • BuildBuddy: https://app.buildbuddy.io/health
  • Buildbarn (scheduler): https://scheduler.yourdomain.com/readyz
  • Google RBE: monitored via GCP status page — use TCP to verify your org's quota endpoint

Add a Vigilmon HTTP monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your RBE health endpoint URL.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

For RBE backends that only expose gRPC (no HTTP), add a TCP port monitor instead:

  1. Click Add Monitor → TCP Port.
  2. Enter the RBE scheduler hostname and gRPC port (typically 8980 or 443).
  3. Set Check interval to 1 minute.

Step 2: Monitor the Bazel HTTP Build Cache

Bazel supports an HTTP-based remote cache (--remote_cache=http://...) that's simpler to self-host than a full REAPI backend. This cache stores action results and output files — losing it means every build runs cold.

If you're running an HTTP cache server (nginx with WebDAV, Bazel's own reference server, or a custom implementation), add a health endpoint:

# nginx.conf — expose a health check alongside the WebDAV cache
server {
    listen 9090;
    location /health {
        return 200 '{"status":"ok"}';
        add_header Content-Type application/json;
    }
}

server {
    listen 8080;
    # WebDAV cache configuration
    root /var/cache/bazel;
    dav_methods PUT DELETE;
    dav_ext_methods PROPFIND OPTIONS;
    create_full_put_path on;
    dav_access user:rw group:r all:r;
}

For a custom Go-based cache server:

// health handler
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
    // Check disk space and storage accessibility
    var stat syscall.Statfs_t
    if err := syscall.Statfs(cacheDir, &stat); err != nil {
        http.Error(w, `{"status":"error"}`, http.StatusServiceUnavailable)
        return
    }
    freeGB := float64(stat.Bavail*uint64(stat.Bsize)) / 1e9
    if freeGB < 10 {
        http.Error(w, `{"status":"low_disk"}`, http.StatusServiceUnavailable)
        return
    }
    w.Header().Set("Content-Type", "application/json")
    fmt.Fprintf(w, `{"status":"ok","free_gb":%.1f}`, freeGB)
})

Point Vigilmon at https://bazel-cache.yourdomain.com:9090/health. The free disk space check ensures Vigilmon alerts before the cache fills up and starts evicting or failing writes.

Step 3: Monitor RBE Worker Availability

Bazel's distributed execution depends on healthy worker pools. Buildbarn and similar systems expose per-worker metrics — but a simpler signal is whether the scheduler can accept work.

Add a wrapper script that tests action dispatch and alerts if workers are unavailable:

#!/usr/bin/env python3
# scripts/check-rbe-workers.py
import subprocess
import sys
import os
import json
import urllib.request

VIGILMON_WEBHOOK = os.environ.get('VIGILMON_WEBHOOK_URL')
MIN_WORKER_COUNT = int(os.environ.get('MIN_RBE_WORKERS', '2'))

# Run a trivial Bazel build that uses RBE and check execution strategy
result = subprocess.run(
    ['bazel', 'build', '--config=remote', '--verbose_failures',
     '--execution_log_json_file=/tmp/exec_log.json', '//tools:noop'],
    capture_output=True, text=True
)

if result.returncode != 0:
    print(f"RBE build failed: {result.stderr}", file=sys.stderr)
    if VIGILMON_WEBHOOK:
        data = json.dumps({'event': 'rbe_build_failed', 'stderr': result.stderr[:500]}).encode()
        req = urllib.request.Request(VIGILMON_WEBHOOK, data=data,
                                     headers={'Content-Type': 'application/json'})
        urllib.request.urlopen(req)
    sys.exit(1)

# Parse execution log to count remote actions
with open('/tmp/exec_log.json') as f:
    remote_actions = sum(
        1 for line in f
        if '"type":"REMOTE"' in line or '"runner":"REMOTE"' in line
    )

print(f"Remote actions executed: {remote_actions}")
if remote_actions < 1 and VIGILMON_WEBHOOK:
    data = json.dumps({'event': 'rbe_zero_remote_actions'}).encode()
    req = urllib.request.Request(VIGILMON_WEBHOOK, data=data,
                                 headers={'Content-Type': 'application/json'})
    urllib.request.urlopen(req)

Run this script in CI alongside your build health checks. Zero remote actions on a build that should use RBE indicates workers fell back to local execution.

Step 4: Heartbeat Monitoring for Scheduled Full-Workspace Builds

Large Bazel monorepos typically run a nightly full build to warm the remote cache and catch integration failures before developers arrive. If the nightly pipeline silently fails, the next day's CI runs will be slower and may miss broken targets.

Set up a Vigilmon cron heartbeat:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 1440 minutes (24 hours).
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/bazel123.
  4. Wire it into your nightly build:
#!/bin/bash
set -euo pipefail

echo "Starting Bazel full-workspace build..."

# Build and test everything with remote execution
bazel build --config=remote //...
bazel test --config=remote //... --test_output=errors

echo "Bazel build complete"
# Signal success to Vigilmon — only fires if both commands succeeded
curl -fsS https://vigilmon.online/heartbeat/bazel123

GitHub Actions configuration:

name: Nightly Bazel build
on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC

jobs:
  full-build:
    runs-on: self-hosted
    timeout-minutes: 120
    steps:
      - uses: actions/checkout@v4
      - name: Set up Bazelisk
        uses: bazelbuild/setup-bazelisk@v3
      - name: Bazel build
        run: bazel build --config=remote //...
        env:
          BAZEL_REMOTE_CACHE: ${{ vars.BAZEL_REMOTE_CACHE }}
          BAZEL_RBE_ENDPOINT: ${{ vars.BAZEL_RBE_ENDPOINT }}
      - name: Bazel test
        run: bazel test --config=remote //... --test_output=errors
        env:
          BAZEL_REMOTE_CACHE: ${{ vars.BAZEL_REMOTE_CACHE }}
          BAZEL_RBE_ENDPOINT: ${{ vars.BAZEL_RBE_ENDPOINT }}
      - name: Signal success to Vigilmon
        if: success()
        run: curl -fsS https://vigilmon.online/heartbeat/bazel123

Step 5: SSL Certificate Alerts for RBE and Cache Servers

Bazel uses gRPC over TLS to communicate with RBE workers and REAPI cache backends. An expired certificate causes Bazel to reject all remote connections with a TLS error, forcing every action to run locally.

Add SSL certificate monitoring in Vigilmon for each server:

  1. Open the HTTP monitor for your RBE health endpoint.
  2. Enable Monitor SSL certificate.
  3. Set Alert when certificate expires in less than 21 days.
  4. Repeat for the build cache health endpoint if it's on a separate domain.
  5. Click Save.

Check certificate expiry manually:

# RBE scheduler certificate
echo | openssl s_client -servername scheduler.yourdomain.com \
  -connect scheduler.yourdomain.com:443 2>/dev/null \
  | openssl x509 -noout -enddate

# Build cache certificate
echo | openssl s_client -servername bazel-cache.yourdomain.com \
  -connect bazel-cache.yourdomain.com:9090 2>/dev/null \
  | openssl x509 -noout -enddate

For mTLS configurations (where Bazel presents a client certificate to the RBE backend), also monitor the client certificate expiry:

openssl x509 -noout -enddate -in /etc/bazel/client.crt

Step 6: Maintenance Windows for RBE Worker Deployments

When deploying new RBE worker images or scaling the worker pool, add a maintenance window to suppress false-positive alerts during the rollout:

MONITOR_ID="your-monitor-id"
API_KEY="your-vigilmon-api-key"

# Open a 15-minute maintenance window before starting the rollout
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"monitor_id\": \"$MONITOR_ID\", \"duration_minutes\": 15}"

# Perform the RBE worker rolling update
kubectl set image deployment/rbe-worker \
  worker=gcr.io/yourproject/rbe-worker:v2.1.0
kubectl rollout status deployment/rbe-worker --timeout=10m

# Validate build still works with the new workers
bazel build --config=remote //tools:validate

echo "Rollout complete — maintenance window will expire automatically"

For major Bazel version upgrades that require output base migration (bazel clean --expunge), extend the window to 30 minutes to cover the clean build of critical targets.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health check | RBE scheduler /readyz | Remote execution backend down | | HTTP health check | Cache server /health | Build cache unreachable, low disk | | TCP port | gRPC endpoint (:8980) | Backend not accepting connections | | Cron heartbeat | Nightly build heartbeat URL | Missed full build, CI cron failure | | SSL certificate | RBE scheduler domain | TLS cert expiry blocking all remote actions | | SSL certificate | Cache server domain | TLS cert expiry breaking cache reads/writes | | Webhook | Zero remote actions | Silent fallback to local execution |

Bazel's power comes from its distributed execution model — but that distribution means more moving parts that can fail silently. With Vigilmon monitoring your RBE backend health, build cache availability, SSL certificates, and nightly build heartbeats, you'll catch infrastructure failures in minutes rather than discovering them when your CI queue has backed up and developers are waiting.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles