tutorial

Monitoring Buck2 with Vigilmon

Buck2 is Meta's open-source distributed build system — but when the Buck2 daemon crashes, its remote execution endpoint becomes unreachable, or its build cache fill up, builds silently fall back to local execution or fail entirely. Here's how to monitor Buck2 daemon health, remote execution endpoint availability, build cache hit rates, and CI build pipeline liveness with Vigilmon.

Buck2 is Meta's open-source, high-performance distributed build system written in Rust. It supports remote execution, distributed build caching via RE APIs (Remote Execution API, compatible with BuildBarn, Buildfarm, and EngFlow), and first-class integration with Starlark-based build rules. Large engineering organizations using Buck2 benefit from reproducible, hermetic builds and dramatic build time reductions through caching and distribution — but when the Buck2 daemon becomes unresponsive, a remote execution cluster drops a worker, or the cache backend becomes unreachable, builds silently degrade to slow local execution or fail with opaque errors. Vigilmon gives you external monitoring for Buck2 daemon health, remote execution endpoint availability, build cache backend reachability, and CI build pipeline liveness so build infrastructure failures surface within minutes rather than through developer frustration during a deploy.

What You'll Set Up

  • Buck2 daemon availability via health endpoint
  • Remote execution (RE) cluster endpoint reachability
  • Build cache backend (CAS) availability
  • CI build pipeline liveness via heartbeat
  • Build failure rate monitoring

Prerequisites

  • Buck2 installed and configured in your repository
  • Remote execution cluster (BuildBarn, Buildfarm, EngFlow, or similar) — optional but covered
  • Access to a CI host for cron scripts
  • A free Vigilmon account

Step 1: Monitor the Buck2 Daemon

Buck2 runs a long-lived daemon process (buck2d) that caches dependency graphs, manages build state, and handles remote execution connections. When the daemon crashes or becomes unresponsive, the next build invocation restarts it with a cold graph — no incremental build benefit and increased build latency. Monitor daemon liveness with a health check script:

#!/bin/bash
# /usr/local/bin/buck2-daemon-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
BUCK2_BIN="/usr/local/bin/buck2"   # Adjust to your Buck2 binary path
MAX_DAEMON_AGE_HOURS=48            # Alert if daemon hasn't been running continuously

# Check daemon status via buck2 status
STATUS=$("$BUCK2_BIN" status --json 2>/dev/null)

if [ $? -ne 0 ] || [ -z "$STATUS" ]; then
  echo "Buck2 daemon not running or buck2 status failed"
  exit 1
fi

RESULT=$(echo "$STATUS" | python3 -c "
import sys, json

try:
    data = json.load(sys.stdin)
except Exception:
    # buck2 status may not always return valid JSON
    print('ok')
    sys.exit(0)

process_info = data.get('process_info', {})
pid = process_info.get('pid')
start_time = process_info.get('start_time')

if not pid:
    print('no_pid')
    sys.exit(1)

print(f'pid={pid}')
" 2>/dev/null)

if [[ "$RESULT" == no_pid* ]] || [ -z "$RESULT" ]; then
  echo "Buck2 daemon not running: ${RESULT}"
  exit 1
fi

echo "Buck2 daemon OK: ${RESULT}"
curl -s "$HEARTBEAT_URL"

For monitoring on CI machines where Buck2 is invoked per-job (daemon may not always be running), skip this check and use the build pipeline heartbeat in Step 4 instead. The daemon health check is most valuable for persistent developer machines and dedicated build hosts where you expect the daemon to run continuously.

Schedule as a cron job with Vigilmon heartbeat expected interval of 5 minutes:

*/4 * * * * /usr/local/bin/buck2-daemon-check.sh >> /var/log/buck2-daemon-check.log 2>&1

Step 2: Monitor the Remote Execution Endpoint

Buck2's remote execution (RE) support dramatically accelerates build times by distributing compilation and test execution across a cluster. When the RE endpoint becomes unreachable — due to a scheduler crash, network partition, or TLS certificate expiry — Buck2 falls back to local execution. This fallback is silent: builds succeed but take significantly longer, and developers may not notice until the build time regression accumulates over days. Monitor the RE scheduler endpoint directly:

  1. In Vigilmon, click Add Monitor → TCP.
  2. Set Host to your RE scheduler hostname (e.g., re-scheduler.your-org.internal).
  3. Set Port to 8980 (default gRPC port for RE schedulers like BuildBarn) or 443 for HTTPS endpoints.
  4. Set Check interval to 1 minute.
  5. Set Timeout to 10 seconds.

For RE schedulers with an HTTP health or admin endpoint:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://re-scheduler.your-org.internal/readyz or https://re-scheduler.your-org.internal/metrics.
  3. Set Check interval to 1 minute.
  4. Under Expected response code, set to 200.
  5. Set Timeout to 10 seconds.

For BuildBarn specifically:

# BuildBarn scheduler health check
curl -sf --max-time 10 https://buildbarn.your-org.internal:7982/
# BuildBarn exposes a simple HTTP server on 7982 for admin/health

For EngFlow or similar commercial RE providers, use the health endpoint from their documentation or monitor the gRPC port with a TCP monitor.

Step 3: Monitor the Content Addressable Storage (CAS) Backend

Buck2's build cache stores compiled artifacts and action results in a Content Addressable Storage service. When the CAS backend becomes unreachable, Buck2 cannot upload or download cached artifacts — every build must recompile everything locally, turning 30-second incremental builds into 10-minute full rebuilds. Monitor CAS availability:

  1. In Vigilmon, click Add Monitor → TCP.
  2. Set Host to your CAS backend hostname.
  3. Set Port to 8980 (gRPC) or the HTTP port for your CAS implementation.
  4. Set Check interval to 2 minutes.

For CAS backends with HTTP health endpoints (e.g., remote-apis-testing, self-hosted S3-backed CAS):

#!/bin/bash
# /usr/local/bin/buck2-cas-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
CAS_HOST="cas.your-org.internal"
CAS_PORT="8980"

# Test TCP connectivity to CAS gRPC endpoint
if nc -z -w 10 "$CAS_HOST" "$CAS_PORT" 2>/dev/null; then
  echo "Buck2 CAS backend reachable at ${CAS_HOST}:${CAS_PORT}"
  curl -s "$HEARTBEAT_URL"
else
  echo "Buck2 CAS backend unreachable at ${CAS_HOST}:${CAS_PORT}"
  exit 1
fi

For Buck2 configurations using an HTTP proxy cache (like Bazel's cache protocol over HTTP), add an HTTP monitor:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://buck2-cache.your-org.internal/status.
  3. Set Expected response code to 200.
  4. Set Check interval to 2 minutes.

Step 4: Monitor Build Pipeline Liveness

Buck2 daemon health and RE endpoint availability confirm infrastructure is running — but not that builds are actually completing successfully. A broken Starlark rule, a misconfigured toolchain, or a changed cell root can cause all builds to fail even with healthy infrastructure. Monitor actual build success from CI:

#!/bin/bash
# /usr/local/bin/buck2-build-check.sh
# Run on CI or a dedicated build host with Buck2 configured

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
REPO_PATH="/path/to/your-repo"
PROBE_TARGET="//tools/probe:hello"   # A fast, minimal build target for liveness
MAX_BUILD_TIME=120                    # Alert if the probe build takes more than 120 seconds

cd "$REPO_PATH" || exit 1

START_TS=$(date +%s)

# Run a minimal build target to verify end-to-end build pipeline
if buck2 build "$PROBE_TARGET" --config client.id=vigilmon-probe 2>/tmp/buck2-probe.log; then
  END_TS=$(date +%s)
  BUILD_TIME=$((END_TS - START_TS))

  if [ "$BUILD_TIME" -le "$MAX_BUILD_TIME" ]; then
    echo "Buck2 build pipeline healthy: probe build completed in ${BUILD_TIME}s"
    curl -s "$HEARTBEAT_URL"
  else
    echo "Buck2 build pipeline slow: probe build took ${BUILD_TIME}s (max: ${MAX_BUILD_TIME}s)"
    # Still ping — slow build is not the same as broken build
    curl -s "$HEARTBEAT_URL"
  fi
else
  echo "Buck2 probe build failed:"
  tail -20 /tmp/buck2-probe.log
  exit 1
fi

Create a minimal probe target in your repository for this purpose:

# tools/probe/BUCK
genrule(
    name = "hello",
    out = "hello.txt",
    cmd = "echo 'buck2 probe ok' > $OUT",
)

Schedule every 10 minutes with a Vigilmon heartbeat expected interval of 12 minutes:

*/10 * * * * /usr/local/bin/buck2-build-check.sh >> /var/log/buck2-build-check.log 2>&1

Step 5: Monitor Build Cache Hit Rate

A healthy Buck2 setup with remote caching should see high cache hit rates on incremental builds. A sudden drop in cache hit rate indicates a broken cache configuration, expired credentials, a cache invalidation event (toolchain upgrade, compiler flag change), or a cache backend that is accepting writes but returning misses on reads. Track cache performance:

#!/bin/bash
# /usr/local/bin/buck2-cache-hit-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
REPO_PATH="/path/to/your-repo"
PROBE_TARGET="//tools/probe:hello"
MIN_CACHE_HIT_PCT=60   # Alert if cache hit rate falls below 60% on known-cached targets

cd "$REPO_PATH" || exit 1

# Build the probe target and capture RE stats
STATS=$(buck2 build "$PROBE_TARGET" \
  --config client.id=vigilmon-cache-probe \
  --show-output 2>&1)

# Extract RE action cache stats from buck2 output
HIT_COUNT=$(echo "$STATS" | grep -i "cache hits\|RE cache" | grep -oP '\d+(?= hit)' | head -1 || echo 0)
MISS_COUNT=$(echo "$STATS" | grep -i "cache miss\|RE miss" | grep -oP '\d+(?= miss)' | head -1 || echo 0)
TOTAL=$((HIT_COUNT + MISS_COUNT))

if [ "$TOTAL" -eq 0 ]; then
  # No RE stats available — local build or stats not in this buck2 version
  echo "No RE cache stats available — local build or cache not configured"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

HIT_PCT=$((HIT_COUNT * 100 / TOTAL))

if [ "$HIT_PCT" -ge "$MIN_CACHE_HIT_PCT" ]; then
  echo "Buck2 RE cache hit rate OK: ${HIT_PCT}% (${HIT_COUNT}/${TOTAL})"
  curl -s "$HEARTBEAT_URL"
else
  echo "Buck2 RE cache hit rate degraded: ${HIT_PCT}% (${HIT_COUNT}/${TOTAL}) — below ${MIN_CACHE_HIT_PCT}%"
  exit 1
fi

Note: Buck2's cache statistics output format varies by version. Adjust the parsing to match your Buck2 version's output. You can also use buck2 log what-ran on recent build logs to extract RE action stats programmatically.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Buck2 alerts to the build infrastructure team:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #build-infra or #eng-productivity — Buck2 failures affect developer velocity across the entire engineering organization.
  3. Add PagerDuty for the RE endpoint and CAS monitors — when these go offline, every build in the organization falls back to slow local execution.
  4. Add email for the cache hit rate monitor — a degraded cache is urgent but typically gives time for a non-emergency response.
  5. Set Consecutive failures before alert to 2 for the daemon check — daemon restarts during system maintenance cause single-probe gaps.

Add maintenance windows during RE cluster upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "BUCK2_RE_MONITOR_ID",
    "duration_minutes": 30,
    "reason": "BuildBarn RE cluster upgrade — brief connectivity interruption expected"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (daemon) | buck2 status check | Daemon crash, unresponsive daemon, cold graph rebuild | | TCP monitor (RE scheduler) | RE scheduler gRPC port | Remote execution cluster unreachable | | TCP/HTTP monitor (CAS) | CAS gRPC/HTTP endpoint | Cache backend unreachable — full rebuilds on every run | | Cron heartbeat (build pipeline) | Probe target build | End-to-end build broken, toolchain misconfiguration | | Cron heartbeat (cache hit rate) | RE cache statistics | Cache backend failing reads, credential expiry, invalidation |

Buck2 builds trust in engineering organizations by making builds fast, reproducible, and scalable. When the remote execution cluster becomes unreachable, the CAS backend fails writes, or the daemon crashes on every build host, those build time guarantees evaporate — developers see slow builds, CI times spike, and the team loses confidence in the build system. Vigilmon gives you external monitoring for every layer of Buck2's build infrastructure so failures surface within minutes rather than through slow accumulation of developer complaints.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles