tutorial

Monitoring BuildKit (Docker Build System) with Vigilmon

BuildKit is Docker's modern build engine — but a hung build daemon, exhausted build cache, or resource-saturated host silently stalls every image build in your CI/CD pipeline. Here's how to monitor BuildKit daemon health, build cache utilization, concurrent build queue depth, build latency, and host resource pressure with Vigilmon.

BuildKit is the open-source build toolkit that powers docker build in modern Docker installations and CI environments. It runs as a persistent daemon (buildkitd) that handles build requests, manages a layer cache, and executes build instructions in parallel using a content-addressable storage backend. When buildkitd crashes, its cache grows stale and triggers massive image rebuilds, or the host's disk fills with unreferenced layers, every container build in your pipeline stalls — often silently. CI jobs report "Building image…" and hang indefinitely until they time out. Vigilmon gives you external monitoring for BuildKit daemon health, build cache pressure, concurrent build utilization, build execution latency, and host resource saturation so you catch build infrastructure failures before they cascade into blocked deployments.

What You'll Set Up

  • BuildKit daemon (buildkitd) process health via cron heartbeats
  • Build cache size and layer count monitoring
  • Concurrent build queue depth tracking
  • Build execution latency measurement
  • Host resource utilization (disk, CPU, memory)

Prerequisites

  • BuildKit 0.12+ installed (buildkitd running as a daemon or via Docker Desktop)
  • buildctl CLI available on the monitoring host
  • Access to the BuildKit socket (default: /run/buildkit/buildkitd.sock)
  • A free Vigilmon account

Step 1: Monitor BuildKit Daemon Health

BuildKit runs as a long-lived daemon process. When buildkitd crashes — due to OOM, a corrupted content-addressable store, or a gRPC server panic — all subsequent build requests immediately fail with "Cannot connect to the Docker daemon" or socket timeout errors. CI jobs that call docker build or buildctl build see connection refused and may retry indefinitely or fail with confusing errors. Monitor the daemon process directly:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 2 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the daemon health check script:

#!/bin/bash
# /usr/local/bin/buildkit-health-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
BUILDKIT_SOCKET="/run/buildkit/buildkitd.sock"
TIMEOUT=10

# Check buildkitd process
if ! pgrep -x buildkitd > /dev/null; then
  echo "buildkitd not running"
  exit 1
fi

# Verify the daemon responds over its socket
if buildctl --addr "unix://${BUILDKIT_SOCKET}" debug info \
    --timeout "${TIMEOUT}s" > /dev/null 2>&1; then
  echo "BuildKit daemon healthy"
  curl -s "$HEARTBEAT_URL"
else
  echo "BuildKit daemon not responding on socket"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/buildkit-health-check.sh
(crontab -l 2>/dev/null; echo "*/2 * * * * /usr/local/bin/buildkit-health-check.sh >> /var/log/buildkit-health.log 2>&1") | crontab -

If the heartbeat stops arriving, Vigilmon alerts within 2 minutes. A buildkitd that doesn't respond to buildctl debug info but still appears in pgrep output usually means the gRPC server has deadlocked — restart the daemon immediately.

Step 2: Monitor Build Cache Utilization

BuildKit maintains a content-addressable layer cache that dramatically accelerates repeated builds. When the cache grows unbounded — particularly on hosts that build many different images — it consumes disk space and eventually causes builds to fail with "no space left on device." Monitor cache size to catch this before it fills the disk:

#!/bin/bash
# /usr/local/bin/buildkit-cache-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
BUILDKIT_SOCKET="/run/buildkit/buildkitd.sock"
MAX_CACHE_GB=50   # Alert if cache exceeds 50 GB
MAX_CACHE_RECORDS=5000  # Alert if more than 5000 cache records

# Get cache disk usage (du on the BuildKit storage directory)
BUILDKIT_DATA_DIR="/var/lib/buildkit"
if [ -d "$BUILDKIT_DATA_DIR" ]; then
  CACHE_GB=$(du -s --block-size=1G "$BUILDKIT_DATA_DIR" 2>/dev/null | awk '{print $1}')
else
  # Docker Desktop embeds BuildKit; check via buildctl
  CACHE_GB=0
fi

CACHE_GB="${CACHE_GB:-0}"

# Get cache record count via buildctl
CACHE_RECORDS=$(buildctl --addr "unix://${BUILDKIT_SOCKET}" du 2>/dev/null | \
  grep -c "^ID:" || echo 0)

FAILED=0

if [ "$CACHE_GB" -ge "$MAX_CACHE_GB" ]; then
  echo "BuildKit cache too large: ${CACHE_GB}GB (max: ${MAX_CACHE_GB}GB)"
  FAILED=1
fi

if [ "$CACHE_RECORDS" -ge "$MAX_CACHE_RECORDS" ]; then
  echo "BuildKit cache record count high: ${CACHE_RECORDS} (max: ${MAX_CACHE_RECORDS})"
  FAILED=1
fi

if [ "$FAILED" -eq 0 ]; then
  echo "BuildKit cache OK: ${CACHE_GB}GB, ${CACHE_RECORDS} records"
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Create a Vigilmon heartbeat with a 15 minute expected interval. When the cache exceeds your threshold, run buildctl prune --keep-duration 24h to reclaim space without fully invalidating recent builds. Add this to a daily cron job to prevent unbounded growth.

Step 3: Monitor Concurrent Build Queue Depth

BuildKit handles concurrent builds using a worker pool. When more build requests arrive than there are available workers — common during peak CI hours when many PRs merge simultaneously — builds queue and wait for a slot. Monitor active builds versus worker capacity to catch queue saturation:

#!/bin/bash
# /usr/local/bin/buildkit-queue-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
BUILDKIT_SOCKET="/run/buildkit/buildkitd.sock"
MAX_QUEUE_DEPTH=8   # Alert if more than 8 builds are waiting
TIMEOUT=10

# Get BuildKit worker info including active and waiting jobs
WORKER_INFO=$(buildctl --addr "unix://${BUILDKIT_SOCKET}" debug workers \
  --timeout "${TIMEOUT}s" 2>/dev/null)

if [ -z "$WORKER_INFO" ]; then
  echo "Could not retrieve BuildKit worker info"
  exit 1
fi

# Count active builds
ACTIVE_BUILDS=$(buildctl --addr "unix://${BUILDKIT_SOCKET}" debug info \
  --timeout "${TIMEOUT}s" 2>/dev/null | \
  grep -c "^Build:" || echo 0)

# Check for OCI worker process count as a proxy for queue depth
WORKER_PROCS=$(pgrep -c "buildkit-runc\|runc" 2>/dev/null || echo 0)

if [ "$ACTIVE_BUILDS" -le "$MAX_QUEUE_DEPTH" ]; then
  echo "Build queue OK: ${ACTIVE_BUILDS} active builds"
  curl -s "$HEARTBEAT_URL"
else
  echo "Build queue backed up: ${ACTIVE_BUILDS} active builds (max: ${MAX_QUEUE_DEPTH})"
  exit 1
fi

For multi-node BuildKit deployments using moby/buildkit with multiple workers, track active build count across all workers:

# For a BuildKit instance exposed via TCP
BUILDKIT_ADDR="tcp://buildkit.internal:8080"

ACTIVE=$(buildctl --addr "$BUILDKIT_ADDR" debug info 2>/dev/null | \
  grep -E "^[[:space:]]+ID:" | wc -l)

echo "Active builds across all workers: $ACTIVE"

Set the Vigilmon heartbeat to 5 minutes. A queue that consistently runs at capacity indicates the buildkitd worker limit needs increasing — edit /etc/buildkit/buildkitd.toml and set [worker.oci] max-parallelism = <N>.

Step 4: Track Build Execution Latency

Build execution latency directly impacts CI cycle time. When BuildKit's layer cache is cold (after a host restart or cache prune), Docker base image pulls are slow, or the host's CPU is saturated, build times spike from seconds to minutes. Track build latency using a synthetic probe build:

#!/bin/bash
# /usr/local/bin/buildkit-latency-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
BUILDKIT_SOCKET="/run/buildkit/buildkitd.sock"
MAX_BUILD_SECONDS=60   # Alert if a simple cached build takes more than 60 seconds
PROBE_CONTEXT_DIR="/tmp/buildkit-probe"

# Create a minimal Dockerfile for latency measurement
mkdir -p "$PROBE_CONTEXT_DIR"
cat > "${PROBE_CONTEXT_DIR}/Dockerfile" <<'EOF'
FROM alpine:3.19
RUN echo "latency probe"
EOF

BUILD_START=$(date +%s)

# Run a minimal build with --no-cache to test actual build speed
buildctl --addr "unix://${BUILDKIT_SOCKET}" build \
  --frontend dockerfile.v0 \
  --local context="${PROBE_CONTEXT_DIR}" \
  --local dockerfile="${PROBE_CONTEXT_DIR}" \
  --opt no-cache=true \
  --output type=image,name=buildkit-probe:latest,push=false \
  --timeout 120s > /dev/null 2>&1
BUILD_RESULT=$?

BUILD_END=$(date +%s)
BUILD_DURATION=$(( BUILD_END - BUILD_START ))

if [ "$BUILD_RESULT" -ne 0 ]; then
  echo "BuildKit probe build failed after ${BUILD_DURATION}s"
  exit 1
fi

if [ "$BUILD_DURATION" -le "$MAX_BUILD_SECONDS" ]; then
  echo "BuildKit latency OK: ${BUILD_DURATION}s (max: ${MAX_BUILD_SECONDS}s)"
  curl -s "$HEARTBEAT_URL"
else
  echo "BuildKit latency high: ${BUILD_DURATION}s (max: ${MAX_BUILD_SECONDS}s)"
  exit 1
fi

rm -rf "$PROBE_CONTEXT_DIR"

Run this check every 10 minutes with a Vigilmon heartbeat at 15 minute expected interval. A synthetic build that probes only alpine:3.19 (which should be cached) isolates build daemon latency from image-specific issues. If the probe build is slow but your actual production builds are fast, the daemon's scheduler is the bottleneck.

Step 5: Monitor Host Resource Utilization

BuildKit's container build execution is CPU-intensive and disk-heavy. The buildkitd daemon, runc child processes for each build step, and the content-addressable storage layer all compete for the same host resources. Track host resource pressure to detect saturation before builds fail:

#!/bin/bash
# /usr/local/bin/buildkit-resources-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
CPU_THRESHOLD=90      # percent
MEMORY_THRESHOLD=85   # percent
DISK_THRESHOLD=80     # percent
BUILDKIT_DATA_DIR="/var/lib/buildkit"

# CPU utilization (1-minute average)
CPU_IDLE=$(vmstat 1 2 | tail -1 | awk '{print $15}')
CPU_USED=$(( 100 - CPU_IDLE ))

# Memory utilization
MEM_TOTAL=$(free | awk '/^Mem:/ {print $2}')
MEM_USED=$(free | awk '/^Mem:/ {print $3}')
MEM_PCT=$(( MEM_USED * 100 / MEM_TOTAL ))

# Disk where BuildKit stores its cache
BUILD_DISK=$(df "${BUILDKIT_DATA_DIR}" 2>/dev/null | \
  awk 'NR==2 {gsub(/%/,""); print $5}' || \
  df /var/lib/docker | awk 'NR==2 {gsub(/%/,""); print $5}')

FAILED=0

if [ "$CPU_USED" -ge "$CPU_THRESHOLD" ]; then
  echo "CPU saturated during builds: ${CPU_USED}% (threshold: ${CPU_THRESHOLD}%)"
  FAILED=1
fi

if [ "$MEM_PCT" -ge "$MEMORY_THRESHOLD" ]; then
  echo "Memory pressure: ${MEM_PCT}% (threshold: ${MEMORY_THRESHOLD}%)"
  FAILED=1
fi

if [ "$BUILD_DISK" -ge "$DISK_THRESHOLD" ]; then
  echo "Build storage disk full: ${BUILD_DISK}% (threshold: ${DISK_THRESHOLD}%)"
  FAILED=1
fi

if [ "$FAILED" -eq 0 ]; then
  echo "Resources OK: CPU ${CPU_USED}%, Mem ${MEM_PCT}%, Disk ${BUILD_DISK}%"
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Set the Vigilmon heartbeat to 5 minutes. Disk saturation is the most common BuildKit failure mode in production — the cache grows faster than old layers are pruned. Set gc.defaultKeepStorage in /etc/buildkit/buildkitd.toml to auto-prune when cache exceeds a size limit:

[worker.oci]
  gc = true
  gckeepstorage = 50000  # 50 GB in MB

[[worker.oci.gcpolicy]]
  keepDuration = 172800  # 48 hours in seconds
  filters = ["type==source.local"]

Step 6: Set Up Alert Channels

Configure Vigilmon to route BuildKit alerts to your platform team:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #ci-alerts — build failures block all image deployments.
  3. Add PagerDuty for the daemon health and latency monitors — these indicate complete build infrastructure failure.
  4. Add email notification for cache and resource monitors (capacity issues, not complete failure).
  5. Set Consecutive failures before alert to 2 for resource monitors — transient CPU spikes during a large build are normal.

Add maintenance windows during BuildKit upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "BUILDKIT_MONITOR_ID",
    "duration_minutes": 15,
    "reason": "buildkitd upgrade and cache warm-up"
  }'

# Stop daemon, upgrade, restart, and warm cache
systemctl stop buildkitd
# Upgrade buildkitd binary...
systemctl start buildkitd
# Pull common base images to warm the cache
docker pull alpine:3.19 node:20-alpine python:3.12-slim

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (daemon) | buildkitd socket probe | Daemon crash, gRPC deadlock, socket unavailable | | Cron heartbeat (cache) | Cache size & record count | Unbounded cache growth, imminent disk exhaustion | | Cron heartbeat (queue) | Active build count | Worker pool saturation, queue backup | | Cron heartbeat (latency) | Synthetic probe build | Performance regression, cold cache, CPU saturation | | Cron heartbeat (resources) | CPU / memory / disk | Host saturation, disk full during builds |

BuildKit's failure mode in CI is deceptive — the daemon disappears and docker build commands immediately fail with cryptic socket errors, but there is no native alerting or health endpoint. Vigilmon's external heartbeat monitoring fills that gap: you know about build infrastructure failures within minutes, not after a developer escalates a stalled deployment pipeline.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles