tutorial

Monitoring Buildah Container Builds with Vigilmon

Buildah builds OCI images without a daemon — but silent build failures, stale layer caches, and slow registry pushes can break your CI pipelines invisibly. Here's how to monitor Buildah build health, cache efficiency, and registry push latency with Vigilmon.

Buildah is a daemonless OCI image build tool that gives you rootless, scriptable container builds without the Docker daemon overhead. CI pipelines running Buildah look clean when things work — but when layer cache storage corrupts, registry push latency spikes, or rootless user namespace configuration breaks, builds either fail silently or waste time without a clear root cause. Vigilmon adds the external monitoring layer that catches build failure rate spikes, cache health degradation, and registry connectivity issues before they block your team's deployments.

What You'll Set Up

  • Build success rate tracking via cron heartbeats
  • Layer cache health monitoring
  • Registry push latency and availability checks
  • Rootless build environment validation
  • CI pipeline integration with maintenance windows

Prerequisites

  • Buildah installed (1.30+ recommended)
  • A container registry accessible from your build environment
  • A free Vigilmon account

Step 1: Track Build Success Rates with Heartbeats

Buildah builds run as part of CI jobs, not as persistent services — so there's no HTTP endpoint to probe. Use Vigilmon's cron heartbeat to confirm that builds are succeeding at the expected frequency.

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to match your build frequency (e.g., 60 minutes for an hourly CI trigger).
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Add the heartbeat call to your build script, after a successful build:

#!/bin/bash
set -e

IMAGE_NAME="myrepo/myapp"
TAG="${GIT_COMMIT:-latest}"

# Build the image
buildah bud \
  --tag "$IMAGE_NAME:$TAG" \
  --tag "$IMAGE_NAME:latest" \
  --file Dockerfile \
  .

# Push to registry
buildah push "$IMAGE_NAME:$TAG"
buildah push "$IMAGE_NAME:latest"

# Signal success to Vigilmon
curl -s https://vigilmon.online/heartbeat/abc123

The set -e ensures the heartbeat only fires on fully successful builds. If buildah bud fails (syntax error, missing base image, build step failure) or buildah push fails (registry error, auth failure), the script exits before the ping — and Vigilmon alerts after the expected interval.

Step 2: Monitor Registry Availability and Push Latency

Build pipelines that push to a private registry depend on registry availability. Registry outages or high push latency can stall builds indefinitely if there's no timeout enforcement.

Add an HTTP monitor for your registry's health endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: your registry's health endpoint, for example:
    • Docker Hub: https://index.docker.io/v2/
    • Quay.io: https://quay.io/health
    • Your private registry: https://registry.yourdomain.com/v2/
  3. Set Check interval to 2 minutes.
  4. Set Expected HTTP status to 200 (or 401 for Docker's v2 API, which returns 401 for unauthenticated requests but means the registry is up).
  5. Enable Monitor SSL certificate with a 21 day alert threshold.
  6. Click Save.

For a private registry behind your firewall, add a TCP port monitor instead of HTTP if the API isn't publicly accessible:

  1. Click Add Monitor → TCP Port.
  2. Host: your registry hostname.
  3. Port: 5000 (Docker's default) or 443 for TLS.
  4. Interval: 1 minute.

Step 3: Layer Cache Health Monitoring

Buildah stores layer cache on the local filesystem (typically under /var/lib/containers for root builds or ~/.local/share/containers for rootless). Cache corruption or unexpected cache pruning causes builds to pull all base layers on every run, dramatically increasing build times and registry bandwidth.

Add a cache health heartbeat:

#!/bin/bash
# /usr/local/bin/buildah-cache-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
CACHE_THRESHOLD=10  # Minimum expected cached images

# Count images in local storage
CACHED_COUNT=$(buildah images --quiet 2>/dev/null | wc -l)

# Check storage metadata integrity
if buildah images >/dev/null 2>&1 && [ "$CACHED_COUNT" -ge "$CACHED_THRESHOLD" ]; then
  curl -s "$HEARTBEAT_URL"
else
  echo "Cache issue: $CACHED_COUNT images found (expected >= $CACHED_THRESHOLD)"
fi

Set the Vigilmon cron heartbeat to a 30 minute interval. Run this script before your build jobs to catch cache corruption early.

To also monitor disk space for the cache directory:

# Add to the cache check script
CACHE_DIR="${XDG_DATA_HOME:-$HOME/.local/share}/containers"
DISK_USED=$(df "$CACHE_DIR" | awk 'NR==2 {gsub(/%/, ""); print $5}')

if [ "$DISK_USED" -gt 85 ]; then
  echo "Cache disk ${DISK_USED}% full — builds may fail"
  exit 1
fi

High cache disk usage typically precedes build failures by hours — catching it here prevents emergency cleanup during a deployment.

Step 4: Validate the Rootless Build Environment

Rootless Buildah requires user namespace support (/proc/sys/user/max_user_namespaces), a valid /etc/subuid and /etc/subgid configuration, and the newuidmap/newgidmap binaries. System updates, kernel version changes, or container runtime package updates can break this configuration silently.

Add a rootless environment validation heartbeat:

#!/bin/bash
# /usr/local/bin/buildah-rootless-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
BUILD_USER="ci"  # The user running rootless builds

# Check user namespace support
MAX_NS=$(cat /proc/sys/user/max_user_namespaces 2>/dev/null || echo 0)
if [ "$MAX_NS" -lt 1000 ]; then
  echo "User namespaces not available or insufficient: $MAX_NS"
  exit 1
fi

# Check subuid/subgid configuration
if ! grep -q "^${BUILD_USER}:" /etc/subuid; then
  echo "No subuid entry for ${BUILD_USER}"
  exit 1
fi

if ! grep -q "^${BUILD_USER}:" /etc/subgid; then
  echo "No subgid entry for ${BUILD_USER}"
  exit 1
fi

# Validate with a minimal rootless build
if buildah from --quiet scratch >/dev/null 2>&1; then
  buildah rm -a >/dev/null 2>&1
  curl -s "$HEARTBEAT_URL"
else
  echo "Rootless build test failed"
  exit 1
fi

Set a Vigilmon cron heartbeat with a 1 hour interval. Schedule this script to run before your CI build windows to catch environment regressions from OS or kernel updates.

Step 5: Monitor Build Duration and Detect Slowdowns

Sudden build slowdowns (not failures) are often the first signal of underlying problems: cache misses pulling gigabytes of layers, network congestion to the registry, or storage I/O contention. Track build duration by comparing against a baseline:

#!/bin/bash
# build-with-timing.sh

IMAGE_NAME="myrepo/myapp"
TAG="${GIT_COMMIT:-latest}"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
SLOW_THRESHOLD=300  # Alert if build takes more than 5 minutes

START=$(date +%s)

buildah bud --tag "$IMAGE_NAME:$TAG" --file Dockerfile . || exit 1
buildah push "$IMAGE_NAME:$TAG" || exit 1

END=$(date +%s)
DURATION=$(( END - START ))

if [ "$DURATION" -gt "$SLOW_THRESHOLD" ]; then
  echo "Slow build: ${DURATION}s (threshold: ${SLOW_THRESHOLD}s)"
  # Still ping so you know the build succeeded, but log the slowdown
  # You can also skip the ping to treat slow builds as failures
fi

curl -s "$HEARTBEAT_URL"

# Log build time for trend analysis
echo "BUILD_DURATION=$DURATION BUILD_TAG=$TAG" >> /var/log/buildah-metrics.log

Pair this with a second Vigilmon heartbeat monitor with a tighter interval (e.g., 30 minutes) that this script only pings when builds complete within the expected duration. Two heartbeats — one for success, one for timely success — gives you both availability and performance signals.

Step 6: CI/CD Pipeline Integration

Buildah runs inside CI — integrate Vigilmon into your pipeline for complete visibility:

# .github/workflows/build.yaml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build with Buildah
        run: |
          buildah bud \
            --tag ${{ env.IMAGE_NAME }}:${{ github.sha }} \
            --file Dockerfile \
            .

      - name: Push to registry
        run: |
          echo "${{ secrets.REGISTRY_PASSWORD }}" | \
            buildah login -u "${{ secrets.REGISTRY_USER }}" --password-stdin registry.yourdomain.com
          buildah push ${{ env.IMAGE_NAME }}:${{ github.sha }}

      - name: Signal success to Vigilmon
        if: success()
        run: curl -s ${{ secrets.VIGILMON_HEARTBEAT_URL }}

      - name: Open maintenance window for deploy
        if: success()
        run: |
          curl -X POST https://vigilmon.online/api/maintenance \
            -H "Authorization: Bearer ${{ secrets.VIGILMON_API_KEY }}" \
            -d '{"monitor_id": "${{ secrets.DEPLOY_MONITOR_ID }}", "duration_minutes": 5}'

Store the Vigilmon heartbeat URL and API key in CI secrets. The if: success() condition ensures the heartbeat only fires on clean builds — failed pipelines leave it unfired, triggering the Vigilmon alert after the expected interval.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (build) | Build success ping | Build failures, script errors | | HTTP health | Registry /v2/ endpoint | Registry outage, auth failures | | TCP port | Registry port | Network connectivity loss | | Cron heartbeat (cache) | Cache integrity check | Cache corruption, disk full | | Cron heartbeat (rootless) | Environment validation | Kernel/OS regression breaking namespaces | | Cron heartbeat (timing) | Duration-gated ping | Cache miss slowdowns, network congestion |

Buildah's daemonless architecture removes one class of failure (daemon crashes) but introduces others: rootless namespace configuration, local cache health, and tighter coupling to kernel features. Vigilmon gives your build pipeline the external health checks to catch all of these before they silently fail in production — or worse, succeed slowly enough that nobody notices until the deployment timeline slips.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles