How to Monitor Earthly Builds and CI Pipelines with Vigilmon

Earthly is a reproducible build system that combines Makefile-style targets with Docker-layer caching to create portable, hermetic builds that run identically on a developer laptop and in CI. When Earthly is at the center of your delivery pipeline, build health becomes infrastructure health: a stalled build runner, a corrupted cache, or a registry push failure blocks every engineer from shipping.

In this tutorial you'll set up uptime and health monitoring for your Earthly build infrastructure using Vigilmon — free tier, no credit card required.

Why Earthly build infrastructure needs external monitoring

Earthly builds are deterministic, but the infrastructure they run on is not. CI runners go away, remote caches fill up and start rejecting writes, satellite instances lose connectivity to BuildKit, and artifact registries rate-limit pushes. None of these show up in Earthly's exit code until a developer's pipeline actually fails.

Key failure modes that CI dashboards miss:

BuildKit daemon unavailability — Earthly delegates container builds to BuildKit; when the daemon crashes or hangs, all builds queue indefinitely rather than failing fast
Remote cache saturation — Earthly's --remote-cache bucket fills and starts evicting critical layers; cache hit rate drops silently and build times double
Earthly Satellite connection failure — managed satellite instances lose their connection to the Earthly cloud; all builds fall back to local mode without warning
Registry push throttling — artifact registries (Docker Hub, GHCR, ECR) return 429s during push; Earthly retries silently but eventually times out, blocking releases
Shared BuildKit runner overload — a single BuildKit instance shared across many CI workers saturates CPU/memory; subsequent builds wait for slot availability
Cache import/export corruption — cache layers stored in S3 become corrupted or unreachable; Earthly falls back to full rebuilds, causing cascading CI timeouts

External monitoring catches these by watching the health APIs and build runner endpoints directly, before a blocked pipeline becomes an engineer's problem.

What you'll need

Earthly installed and configured in your CI environment (earthly CLI ≥ 0.7.0)
A BuildKit daemon or Earthly Satellite your CI connects to
A free Vigilmon account (sign up in 30 seconds)

Step 1: Expose a BuildKit health endpoint

Earthly wraps BuildKit, which exposes a health check gRPC service. Expose it as an HTTP endpoint for Vigilmon to poll:

# Check BuildKit version and health directly
buildctl --addr tcp://your-buildkit-host:1234 debug workers

Create a lightweight health proxy that queries BuildKit and returns HTTP:

# buildkit-health/app.py
import subprocess
import json
from flask import Flask, jsonify

app = Flask(__name__)
BUILDKITD_ADDR = "tcp://localhost:1234"

@app.route("/health")
def buildkit_health():
    try:
        result = subprocess.run(
            ["buildctl", "--addr", BUILDKITD_ADDR, "debug", "workers"],
            capture_output=True,
            text=True,
            timeout=10,
        )
        if result.returncode != 0:
            return jsonify({"status": "error", "detail": result.stderr.strip()}), 503

        # Parse worker output — at least one worker must be listed
        lines = [l for l in result.stdout.splitlines() if l.strip()]
        worker_count = len(lines) - 1  # subtract header line
        if worker_count < 1:
            return jsonify({"status": "degraded", "workers": 0}), 503

        return jsonify({"status": "ok", "workers": worker_count})
    except subprocess.TimeoutExpired:
        return jsonify({"status": "timeout", "detail": "buildctl timed out"}), 503
    except Exception as e:
        return jsonify({"status": "error", "detail": str(e)}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9200)

Run this sidecar on your BuildKit host:

pip install flask
python buildkit-health/app.py &

Step 2: Add a cache hit rate monitoring endpoint

Earthly's remote cache is critical for fast CI. Monitor cache effectiveness by tracking hit rates from recent builds:

#!/bin/bash
# cache-stats.sh — run after each Earthly build and POST stats to a collector
BUILD_LOG=$1
HITS=$(grep -c "CACHE HIT" "$BUILD_LOG" 2>/dev/null || echo 0)
MISSES=$(grep -c "CACHE MISS\|buildkit.cache.result" "$BUILD_LOG" 2>/dev/null || echo 0)
TOTAL=$((HITS + MISSES))

if [ "$TOTAL" -gt 0 ]; then
    RATE=$(echo "scale=2; $HITS * 100 / $TOTAL" | bc)
else
    RATE=0
fi

curl -s -X POST http://localhost:9201/cache-stats \
  -H "Content-Type: application/json" \
  -d "{\"hits\": $HITS, \"misses\": $MISSES, \"hit_rate\": $RATE}"

Collect and expose these stats:

# cache-stats-server/app.py
from flask import Flask, jsonify, request
from collections import deque
import threading

app = Flask(__name__)
lock = threading.Lock()
recent_builds = deque(maxlen=20)

@app.route("/cache-stats", methods=["POST"])
def record_stats():
    data = request.get_json()
    with lock:
        recent_builds.append(data)
    return jsonify({"ok": True})

@app.route("/health/cache")
def cache_health():
    with lock:
        builds = list(recent_builds)

    if not builds:
        return jsonify({"status": "unknown", "detail": "no recent builds"}), 200

    avg_hit_rate = sum(b.get("hit_rate", 0) for b in builds) / len(builds)

    if avg_hit_rate < 30:
        return jsonify({
            "status": "degraded",
            "avg_cache_hit_rate_pct": round(avg_hit_rate, 1),
            "detail": "cache hit rate below 30%",
        }), 503

    return jsonify({
        "status": "ok",
        "avg_cache_hit_rate_pct": round(avg_hit_rate, 1),
        "recent_builds": len(builds),
    })

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9201)

Step 3: Monitor Earthly Satellite connectivity

If you use Earthly Satellites (managed remote runners), monitor their status via the Earthly CLI:

# satellite-health/app.py
import subprocess
import json
from flask import Flask, jsonify

app = Flask(__name__)

@app.route("/health/satellite")
def satellite_health():
    try:
        result = subprocess.run(
            ["earthly", "satellite", "ls", "--json"],
            capture_output=True,
            text=True,
            timeout=15,
        )
        if result.returncode != 0:
            return jsonify({"status": "error", "detail": result.stderr.strip()}), 503

        satellites = json.loads(result.stdout or "[]")
        offline = [s["name"] for s in satellites if s.get("state") != "Operational"]

        if offline:
            return jsonify({
                "status": "degraded",
                "offline_satellites": offline,
            }), 503

        return jsonify({
            "status": "ok",
            "satellites": len(satellites),
        })
    except subprocess.TimeoutExpired:
        return jsonify({"status": "timeout"}), 503
    except Exception as e:
        return jsonify({"status": "error", "detail": str(e)}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9202)

Step 4: Add Vigilmon monitors

BuildKit health monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Earthly — BuildKit daemon | | URL | http://<buildkit-host>:9200/health | | Check interval | 1 minute | | HTTP method | GET | | Expected status | 200 | | Response must contain | "status":"ok" |

Cache health monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Earthly — remote cache health | | URL | http://<cache-stats-host>:9201/health/cache | | Check interval | 5 minutes | | Expected status | 200 | | Response must contain | "status":"ok" |

Satellite health monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Earthly — Satellite connectivity | | URL | http://<your-ci-host>:9202/health/satellite | | Check interval | 2 minutes | | Expected status | 200 | | Response must contain | "status":"ok" |

Step 5: Monitor the artifact registry push endpoint

Earthly builds are only useful if artifacts reach the registry. Monitor the registry's API endpoint to catch rate limits and outages before they block releases:

# For Docker Hub
curl -s https://hub.docker.com/v2/ | jq .

# For GHCR
curl -s https://ghcr.io/v2/ -I

Add a Vigilmon HTTP monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Artifact registry — GHCR availability | | URL | https://ghcr.io/v2/ | | Check interval | 2 minutes | | Expected status | 200 or 401 (401 means the API is up but unauthenticated) |

Step 6: Add a CI integration health endpoint

Add a step to your CI pipeline that POSTs build completion status to a collector, giving Vigilmon a time-based heartbeat:

# .github/workflows/build.yml (GitHub Actions)
name: Build

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Earthly
        uses: earthly/actions-setup@v1
        with:
          version: latest

      - name: Build
        id: earthly_build
        run: earthly +all

      - name: Report build status to Vigilmon heartbeat
        if: always()
        run: |
          STATUS="${{ steps.earthly_build.outcome }}"
          curl -s -X POST https://your-ci-health-server.example.com/ci-heartbeat \
            -H "Content-Type: application/json" \
            -d "{\"status\": \"$STATUS\", \"ref\": \"${{ github.ref }}\"}"

Your CI heartbeat server:

# ci-heartbeat/app.py
from flask import Flask, jsonify, request
import time

app = Flask(__name__)
last_success = {"time": None, "ref": None}
last_failure = {"time": None, "ref": None}

@app.route("/ci-heartbeat", methods=["POST"])
def ci_heartbeat():
    data = request.get_json()
    now = time.time()
    if data.get("status") == "success":
        last_success["time"] = now
        last_success["ref"] = data.get("ref")
    else:
        last_failure["time"] = now
        last_failure["ref"] = data.get("ref")
    return jsonify({"ok": True})

@app.route("/health/ci")
def ci_health():
    now = time.time()
    if last_success["time"] is None:
        return jsonify({"status": "unknown"}), 200

    minutes_since_success = (now - last_success["time"]) / 60
    if minutes_since_success > 120:
        return jsonify({
            "status": "stale",
            "minutes_since_last_success": round(minutes_since_success),
        }), 503

    return jsonify({
        "status": "ok",
        "last_success_ref": last_success["ref"],
        "minutes_since_last_success": round(minutes_since_success),
    })

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9203)

Step 7: Configure alerts

In Vigilmon, go to Alerts → Alert Channels and connect your notification channels (email, Slack, PagerDuty).

Apply an alert policy to each Earthly monitor:

| Setting | Recommended value | |---------|-------------------| | Alert after | 2 consecutive failures (for BuildKit — transient blips are common) | | Alert after | 1 failure (for satellite and cache monitors) | | Recovery alert | Enabled | | Alert channel | Your CI ops channel |

What you're monitoring now

| Monitor | What it detects | |---------|-----------------| | BuildKit daemon HTTP | Daemon crashes, worker unavailability, connection timeouts | | Cache health HTTP | Hit rate collapse, cache eviction, S3 connectivity loss | | Satellite health HTTP | Satellite offline, Earthly cloud connectivity failure | | Registry availability HTTP | Registry outages, rate limiting blocking artifact push | | CI heartbeat HTTP | Pipeline stalls, no successful builds in the last 2 hours |

Conclusion

Earthly makes builds reproducible, but it can't make the infrastructure underneath it reliable. A crashed BuildKit daemon, a full remote cache, or a throttled registry will silently block every build in your organization. Vigilmon monitors all of these from the outside, so you know about infrastructure failures before they become blocked deployments and frustrated engineers.