tutorial

Monitoring Pants Build System with Vigilmon

Pants v2 is a scalable monorepo build system built for large Python, Java, and Go codebases — but when the Pants daemon stalls, remote caching becomes unreachable, or dependency resolution regresses, builds silently slow down. Here's how to monitor Pants build health with Vigilmon.

Pants (v2) is a polyglot build system designed for large-scale monorepos, offering fine-grained dependency resolution, remote execution, and a persistent daemon that avoids JVM startup overhead on every invocation. It's favored by Python and Java-heavy organizations that need reproducible builds across hundreds of packages. But Pants' distributed nature — a local daemon, optional remote caching, and remote execution clusters — creates multiple failure surfaces. Vigilmon gives you the observability to catch those failures: HTTP health checks on remote cache backends, cron heartbeats on scheduled builds, and TCP monitoring on the daemon's process port.

What You'll Set Up

  • HTTP health monitor for the Pants remote cache backend
  • Cron heartbeat to confirm scheduled full-workspace builds complete on time
  • TCP port monitor for the Pants daemon health port
  • SSL certificate monitoring for remote cache servers
  • API-driven maintenance windows for cache infrastructure upgrades

Prerequisites

  • Pants v2 (2.x) monorepo with pants.toml configured
  • Remote cache backend (remote caching via gRPC or HTTP) — optional but recommended
  • CI pipeline running pants test :: or pants lint :: on pull requests
  • A free Vigilmon account

Step 1: Monitor the Remote Cache Backend

Pants supports remote caching via the Remote Execution API (REAPI), typically backed by Buildbarn, BuildBuddy, or a custom gRPC server. When the cache backend is unreachable, Pants falls back to local execution and build times can increase dramatically.

If your remote cache exposes an HTTP health endpoint, add a Vigilmon HTTP monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your cache server's health URL. For BuildBuddy: https://app.buildbuddy.io/health. For self-hosted Buildbarn: https://buildbarn.yourdomain.com/readyz.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

For pure gRPC-based backends without an HTTP health endpoint, monitor the TCP port instead:

  1. Click Add Monitor → TCP Port.
  2. Enter the hostname and port of your cache backend (typically 8980 for gRPC or 443 for gRPC-over-TLS).
  3. Set Check interval to 1 minute.

This confirms the backend is at least accepting connections, even if it doesn't expose an HTTP probe.

Step 2: Add a Health Endpoint to a Self-Hosted Cache Server

If you're operating a self-hosted Pants-compatible cache server, add an HTTP health probe that Vigilmon can query without authentication:

# health_server.py (minimal aiohttp example)
from aiohttp import web
import aiobotocore

async def health_handler(request):
    try:
        # Check S3 bucket accessibility (used as cache storage)
        session = aiobotocore.get_session()
        async with session.create_client('s3') as client:
            await client.head_bucket(Bucket='pants-cache')
        return web.json_response({'status': 'ok', 'storage': 'reachable'})
    except Exception as e:
        return web.json_response(
            {'status': 'error', 'message': str(e)},
            status=503
        )

app = web.Application()
app.router.add_get('/health', health_handler)

if __name__ == '__main__':
    web.run_app(app, port=8000)

Configure nginx to expose this on a separate port from the gRPC cache service:

server {
    listen 8080;
    location /health {
        proxy_pass http://localhost:8000;
        auth_basic off;
    }
}

Point Vigilmon at https://buildbarn.yourdomain.com:8080/health. A 503 response fires an alert before your next CI run hits a cold build.

Step 3: Monitor the Pants Daemon via TCP

Pants runs a persistent daemon (pantsd) that caches the build graph and avoids repeated initialization cost. The daemon listens on a local socket by default, but on CI machines it may also bind to a TCP port. You can expose a simple TCP health check for Vigilmon.

Add a lightweight TCP health server to your Pants plugin or build tooling:

# pants-plugins/health/health_server.py
import socket
import threading

def run_health_server(port=9876):
    """Accept connections on :9876 to signal pantsd is alive."""
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        s.bind(('', port))
        s.listen(1)
        while True:
            conn, _ = s.accept()
            conn.close()  # Accept and immediately close — just confirming daemon is alive

def start():
    t = threading.Thread(target=run_health_server, daemon=True)
    t.start()

In Vigilmon:

  1. Click Add Monitor → TCP Port.
  2. Enter your CI build host and port 9876.
  3. Set Check interval to 5 minutes.
  4. Click Save.

For ephemeral CI agents, this check is most useful on persistent self-hosted runners where pantsd runs continuously. A TCP failure indicates the daemon has crashed and needs a restart.

Step 4: Heartbeat Monitoring for Scheduled Full-Workspace Builds

Many Pants teams run a nightly full build (pants test :: or pants lint ::) to catch integration failures and pre-warm the remote cache before business hours. If that job silently fails due to a flaky test or resource exhaustion, the next day's CI will run slower without a warm cache.

Set up a Vigilmon cron heartbeat:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 1440 minutes (24 hours) for nightly builds.
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/pqr456.
  4. Add the curl call at the end of your nightly build script:
#!/bin/bash
set -e

echo "Starting Pants full-workspace build..."
pants --remote-cache-read=true --remote-cache-write=true \
  lint check test ::

echo "Build complete — signaling Vigilmon"
curl -fsS https://vigilmon.online/heartbeat/pqr456

GitHub Actions configuration:

name: Nightly Pants build
on:
  schedule:
    - cron: '0 3 * * *'  # 3 AM UTC

jobs:
  full-build:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - name: Run full Pants build
        run: |
          pants --remote-cache-read=true --remote-cache-write=true \
            lint check test ::
      - name: Signal success to Vigilmon
        if: success()
        run: curl -fsS https://vigilmon.online/heartbeat/pqr456

Using if: success() ensures the heartbeat fires only on a clean run. Failures prevent the ping and Vigilmon alerts after 24 hours.

Step 5: SSL Certificate Alerts for Cache Servers

Remote cache servers use mTLS or standard TLS to secure artifact transfers and client authentication. An expired certificate causes the Pants client to fail all cache lookups with a TLS handshake error, silently falling back to local execution.

In Vigilmon:

  1. Open the HTTP monitor from Step 1.
  2. Enable Monitor SSL certificate.
  3. Set Alert when certificate expires in less than 21 days.
  4. Click Save.

Verify your cache server certificate manually:

openssl s_client -connect buildbarn.yourdomain.com:443 2>/dev/null \
  | openssl x509 -noout -enddate
# notAfter=Sep 15 00:00:00 2026 GMT

For mTLS configurations, also check the client certificate your CI agents present:

# Check expiry of the Pants client cert
openssl x509 -noout -enddate -in /etc/pants/client.crt

Set Vigilmon to alert 21 days in advance — enough time to rotate certificates and redeploy them to all CI agents before they expire.

Step 6: Maintenance Windows During Cache Infrastructure Changes

When migrating your Pants remote cache to new infrastructure or upgrading the Buildbarn/BuildBuddy version, suppress monitoring alerts during the transition:

MONITOR_ID="your-monitor-id"
API_KEY="your-vigilmon-api-key"

# Open a 20-minute maintenance window
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"monitor_id\": \"$MONITOR_ID\", \"duration_minutes\": 20}"

# Perform the infrastructure change
kubectl apply -f k8s/buildbarn-upgrade.yaml
kubectl rollout status deployment/buildbarn-cache

# Verify Pants can reach the new backend
pants --remote-cache-read=true list ::

# Maintenance window expires automatically after 20 minutes

For Pants version upgrades that require updating pants.toml and regenerating lockfiles across the workspace, set a longer 60-minute window to cover the full migration time.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health check | Cache server /health or /readyz | Cache backend down, storage failure | | TCP port | gRPC cache port (:8980) | Backend unreachable, TLS terminator down | | TCP port | Daemon health port (:9876) | pantsd crash on persistent runners | | Cron heartbeat | Nightly build heartbeat URL | Missed full build, CI cron misconfiguration | | SSL certificate | Cache server domain | TLS cert expiry breaking all cache lookups |

Pants delivers reproducible, incremental builds at scale — but that scale amplifies the cost of cache failures and daemon crashes. With Vigilmon monitoring your remote cache backend, daemon health port, and nightly build heartbeats, you'll catch failures in minutes rather than discovering them when your morning CI queue has already stalled.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles