tutorial

Monitoring Digger with Vigilmon

Digger is an open-source Terraform GitOps CI/CD runner that executes plans and applies from GitHub Actions, GitLab CI, or its orchestrator backend — but when the Digger orchestrator goes offline, webhook delivery fails, or a lock prevents all applies on a stack, your infrastructure automation silently stalls. Here's how to monitor Digger availability, webhook processing, apply pipeline liveness, and stack lock staleness with Vigilmon.

Digger is an open-source Terraform CI/CD automation tool that runs terraform plan and terraform apply from your existing CI pipelines — GitHub Actions, GitLab CI, Azure DevOps — without requiring a dedicated server. Teams use Digger to get GitOps-driven infrastructure automation without the operational overhead of running an always-on Terraform backend. When running Digger in its backend orchestrator mode, however, the backend server must receive webhook events from GitHub/GitLab to orchestrate CI runs; when it goes offline, pull request plan comments stop appearing and apply commands are never processed. Even in agentless mode, the GitHub Actions runner executing Digger jobs can silently fail, state locking can freeze entire stacks, and failed apply jobs create inconsistent infrastructure state. Vigilmon gives you external monitoring for Digger backend availability, webhook processing, apply pipeline liveness, and CI job health so Terraform automation failures are caught before they block infrastructure deployments.

What You'll Set Up

  • Digger backend/orchestrator HTTP availability
  • Webhook endpoint reachability from external networks
  • Apply pipeline liveness via heartbeat
  • Stack lock staleness detection
  • GitHub Actions CI job failure monitoring

Prerequisites

  • Digger 0.5+ (backend mode) or Digger GitHub Actions runner (agentless mode)
  • GitHub or GitLab repository with Digger configured
  • Access to the Digger host for cron scripts
  • A free Vigilmon account

Step 1: Monitor the Digger Backend Health Endpoint

In backend/orchestrator mode, Digger runs an HTTP server that receives GitHub webhooks and dispatches CI jobs. When this server crashes or becomes unreachable, every Terraform PR stops getting plan output — engineers opening infrastructure PRs see nothing, which is indistinguishable from Digger working correctly but Terraform having no changes. Add an HTTP monitor as the first signal:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://digger.your-org.internal/health.
  3. Set Check interval to 1 minute.
  4. Under Expected response code, set to 200.
  5. Set Timeout to 10 seconds.

The Digger health endpoint returns HTTP 200 when the server is running and accepting webhooks. Verify it responds:

curl -sf --max-time 10 https://digger.your-org.internal/health

If you're using Digger Cloud (hosted backend), monitor the webhook delivery endpoint for your organization. If running Digger in GitHub Actions agentless mode, skip this step and proceed to Step 3.

Step 2: Monitor Webhook Endpoint Reachability

Digger's webhook endpoint (/github-app-webhook or /webhook) must be reachable from GitHub's webhook delivery servers. A firewall change, expired TLS certificate, or misconfigured reverse proxy makes the endpoint unreachable — GitHub queues webhook deliveries for 72 hours before dropping them, so failures can go unnoticed for days until someone checks a PR and sees no plan output:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://digger.your-org.internal/github-app-webhook.
  3. Set Check interval to 2 minutes.
  4. Under Expected response code, set to 405 — Digger's webhook endpoint expects POST requests with a valid GitHub signature; a GET returns 405 (Method Not Allowed). A 405 proves the endpoint is reachable and Digger is running; a 502/504 means the proxy or Digger is broken.
  5. Set Timeout to 15 seconds.

Validate webhook delivery health by checking GitHub's webhook delivery log for your app:

#!/bin/bash
# /usr/local/bin/digger-webhook-check.sh
# Checks recent GitHub App webhook delivery failures

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
GITHUB_TOKEN="your-github-token"
GITHUB_APP_ID="your-app-id"
MAX_CONSECUTIVE_FAILURES=3

# Check webhook deliveries for the GitHub App
DELIVERIES=$(curl -sf \
  --max-time 15 \
  -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/app/hook/deliveries?per_page=20" 2>/dev/null)

if [ -z "$DELIVERIES" ]; then
  echo "Could not fetch webhook deliveries"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

RESULT=$(echo "$DELIVERIES" | python3 -c "
import sys, json

deliveries = json.load(sys.stdin)

# Count consecutive failures from most recent
consecutive_failures = 0
for d in deliveries:
    status = d.get('status', '')
    if status == 'failed' or d.get('status_code', 200) >= 400:
        consecutive_failures += 1
    else:
        break

print(consecutive_failures)
" 2>/dev/null || echo 0)

if [ "${RESULT:-0}" -lt "$MAX_CONSECUTIVE_FAILURES" ]; then
  echo "Digger webhook deliveries OK: ${RESULT} consecutive failures (threshold: ${MAX_CONSECUTIVE_FAILURES})"
  curl -s "$HEARTBEAT_URL"
else
  echo "Digger webhook failure: ${RESULT} consecutive failed deliveries"
  exit 1
fi

Step 3: Monitor Apply Pipeline Liveness

Digger's value is running terraform apply automatically on PR merge. If the apply pipeline breaks — a bad GitHub Actions secret, a corrupt Terraform state file, a provider credential expiry — PRs merge without applying their infrastructure changes. Monitor pipeline throughput with a heartbeat from your CI workflow:

# .github/workflows/digger.yml — add to your existing Digger workflow
jobs:
  digger-apply:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Digger Terraform
        uses: diggerhq/digger@v0.5.x
        with:
          setup-aws: true
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Ping Vigilmon on successful apply
        if: success() && github.event_name == 'push' && github.ref == 'refs/heads/main'
        run: |
          curl -s "https://vigilmon.online/heartbeat/${{ secrets.VIGILMON_DIGGER_HEARTBEAT_TOKEN }}"

Create a Vigilmon heartbeat monitor with expected interval of 24 hours — if no apply completes within a day, you want to know. For teams with multiple Terraform stacks and frequent infrastructure changes, reduce the expected interval to 4 hours.

For teams using Digger in backend mode, you can also instrument the Digger backend directly to send a heartbeat after each successful apply:

#!/bin/bash
# Called from Digger's post-apply hook
HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
APPLY_STATUS="$1"   # Pass exit code from digger apply

if [ "${APPLY_STATUS:-0}" -eq 0 ]; then
  echo "Digger apply succeeded — pinging Vigilmon"
  curl -s "$HEARTBEAT_URL"
fi

Step 4: Detect Stale Stack Locks

Digger uses per-project locks to prevent concurrent Terraform operations on the same stack. If a plan or apply is interrupted mid-run (runner killed, network timeout, GitHub Actions cancellation), the lock is not released. Subsequent PRs for that stack receive "Lock acquired by PR #N" and cannot plan or apply until someone manually unlocks the stack. This can silently block all infrastructure changes for a stack for days:

#!/bin/bash
# /usr/local/bin/digger-lock-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
DIGGER_API_URL="https://digger.your-org.internal"
DIGGER_API_TOKEN="your-digger-api-token"
MAX_LOCK_AGE_HOURS=8   # Alert if any stack lock is older than 8 hours

# Fetch all active project locks from Digger API
LOCKS=$(curl -sf \
  --max-time 15 \
  -H "Authorization: Bearer ${DIGGER_API_TOKEN}" \
  "${DIGGER_API_URL}/api/projects/locks" 2>/dev/null)

if [ -z "$LOCKS" ]; then
  echo "Could not fetch Digger lock data — API unavailable or no locks endpoint"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

STALE_COUNT=$(echo "$LOCKS" | python3 -c "
import sys, json
from datetime import datetime, timezone

locks = json.load(sys.stdin)
if not isinstance(locks, list):
    locks = locks.get('locks', [])

now = datetime.now(timezone.utc)
stale = 0
max_lock_age = ${MAX_LOCK_AGE_HOURS}

for lock in locks:
    locked_at = lock.get('created_at') or lock.get('lockedAt') or lock.get('timestamp')
    if locked_at:
        try:
            lock_time = datetime.fromisoformat(locked_at.replace('Z', '+00:00'))
            age_hours = (now - lock_time).total_seconds() / 3600
            if age_hours > max_lock_age:
                project = lock.get('project_name') or lock.get('projectId', 'unknown')
                pr = lock.get('pr_number') or lock.get('prNumber', 'unknown')
                print(f'Stale lock: project={project} PR=#{pr} age={age_hours:.1f}h', file=sys.stderr)
                stale += 1
        except Exception as e:
            pass

print(stale)
" 2>&1 | tail -1)

if [ "${STALE_COUNT:-0}" -eq 0 ]; then
  echo "No stale Digger stack locks detected"
  curl -s "$HEARTBEAT_URL"
else
  echo "Found ${STALE_COUNT} stale Digger lock(s) older than ${MAX_LOCK_AGE_HOURS} hours"
  exit 1
fi

Set the heartbeat expected interval to 10 hours. If your team doesn't open infrastructure PRs on weekends, adjust MAX_LOCK_AGE_HOURS to account for weekend gaps.

Step 5: Monitor GitHub Actions Job Failure Rate

In agentless mode, Digger runs entirely within GitHub Actions — there's no backend server to monitor. Instead, monitor the health of the CI workflows that run Digger:

#!/bin/bash
# /usr/local/bin/digger-ci-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
GITHUB_TOKEN="your-github-token"
GITHUB_ORG="your-org"
GITHUB_REPO="your-infra-repo"
WORKFLOW_NAME="digger.yml"
MAX_FAILURE_RATE_PCT=50   # Alert if >50% of recent Digger runs fail
LOOKBACK_RUNS=10

# Fetch recent workflow runs for the Digger workflow
RUNS=$(curl -sf \
  --max-time 15 \
  -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/${GITHUB_ORG}/${GITHUB_REPO}/actions/workflows/${WORKFLOW_NAME}/runs?per_page=${LOOKBACK_RUNS}" \
  2>/dev/null)

if [ -z "$RUNS" ]; then
  echo "Could not fetch GitHub Actions workflow runs"
  exit 1
fi

RESULT=$(echo "$RUNS" | python3 -c "
import sys, json

data = json.load(sys.stdin)
runs = data.get('workflow_runs', [])

if not runs:
    print('0/0')
    sys.exit()

completed = [r for r in runs if r.get('status') == 'completed']
if not completed:
    print('0/0')
    sys.exit()

failed = [r for r in completed if r.get('conclusion') in ['failure', 'timed_out', 'cancelled']]
print(f'{len(failed)}/{len(completed)}')
" 2>/dev/null)

TOTAL=$(echo "$RESULT" | cut -d/ -f2)
FAILED=$(echo "$RESULT" | cut -d/ -f1)

if [ -z "$TOTAL" ] || [ "$TOTAL" -eq 0 ]; then
  echo "No completed Digger CI runs found"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

FAILURE_PCT=$(( FAILED * 100 / TOTAL ))

if [ "$FAILURE_PCT" -le "$MAX_FAILURE_RATE_PCT" ]; then
  echo "Digger CI health OK: ${FAILED}/${TOTAL} recent runs failed (${FAILURE_PCT}%)"
  curl -s "$HEARTBEAT_URL"
else
  echo "Digger CI failure rate elevated: ${FAILED}/${TOTAL} runs failed (${FAILURE_PCT}%)"
  exit 1
fi

Schedule every 30 minutes with a Vigilmon heartbeat expected interval of 35 minutes.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Digger alerts to the infrastructure team:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #infra-deploys so engineers opening Terraform PRs know when Digger is unavailable.
  3. Add PagerDuty for the backend health and webhook reachability monitors — when Digger is down, all infrastructure deployments are blocked.
  4. Add email for the stale lock monitor — stale locks need manual intervention but aren't emergency-level.
  5. Set Consecutive failures before alert to 2 for the health monitor — brief restarts during deployments cause single-probe gaps.

Add maintenance windows during Digger upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "DIGGER_HEALTH_MONITOR_ID",
    "duration_minutes": 10,
    "reason": "Digger backend upgrade to 0.6.x"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (health) | /health endpoint | Digger backend crash, OOM, container eviction | | HTTP monitor (webhook) | /github-app-webhook (expect 405) | Proxy misconfiguration, TLS expiry, network ACL | | Cron heartbeat (webhook deliveries) | GitHub App delivery failure count | GitHub-side webhook failures, delivery drops | | Cron heartbeat (apply pipeline) | CI workflow success heartbeat | Broken apply pipeline, expired credentials | | Cron heartbeat (stack locks) | Lock age check | Stale locks blocking all PRs on a stack | | Cron heartbeat (CI failure rate) | GitHub Actions failure rate | Elevated CI job failures, agentless mode breakage |

Digger automates the highest-stakes part of the software delivery lifecycle — infrastructure changes. When its backend server crashes, a webhook fails to deliver, or a stale lock blocks all applies for a stack, engineers continue opening PRs with no feedback and infrastructure changes never land. Vigilmon gives you external monitoring for every layer of Digger so automation failures surface within minutes rather than being discovered when a critical deployment never happens.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles