tutorial

Monitoring Atlantis with Vigilmon

Atlantis automates Terraform plan and apply directly from pull requests — but when the Atlantis server goes offline, webhook delivery fails, or a plan lock hangs indefinitely, your entire infrastructure review process silently stops working. Here's how to monitor Atlantis server health, webhook processing, plan lock staleness, and apply pipeline liveness with Vigilmon.

Atlantis is a self-hosted Terraform pull request automation tool that runs terraform plan on every PR and gates terraform apply behind a PR comment approval. Teams using Atlantis eliminate the "run plan locally and copy the output to the PR" workflow — every diff is planned automatically, and every apply is traceable to a PR comment. When the Atlantis server goes offline, GitHub/GitLab webhooks silently accumulate in the provider's queue with no delivery feedback; when a plan lock hangs because a previous apply was interrupted, all subsequent PRs for the same module queue behind it indefinitely; and when the Atlantis ngrok tunnel or reverse proxy drops, the webhook endpoint becomes unreachable with no immediate signal. Vigilmon gives you external monitoring for Atlantis server availability, webhook endpoint reachability, plan lock health, and apply pipeline liveness so infrastructure PR automation failures surface within minutes.

What You'll Set Up

  • Atlantis HTTP server availability and health endpoint
  • Webhook endpoint reachability from external networks
  • Plan lock staleness detection
  • Apply pipeline liveness monitoring
  • Atlantis worker queue depth

Prerequisites

  • Atlantis 0.22+ with the REST API and health endpoint accessible
  • GitHub, GitLab, or Bitbucket webhook configured to deliver to Atlantis
  • Access to the Atlantis server host or container
  • A free Vigilmon account

Step 1: Monitor the Atlantis Health Endpoint

Atlantis exposes a /healthz endpoint that returns HTTP 200 when the server is running and ready to receive webhooks. When the process crashes, runs out of memory, or is evicted from Kubernetes, webhook deliveries from GitHub/GitLab fail with connection errors — but the VCS provider queues them silently and retries for a limited window. After that window closes, events are dropped and PRs never receive plan output. Add Vigilmon's HTTP monitor as the first line of defense:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://atlantis.your-org.internal/healthz.
  3. Set Check interval to 1 minute.
  4. Under Expected response, add keyword "status":"ok" — the healthz endpoint returns this JSON key when healthy.
  5. Set Timeout to 10 seconds.

The Atlantis health endpoint returns:

{"status":"ok"}

If you also have Atlantis behind a reverse proxy (nginx, Traefik, Caddy), add a second HTTP monitor for the proxy layer to distinguish between Atlantis process failures and proxy configuration regressions:

# Verify both the proxy layer and the Atlantis process respond
curl -sf --max-time 10 https://atlantis.your-org.internal/healthz
# Also verify the proxy passes through correctly
curl -sf --max-time 10 https://atlantis.your-org.internal/ | grep -q "Atlantis"

Step 2: Monitor Webhook Endpoint Reachability

Atlantis is useless if GitHub or GitLab can't reach its webhook endpoint. Network topology changes, misconfigured security groups, expired TLS certificates, or a dropped reverse proxy rule can make the endpoint unreachable from the public internet even when Atlantis is running locally on the host. Test reachability from an external perspective:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://atlantis.your-org.internal/events.
  3. Set Check interval to 2 minutes.
  4. Under Expected response code, set to 400 — Vigilmon's monitor sends a GET request; Atlantis expects a POST with a valid webhook signature and returns 400 for malformed requests. A 400 means the endpoint is reachable and Atlantis is running; a 502/504/connection error means the proxy or endpoint is broken.
  5. Set Timeout to 15 seconds.

This is the most important Atlantis monitor because it validates the exact path GitHub/GitLab uses to deliver webhooks. A 400 from Atlantis on a GET request is healthy — it proves the endpoint is reachable and Atlantis is processing incoming requests.

For stricter webhook validation, monitor webhook delivery stats directly from your VCS provider:

#!/bin/bash
# /usr/local/bin/atlantis-webhook-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
GITHUB_TOKEN="your-github-token"
GITHUB_ORG="your-org"
GITHUB_REPO="your-infra-repo"
MAX_FAILED_DELIVERIES=5

# Check recent webhook delivery failures via GitHub API
FAILED=$(curl -sf \
  --max-time 15 \
  -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/${GITHUB_ORG}/${GITHUB_REPO}/hooks" | \
python3 -c "
import sys, json
hooks = json.load(sys.stdin)
total_failed = 0
for h in hooks:
    if 'atlantis' in h.get('config', {}).get('url', '').lower():
        # last_response is available per hook
        lr = h.get('last_response', {})
        if lr.get('code') not in [200, 201, 204, None]:
            total_failed += 1
print(total_failed)
" 2>/dev/null || echo 0)

if [ "${FAILED:-0}" -lt "$MAX_FAILED_DELIVERIES" ]; then
  echo "Atlantis webhook deliveries OK: ${FAILED} failures"
  curl -s "$HEARTBEAT_URL"
else
  echo "Atlantis webhook delivery failures: ${FAILED}"
  exit 1
fi

Step 3: Detect Stale Plan Locks

When Atlantis runs terraform plan or terraform apply for a module, it acquires a lock on that module's working directory. If the apply process is killed mid-run (OOM, SIGKILL, pod eviction), the lock file is not cleaned up. Every subsequent PR targeting the same module receives "This project is currently locked by PR #N" until someone manually runs atlantis unlock on the stale PR.

#!/bin/bash
# /usr/local/bin/atlantis-lock-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
ATLANTIS_URL="https://atlantis.your-org.internal"
ATLANTIS_TOKEN="your-atlantis-api-token"
MAX_LOCK_AGE_HOURS=4   # Alert if any lock is older than 4 hours

# Fetch active locks from Atlantis API
LOCKS=$(curl -sf \
  --max-time 15 \
  -H "X-Atlantis-Token: ${ATLANTIS_TOKEN}" \
  "${ATLANTIS_URL}/api/locks" 2>/dev/null)

if [ -z "$LOCKS" ]; then
  echo "Could not fetch Atlantis lock data"
  curl -s "$HEARTBEAT_URL"  # Don't alert here — use the health monitor for API failures
  exit 0
fi

STALE_COUNT=$(echo "$LOCKS" | python3 -c "
import sys, json
from datetime import datetime, timezone

locks = json.load(sys.stdin)
now = datetime.now(timezone.utc)
stale = 0

for lock in locks:
    locked_at = lock.get('time')
    if locked_at:
        try:
            lock_time = datetime.fromisoformat(locked_at.replace('Z', '+00:00'))
            age_hours = (now - lock_time).total_seconds() / 3600
            if age_hours > ${MAX_LOCK_AGE_HOURS}:
                print(f'Stale lock: PR #{lock.get(\"pullNum\", \"unknown\")} on {lock.get(\"repoName\", \"unknown\")} ({age_hours:.1f}h old)', file=sys.stderr)
                stale += 1
        except:
            pass

print(stale)
" 2>&1 | tail -1)

if [ "${STALE_COUNT:-0}" -eq 0 ]; then
  echo "No stale Atlantis locks detected"
  curl -s "$HEARTBEAT_URL"
else
  echo "Found ${STALE_COUNT} stale Atlantis lock(s) older than ${MAX_LOCK_AGE_HOURS} hours"
  exit 1
fi

Set the cron heartbeat expected interval to 5 hours. Locks older than 4 hours are almost always from interrupted applies — automatically alerting avoids the situation where a Monday-morning PR can't plan because a Friday-afternoon lock was never cleared.

Step 4: Monitor Apply Pipeline Liveness

Atlantis apply is triggered by a comment (atlantis apply) on an approved PR. If Atlantis is running but its worker queue is backed up — too many concurrent applies, a slow Terraform provider, or a hung apply process — new apply comments silently queue with no visible feedback. Monitor apply pipeline throughput with a regular test apply on a non-destructive module:

#!/bin/bash
# /usr/local/bin/atlantis-apply-liveness.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
ATLANTIS_LOG="/var/log/atlantis/atlantis.log"   # Adjust to your log path
MAX_APPLY_AGE_HOURS=26   # Alert if no successful apply in 26 hours (catches missed daily runs)

# Check last successful apply from log
LAST_APPLY=$(grep "Apply complete!" "$ATLANTIS_LOG" 2>/dev/null | tail -1)

if [ -z "$LAST_APPLY" ]; then
  echo "No apply entries found in Atlantis log"
  # Don't fail — new installs may not have run applies yet
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

# Extract timestamp from log line (format varies by log configuration)
LAST_APPLY_TIME=$(echo "$LAST_APPLY" | grep -oP '\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}' | head -1)

if [ -n "$LAST_APPLY_TIME" ]; then
  LAST_TS=$(date -d "$LAST_APPLY_TIME" +%s 2>/dev/null)
  NOW=$(date +%s)
  AGE_HOURS=$(( (NOW - LAST_TS) / 3600 ))

  if [ "$AGE_HOURS" -le "$MAX_APPLY_AGE_HOURS" ]; then
    echo "Atlantis last apply: ${AGE_HOURS}h ago"
    curl -s "$HEARTBEAT_URL"
  else
    echo "Atlantis apply pipeline stale: last apply ${AGE_HOURS}h ago"
    exit 1
  fi
else
  echo "Could not parse apply timestamp"
  curl -s "$HEARTBEAT_URL"
fi

For teams using GitHub Actions or another CI system to trigger Atlantis applies rather than PR comments, instrument the CI step directly:

# .github/workflows/atlantis-apply.yml
- name: Apply via Atlantis and ping Vigilmon
  run: |
    # ... your apply trigger logic ...
    if [ "$APPLY_EXIT" -eq 0 ]; then
      curl -s "https://vigilmon.online/heartbeat/YOUR_TOKEN"
    fi

Step 5: Monitor Atlantis Worker Queue Depth

Atlantis processes webhook events (push, PR open, PR comment) through an internal worker queue. Under heavy PR load — mass rebases, a team pushing a feature branch — this queue can back up, and plan output can take minutes to appear on PRs. Monitor queue depth if Atlantis exposes metrics:

#!/bin/bash
# /usr/local/bin/atlantis-metrics-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
ATLANTIS_URL="https://atlantis.your-org.internal"
MAX_QUEUE_DEPTH=20

# Atlantis exposes Prometheus metrics at /metrics if enabled
METRICS=$(curl -sf --max-time 15 "${ATLANTIS_URL}/metrics" 2>/dev/null)

if [ -z "$METRICS" ]; then
  echo "Atlantis metrics endpoint not available"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

QUEUE_DEPTH=$(echo "$METRICS" | grep "^atlantis_work_queue_depth " | awk '{print $2}' | tr -d '\r')

if [ -z "$QUEUE_DEPTH" ]; then
  echo "Queue depth metric not found — Atlantis may not expose it in this version"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

if [ "${QUEUE_DEPTH:-0}" -le "$MAX_QUEUE_DEPTH" ]; then
  echo "Atlantis worker queue depth OK: ${QUEUE_DEPTH}"
  curl -s "$HEARTBEAT_URL"
else
  echo "Atlantis worker queue backed up: depth ${QUEUE_DEPTH} (max: ${MAX_QUEUE_DEPTH})"
  exit 1
fi

If Atlantis doesn't expose metrics in your version, use the Prometheus text format scraping or check the process health via system metrics (CPU, memory) as a proxy for queue saturation.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Atlantis alerts appropriately:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #infra-prs so engineers submitting Terraform PRs know immediately when Atlantis is unavailable.
  3. Add PagerDuty for the health endpoint and webhook reachability monitors — a downed Atlantis blocks all infrastructure deployments.
  4. Add email for the stale lock monitor — stale locks are blocking but not emergency-level.
  5. Set Consecutive failures before alert to 2 for the webhook reachability monitor — brief TLS renegotiation timeouts can cause single-probe failures.

Add maintenance windows during Atlantis upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "ATLANTIS_HEALTH_MONITOR_ID",
    "duration_minutes": 10,
    "reason": "Atlantis server upgrade to 0.28.x"
  }'

# Perform upgrade
docker pull ghcr.io/runatlantis/atlantis:latest
docker-compose up -d atlantis

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (health) | /healthz keyword check | Atlantis process crash, OOM, pod eviction | | HTTP monitor (webhook) | /events reachability (expect 400) | Proxy misconfiguration, TLS expiry, network ACL changes | | Cron heartbeat (locks) | API lock age check | Stale locks from interrupted applies | | Cron heartbeat (apply) | Last successful apply age | Stuck apply queue, apply pipeline breakage | | Cron heartbeat (metrics) | Worker queue depth | Queue saturation, slow plan/apply throughput |

Atlantis is the bridge between Terraform code and infrastructure changes — when it goes offline or its webhook endpoint becomes unreachable, engineering teams lose visibility into infrastructure diffs on every open PR. Vigilmon gives you external visibility into every layer of Atlantis so a server crash, stale lock, or broken webhook is caught within minutes rather than discovered when a developer asks why their PR never got a plan.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles