tutorial

Monitoring Terragrunt with Vigilmon

Terragrunt keeps your Terraform code DRY across dozens of modules and environments — but when remote state locks stall, plan commands fail silently, or dependency chains break mid-apply, your infrastructure drift goes undetected until a deployment incident surfaces it. Here's how to monitor Terragrunt plan health, state backend availability, and apply pipeline integrity with Vigilmon.

Terragrunt is a thin wrapper around Terraform that adds remote state management, dependency ordering, and DRY configuration across large multi-environment infrastructure stacks. Teams running Terragrunt at scale — dozens of modules across dev, staging, and production — rely on its run-all plan and run-all apply commands to orchestrate changes without missing dependencies or leaving stale locks. When the remote state backend becomes unavailable, a state lock goes stale after a failed apply, or a cascading dependency failure silently skips downstream modules, your infrastructure diverges from code without any immediate alert. Vigilmon gives you external monitoring for Terragrunt's state backend availability, plan execution health, lock staleness, and CI apply pipeline liveness so you catch infrastructure drift at the source.

What You'll Set Up

  • Remote state backend availability (S3, GCS, Azure Blob)
  • Stale state lock detection
  • Terragrunt plan execution health
  • run-all apply pipeline heartbeat monitoring
  • Dependency graph validation

Prerequisites

  • Terragrunt 0.45+ with a remote state backend configured
  • AWS CLI / gsutil / az CLI access to the state backend bucket
  • A CI system running terragrunt run-all plan and apply (GitHub Actions, GitLab CI, etc.)
  • A free Vigilmon account

Step 1: Monitor Remote State Backend Availability

Terragrunt's remote state backend (typically S3, GCS, or Azure Blob) is the single dependency every module shares. If the bucket is inaccessible — wrong IAM permissions after a policy change, bucket accidentally deleted, or a regional outage — every plan and apply fails with a cryptic "failed to get existing workspaces" error. Monitor backend reachability directly:

For S3 (AWS):

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 5 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
#!/bin/bash
# /usr/local/bin/tg-state-backend-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
STATE_BUCKET="your-terraform-state-bucket"
AWS_REGION="us-east-1"
TIMEOUT=15

# Attempt to list the state bucket prefix — validates IAM + bucket existence
if aws s3 ls "s3://${STATE_BUCKET}/" \
  --region "$AWS_REGION" \
  --request-payer requester \
  --max-items 1 \
  --cli-connect-timeout "$TIMEOUT" > /dev/null 2>&1; then
  echo "Terragrunt state backend reachable: s3://${STATE_BUCKET}"
  curl -s "$HEARTBEAT_URL"
else
  echo "Terragrunt state backend unreachable: s3://${STATE_BUCKET}"
  exit 1
fi

For GCS (Google Cloud):

#!/bin/bash
# /usr/local/bin/tg-gcs-backend-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc456"
STATE_BUCKET="your-terraform-state-bucket"

if gsutil ls "gs://${STATE_BUCKET}/" > /dev/null 2>&1; then
  echo "Terragrunt GCS state backend reachable"
  curl -s "$HEARTBEAT_URL"
else
  echo "Terragrunt GCS state backend unreachable"
  exit 1
fi

Schedule both checks with cron:

chmod +x /usr/local/bin/tg-state-backend-check.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/tg-state-backend-check.sh >> /var/log/terragrunt-health.log 2>&1") | crontab -

Step 2: Detect Stale State Locks

Terraform state locks prevent concurrent applies from corrupting state. When an apply is killed mid-run (SIGKILL, CI timeout, OOM), the lock is never released and subsequent operations fail with "Error acquiring the state lock." In large Terragrunt stacks with dozens of modules, a single stale lock can block all applies for that environment.

#!/bin/bash
# /usr/local/bin/tg-lock-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def789"
STATE_BUCKET="your-terraform-state-bucket"
AWS_REGION="us-east-1"
MAX_LOCK_AGE_MINUTES=60   # Alert if any lock is older than 60 minutes

STALE_LOCKS=0
NOW=$(date +%s)

# Scan all .tflock files in the bucket (DynamoDB lock table is the authoritative source, but S3 also shows lock info objects)
LOCK_FILES=$(aws s3 ls "s3://${STATE_BUCKET}/" \
  --region "$AWS_REGION" \
  --recursive \
  --query "Contents[?ends_with(Key, '.tflock')]" \
  --output json 2>/dev/null | python3 -c "
import sys, json
items = json.load(sys.stdin)
for i in items:
    print(i['Key'], i['LastModified'])
" 2>/dev/null)

while IFS= read -r line; do
  [ -z "$line" ] && continue
  LOCK_KEY=$(echo "$line" | awk '{print $1}')
  LOCK_TIME=$(echo "$line" | awk '{print $2, $3}' | cut -d'+' -f1)
  LOCK_TS=$(date -d "$LOCK_TIME" +%s 2>/dev/null || date -j -f "%Y-%m-%d %H:%M:%S" "$LOCK_TIME" +%s 2>/dev/null)
  
  if [ -n "$LOCK_TS" ]; then
    AGE_MINUTES=$(( (NOW - LOCK_TS) / 60 ))
    if [ "$AGE_MINUTES" -gt "$MAX_LOCK_AGE_MINUTES" ]; then
      echo "STALE LOCK: ${LOCK_KEY} (age: ${AGE_MINUTES} min)"
      STALE_LOCKS=$((STALE_LOCKS + 1))
    fi
  fi
done <<< "$LOCK_FILES"

if [ "$STALE_LOCKS" -eq 0 ]; then
  echo "No stale Terragrunt state locks detected"
  curl -s "$HEARTBEAT_URL"
else
  echo "Found ${STALE_LOCKS} stale Terragrunt state lock(s)"
  exit 1
fi

Set the heartbeat interval to 65 minutes (5 minutes past the MAX_LOCK_AGE_MINUTES threshold). A stale lock left overnight reliably blocks the morning deployment run when the team arrives.

Step 3: Monitor Plan Execution Health

A terragrunt run-all plan that exits zero but produces no diff output, or one that errors on a subset of modules while succeeding overall, gives a false "green" signal. Instrument your CI plan jobs to ping Vigilmon only when the plan completes cleanly for all modules:

#!/bin/bash
# ci-terragrunt-plan.sh (runs in your CI pipeline)

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi012"
TG_ROOT="./infrastructure"
PLAN_LOG="/tmp/tg-plan-$(date +%s).log"

# Run plan for all modules, capturing output
terragrunt run-all plan \
  --terragrunt-working-dir "$TG_ROOT" \
  --terragrunt-non-interactive \
  --terragrunt-ignore-external-dependencies \
  2>&1 | tee "$PLAN_LOG"

PLAN_EXIT=${PIPESTATUS[0]}

# Check for error indicators even if exit code is 0
ERROR_COUNT=$(grep -c "Error:" "$PLAN_LOG" || true)
MODULE_ERRORS=$(grep -c "terragrunt run-all had errors" "$PLAN_LOG" || true)

if [ "$PLAN_EXIT" -eq 0 ] && [ "$ERROR_COUNT" -eq 0 ] && [ "$MODULE_ERRORS" -eq 0 ]; then
  echo "Terragrunt run-all plan succeeded for all modules"
  curl -s "$HEARTBEAT_URL"
else
  echo "Terragrunt plan failed: exit=${PLAN_EXIT}, errors=${ERROR_COUNT}, module_errors=${MODULE_ERRORS}"
  # Don't ping heartbeat — Vigilmon will alert on missed ping
  exit 1
fi

rm -f "$PLAN_LOG"

In Vigilmon, set the Cron Heartbeat expected interval to match your CI plan schedule plus a buffer — if plans run every 4 hours, set the expected interval to 5 hours.

Step 4: Monitor Apply Pipeline Liveness

The apply pipeline is where changes actually reach infrastructure. A stuck CI pipeline — frozen at "Waiting for approval" because a Slack notification failed, a blocked queue, or a timed-out runner — means merges pile up without deploying. Monitor apply pipeline liveness with a heartbeat that only fires on successful applies:

#!/bin/bash
# ci-terragrunt-apply.sh (runs after approval in your CI pipeline)

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl345"
TG_ROOT="./infrastructure"
ENVIRONMENT="${TF_ENV:-production}"
APPLY_LOG="/tmp/tg-apply-$(date +%s).log"

echo "Starting Terragrunt apply for environment: ${ENVIRONMENT}"

terragrunt run-all apply \
  --terragrunt-working-dir "$TG_ROOT" \
  --terragrunt-non-interactive \
  --terragrunt-ignore-external-dependencies \
  -auto-approve \
  2>&1 | tee "$APPLY_LOG"

APPLY_EXIT=${PIPESTATUS[0]}

# Count applied modules
APPLIED=$(grep -c "Apply complete!" "$APPLY_LOG" || true)
FAILED=$(grep -c "Apply failed!" "$APPLY_LOG" || true)

echo "Apply result: exit=${APPLY_EXIT}, modules_applied=${APPLIED}, modules_failed=${FAILED}"

if [ "$APPLY_EXIT" -eq 0 ] && [ "$FAILED" -eq 0 ]; then
  echo "Terragrunt apply succeeded: ${APPLIED} module(s) applied"
  curl -s "$HEARTBEAT_URL"
else
  echo "Terragrunt apply failed"
  exit 1
fi

rm -f "$APPLY_LOG"

Set the heartbeat expected interval based on your deploy cadence — for daily deploys, use 25 hours so a missed day triggers an alert but normal weekend variation doesn't.

Step 5: Validate Dependency Graph Integrity

Terragrunt's dependency blocks create an explicit execution order. When a module reference is broken — a module moved, renamed, or its outputs changed — run-all plan may fail on all downstream modules. Validate the dependency graph on every PR to catch structural breaks before they reach main:

#!/bin/bash
# ci-terragrunt-validate.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno678"
TG_ROOT="./infrastructure"
VALIDATE_LOG="/tmp/tg-validate-$(date +%s).log"

# Validate all module configurations and dependency graph
terragrunt run-all validate \
  --terragrunt-working-dir "$TG_ROOT" \
  --terragrunt-non-interactive \
  2>&1 | tee "$VALIDATE_LOG"

VALIDATE_EXIT=${PIPESTATUS[0]}

# Check for dependency resolution errors
DEP_ERRORS=$(grep -c "dependency.*does not exist\|could not read outputs\|circular dependency" "$VALIDATE_LOG" || true)

if [ "$VALIDATE_EXIT" -eq 0 ] && [ "$DEP_ERRORS" -eq 0 ]; then
  echo "Terragrunt dependency graph valid"
  curl -s "$HEARTBEAT_URL"
else
  echo "Terragrunt validation failed: exit=${VALIDATE_EXIT}, dependency_errors=${DEP_ERRORS}"
  cat "$VALIDATE_LOG"
  exit 1
fi

rm -f "$VALIDATE_LOG"

Run this on every PR merge to main. A broken dependency graph is easy to introduce (renaming a module output) and hard to discover without running the full plan.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Terragrunt alerts to your infrastructure team:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #infra-ops for all Terragrunt monitors.
  3. Add PagerDuty for the state backend monitor and stale lock monitor — these block all deployments.
  4. Add email for the apply pipeline monitor — a missed daily deploy is important but not a 3am incident.
  5. Set Consecutive failures before alert to 2 for plan health (transient CI runner issues can cause single failures).

Add maintenance windows during planned Terragrunt migrations:

# Silence monitors during state backend migration
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "TG_BACKEND_MONITOR_ID",
    "duration_minutes": 30,
    "reason": "Migrating state backend from S3 standard to S3 Intelligent-Tiering"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (backend) | S3/GCS bucket list | IAM policy breaks, bucket deletion, regional outage | | Cron heartbeat (locks) | .tflock file age | Stale locks from killed applies, deployment blockers | | Cron heartbeat (plan) | CI run-all plan exit + log | Silent module errors, dependency failures | | Cron heartbeat (apply) | CI run-all apply success | Stuck pipelines, missed deploys, apply failures | | Cron heartbeat (validate) | run-all validate graph check | Broken module references, circular dependencies |

Terragrunt's multi-module orchestration abstracts away a lot of Terraform complexity, but it also means a single broken dependency or stale lock can silently block your entire infrastructure deployment chain. Vigilmon gives you external visibility into every layer — from backend storage to apply completion — so infrastructure drift surfaces within minutes, not after a deployment incident.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles