tutorial

Monitoring Terramate-Managed Infrastructure with Vigilmon: External Uptime, API Health & Heartbeat Monitoring

Practical guide to complementing Terramate's Terraform orchestration with Vigilmon's external uptime checks — health endpoints, synthetic HTTP monitors, and independent alerting for infrastructure deployed through Terramate stacks.

Terramate is a Terraform orchestration and code generation tool that brings order to large-scale infrastructure-as-code repositories. It manages stack dependencies, change detection, and execution ordering — letting teams run terraform apply only on the stacks affected by a given change, without accidentally touching unrelated infrastructure. What Terramate doesn't provide is an independent external health check: a synthetic probe that measures whether the services deployed through your Terramate stacks are reachable and responding correctly from the outside. Vigilmon fills that gap. It sits entirely outside your infrastructure and probes your services from multiple external locations, alerting you immediately when a deployed service goes down, regardless of what Terramate's run logs report.

This tutorial covers adding external uptime monitoring to services deployed and managed through Terramate stacks.

What You'll Build

  • A /health endpoint in a service managed by a Terramate stack
  • A Vigilmon HTTP monitor with response-body assertions
  • A heartbeat monitor for Terramate scheduled plan and apply jobs
  • An alert routing strategy that keeps Terramate deployment alerts and Vigilmon uptime alerts separate but correlated

Prerequisites

  • A Terramate project with at least one deployed service stack
  • At least one service endpoint reachable via a public HTTPS URL
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Service

Terramate orchestrates your infrastructure deployments. Vigilmon needs an HTTP endpoint it can reach from the outside once the stack is deployed. The health check should verify the real service dependencies — database connectivity, upstream API reachability, and application initialization — not just a static 200 that passes even when the service is degraded.

Python (FastAPI)

# app/health.py
import os
import httpx
import asyncpg
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

@app.get("/health")
async def health():
    checks = {}
    ok = True

    # Database connectivity
    try:
        conn = await asyncpg.connect(os.environ["DATABASE_URL"], timeout=3)
        await conn.fetchval("SELECT 1")
        await conn.close()
        checks["database"] = "ok"
    except Exception as exc:
        checks["database"] = f"error: {exc}"
        ok = False

    # Upstream API dependency
    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            resp = await client.get(os.environ["UPSTREAM_API_URL"] + "/ping")
            checks["upstream_api"] = "ok" if resp.status_code == 200 else f"http_{resp.status_code}"
            if resp.status_code != 200:
                ok = False
    except Exception as exc:
        checks["upstream_api"] = f"error: {exc}"
        ok = False

    status_code = 200 if ok else 503
    return JSONResponse(
        status_code=status_code,
        content={"status": "ok" if ok else "degraded", "checks": checks},
    )

Go

// internal/health/handler.go
package health

import (
    "database/sql"
    "encoding/json"
    "net/http"
    "os"
    "time"
)

type HealthResponse struct {
    Status string            `json:"status"`
    Checks map[string]string `json:"checks"`
}

func Handler(db *sql.DB) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        checks := make(map[string]string)
        ok := true

        // Database ping
        if err := db.Ping(); err != nil {
            checks["database"] = "error: " + err.Error()
            ok = false
        } else {
            checks["database"] = "ok"
        }

        // Upstream service check
        client := &http.Client{Timeout: 5 * time.Second}
        resp, err := client.Get(os.Getenv("UPSTREAM_API_URL") + "/ping")
        if err != nil {
            checks["upstream_api"] = "error: " + err.Error()
            ok = false
        } else {
            resp.Body.Close()
            if resp.StatusCode == http.StatusOK {
                checks["upstream_api"] = "ok"
            } else {
                checks["upstream_api"] = fmt.Sprintf("http_%d", resp.StatusCode)
                ok = false
            }
        }

        status := "ok"
        statusCode := http.StatusOK
        if !ok {
            status = "degraded"
            statusCode = http.StatusServiceUnavailable
        }

        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(statusCode)
        json.NewEncoder(w).Encode(HealthResponse{Status: status, Checks: checks})
    }
}

Verify the endpoint before configuring Vigilmon:

curl -s https://your-service.example.com/health | jq .
# {"status":"ok","checks":{"database":"ok","upstream_api":"ok"}}

Step 2: Configure Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-service.example.com/health
  3. Check interval: 60 seconds — appropriate for production services.
  4. Response timeout: 10 seconds.
  5. Expected status code: 200.
  6. JSON body assertion:
    • Path: status
    • Expected value: ok
  7. Click Save.

Vigilmon immediately begins probing from multiple external locations. A non-200 response or a failed JSON assertion triggers an alert, independently of whether Terramate's own deployment pipeline reports success.

Monitoring each stack's service

Terramate organizes infrastructure into stacks, and each stack typically corresponds to one or more deployed services. Create a Vigilmon monitor for each production service endpoint:

| Stack | Service URL | Interval | |---|---|---| | stacks/api | https://api.example.com/health | 60 s | | stacks/worker | https://worker.example.com/health | 60 s | | stacks/frontend | https://app.example.com/health | 60 s | | stacks/api-staging | https://api-staging.example.com/health | 120 s |

Group all monitors under a Monitor group named after the service or environment.

Step 3: Heartbeat Monitor for Scheduled Terramate Jobs

Terramate deployments often run on a schedule — nightly drift detection runs, weekly full plan previews, or CI-gated apply pipelines. Use Vigilmon heartbeats to guarantee these scheduled orchestration jobs complete on time.

#!/bin/bash
# scripts/nightly-drift-check.sh

set -euo pipefail

echo "Running Terramate change detection..."
terramate run --changed -- terraform plan -detailed-exitcode

EXIT_CODE=$?
if [ $EXIT_CODE -eq 0 ] || [ $EXIT_CODE -eq 2 ]; then
  # Exit code 0: no changes. Exit code 2: changes detected (not an error).
  curl -s -X POST "$VIGILMON_HEARTBEAT_URL" --max-time 5
  echo "Drift check complete — heartbeat sent"
else
  echo "Drift check failed with exit code $EXIT_CODE"
  exit $EXIT_CODE
fi

In Vigilmon, create a Heartbeat monitor with a grace period slightly above your cron interval. For a nightly job, use 25 hours. Store the heartbeat URL as a CI secret or environment variable.

GitHub Actions scheduled job

# .github/workflows/nightly-drift-check.yml
name: Nightly Terramate Drift Check

on:
  schedule:
    - cron: '0 3 * * *'  # 3 AM UTC daily

jobs:
  drift-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Terramate needs full history for change detection

      - name: Setup Terramate
        uses: terramate-io/terramate-action@v2

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Run drift detection
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          terramate run --changed -- terraform plan -detailed-exitcode

      - name: Send heartbeat on success
        if: success()
        run: curl -X POST "${{ secrets.VIGILMON_HEARTBEAT_URL }}"

Step 4: Alert Routing Strategy

Terramate alerts on infrastructure orchestration events: stack execution failures, dependency ordering errors, drift between desired and actual state. Vigilmon alerts on the services that Terramate deployed: an endpoint is down, a health check is failing, a drift-check job missed its window. These are distinct failure modes.

| Failure mode | Source | Route to | |---|---|---| | Stack apply failure | Terramate CI | Slack #infra-deployments + on-call | | Drift detected in production stack | Terramate | Infrastructure team + ticket | | Service endpoint down after deploy | Vigilmon | PagerDuty on-call + Slack #prod-alerts | | Drift check job missed | Vigilmon | Slack #infra-ops-critical | | TLS certificate expiring | Vigilmon | DevOps team |

Configure Vigilmon alert channels under Alerts → Add channel:

  • Email: your on-call distribution list.
  • PagerDuty webhook: wire to your primary on-call rotation for endpoint outages.
  • Slack webhook: a dedicated #vigilmon-alerts channel.

Step 5: Correlating Terramate Deployments and Vigilmon Alerts During Incidents

When Vigilmon fires a downtime alert for a service managed by Terramate, use this checklist:

  1. Check Vigilmon probe regions — all regions failing points to the service itself. One region failing suggests a DNS or routing issue.
  2. Check recent Terramate runs — did a stack apply complete just before the alert fired? A bad deployment is the most common cause of immediate post-deploy outages.
  3. Review the stack's Terraform state — compare the previous and current state to identify what changed. A misconfigured load balancer rule or security group can silently break reachability.
  4. Check dependent stacks — Terramate tracks inter-stack dependencies. If an upstream shared infrastructure stack (VPC, DNS, load balancer) was recently applied, it may have affected the failing service.
  5. Inspect the health check in detail — which dependency is failing? Database connectivity issues after a deploy often point to a migration failure or connection pool misconfiguration introduced in the same change.

Step 6: Public Status Page

Terramate run logs and drift reports are internal infrastructure telemetry. Give customers and API consumers a public-facing status page using Vigilmon.

  1. In Vigilmon, go to Status Pages → Create.
  2. Add your production service monitors and any critical heartbeat monitors.
  3. Configure a custom domain (e.g., status.example.com) or use the Vigilmon subdomain.
  4. Embed the status badge in your developer documentation:
<img src="https://vigilmon.online/badge/<monitor-id>.svg" alt="Service Status" />

What Vigilmon Adds to a Terramate Stack

| Capability | Terramate | Vigilmon | |---|---|---| | Stack change detection | Yes | No | | Terraform execution ordering | Yes | No | | Drift detection between runs | Yes | No | | Infrastructure code generation | Yes | No | | External synthetic HTTP probing | No | Yes | | DNS resolution failure detection | No | Yes | | TLS/SSL certificate expiry alerts | No | Yes | | Scheduled job heartbeat monitoring | No | Yes | | Public status page | No | Yes | | Independent of CI/CD pipeline health | No | Yes |

Terramate gives you precise orchestration over which Terraform stacks run, in what order, and when. Vigilmon gives you the external, synthetic view that tells you what your users actually experience after those stacks are deployed. Together they cover every dimension of infrastructure reliability: from orchestrated provisioning to customer-facing uptime.

Add external uptime monitoring to your Terramate-managed services today — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles