tutorial

Monitoring Scalr-Managed Infrastructure with Vigilmon: External Uptime, Health Checks & Heartbeat Monitoring

Practical guide to complementing Scalr's Terraform remote execution and policy enforcement with Vigilmon's external uptime checks — health endpoints, synthetic HTTP monitors, and independent alerting for infrastructure deployed through Scalr workspaces.

Scalr is a Terraform and OpenTofu remote execution backend that adds policy-as-code enforcement, hierarchical variable management, and team access controls on top of standard IaC workflows. It handles the orchestration and governance of infrastructure changes — but the moment a Terraform run completes, Scalr's responsibility ends. Whether the services that run completed successfully are actually reachable and responding correctly to real users is an entirely different question. Vigilmon answers that question. It sits outside your cloud environment and probes your deployed services from multiple external locations, alerting you whenever an endpoint goes down, a health check degrades, or a scheduled job silently stops running.

This tutorial covers adding external uptime monitoring to services deployed through Scalr workspaces.

What You'll Build

  • A /health endpoint in a service deployed through a Scalr workspace
  • A Vigilmon HTTP monitor with JSON body assertions
  • A heartbeat monitor for Scalr scheduled run triggers
  • Alert routing that keeps Scalr run failure alerts and Vigilmon uptime alerts distinct but correlated

Prerequisites

  • A Scalr account with at least one workspace containing a deployed service
  • A public HTTPS endpoint for the service deployed by the workspace
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Deployed Service

Scalr executes Terraform plans and applies. Once the run completes and the infrastructure is provisioned, Vigilmon needs an HTTP endpoint to probe. Build a health check that validates real runtime dependencies — not a static 200 that passes even when the service is unhealthy.

Node.js / Express

// src/health.js
import os from 'os';

export function registerHealthRoute(app, db) {
  app.get('/health', async (req, res) => {
    const checks = {};
    let ok = true;

    // Database connectivity
    try {
      await db.raw('SELECT 1');
      checks.database = 'ok';
    } catch (err) {
      checks.database = `error: ${err.message}`;
      ok = false;
    }

    // Upstream dependency
    try {
      const resp = await fetch(process.env.UPSTREAM_API_URL + '/ping', {
        signal: AbortSignal.timeout(5000),
      });
      checks.upstream_api = resp.ok ? 'ok' : `http_${resp.status}`;
      if (!resp.ok) ok = false;
    } catch (err) {
      checks.upstream_api = `error: ${err.message}`;
      ok = false;
    }

    res.status(ok ? 200 : 503).json({
      status: ok ? 'ok' : 'degraded',
      checks,
      host: os.hostname(),
    });
  });
}

Python (FastAPI)

# app/health.py
import os
import httpx
import asyncpg
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

@app.get("/health")
async def health():
    checks = {}
    ok = True

    try:
        conn = await asyncpg.connect(os.environ["DATABASE_URL"], timeout=3)
        await conn.fetchval("SELECT 1")
        await conn.close()
        checks["database"] = "ok"
    except Exception as exc:
        checks["database"] = f"error: {exc}"
        ok = False

    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            resp = await client.get(os.environ["UPSTREAM_API_URL"] + "/ping")
            checks["upstream_api"] = "ok" if resp.status_code == 200 else f"http_{resp.status_code}"
            if resp.status_code != 200:
                ok = False
    except Exception as exc:
        checks["upstream_api"] = f"error: {exc}"
        ok = False

    return JSONResponse(
        status_code=200 if ok else 503,
        content={"status": "ok" if ok else "degraded", "checks": checks},
    )

Terraform output for the health URL

Expose the health endpoint URL as a Terraform output in your Scalr workspace so it's always visible alongside the workspace run history:

# outputs.tf
output "service_health_url" {
  description = "Vigilmon monitors this endpoint for uptime"
  value       = "https://${aws_lb.main.dns_name}/health"
}

Verify the endpoint before configuring Vigilmon:

curl -s https://your-service.example.com/health | jq .
# {"status":"ok","checks":{"database":"ok","upstream_api":"ok"}}

Step 2: Configure Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-service.example.com/health
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status code: 200.
  6. JSON body assertion:
    • Path: status
    • Expected value: ok
  7. Click Save.

Vigilmon probes from multiple external locations concurrently. It requires a quorum of locations to agree on a failure before firing an alert — this eliminates false positives from transient regional routing issues that have nothing to do with your service.

Monitor per Scalr workspace

Each Scalr workspace typically manages one environment or microservice. Create a separate Vigilmon monitor for each production workspace endpoint:

| Scalr workspace | Service URL | Interval | |---|---|---| | api-production | https://api.example.com/health | 60 s | | worker-production | https://worker.example.com/health | 60 s | | frontend-production | https://app.example.com/health | 60 s | | api-staging | https://api-staging.example.com/health | 120 s |

Group monitors under a Monitor group named after the environment or team.

Step 3: Heartbeat Monitor for Scalr Scheduled Runs

Scalr can trigger workspace runs on a schedule — nightly cost analysis, daily drift detection, or weekly full plan previews. If those scheduled runs silently stop executing, you need to know before drift accumulates. Vigilmon heartbeat monitors detect silent job failures.

Add a heartbeat ping at the end of your Terraform module's null_resource provisioner or in the CI pipeline that triggers the Scalr run:

#!/bin/bash
# scripts/scheduled-drift-check.sh
set -euo pipefail

echo "Triggering Scalr workspace run..."
WORKSPACE_ID="${SCALR_WORKSPACE_ID}"

# Trigger the run via Scalr API
RUN_ID=$(curl -s -X POST \
  -H "Authorization: Bearer ${SCALR_TOKEN}" \
  -H "Content-Type: application/vnd.api+json" \
  "https://${SCALR_ACCOUNT}.scalr.io/api/iacp/v3/runs" \
  -d "{
    \"data\": {
      \"type\": \"runs\",
      \"relationships\": {
        \"workspace\": {
          \"data\": { \"type\": \"workspaces\", \"id\": \"${WORKSPACE_ID}\" }
        }
      }
    }
  }" | jq -r '.data.id')

echo "Run triggered: $RUN_ID"

# Wait for the run to complete (poll status)
while true; do
  STATUS=$(curl -s \
    -H "Authorization: Bearer ${SCALR_TOKEN}" \
    "https://${SCALR_ACCOUNT}.scalr.io/api/iacp/v3/runs/${RUN_ID}" \
    | jq -r '.data.attributes.status')
  echo "Run status: $STATUS"
  case "$STATUS" in
    applied|planned_and_finished) break ;;
    errored|canceled|force_canceled)
      echo "Run failed with status: $STATUS"
      exit 1
      ;;
  esac
  sleep 15
done

# Ping Vigilmon heartbeat on successful completion
curl -s -X POST "${VIGILMON_HEARTBEAT_URL}" --max-time 5
echo "Heartbeat sent"

In Vigilmon, create a Heartbeat monitor with a grace period 30 minutes longer than your scheduled run interval. Store the heartbeat URL as a Scalr workspace variable:

  1. In your Scalr workspace → Variables → Add variable.
  2. Name: VIGILMON_HEARTBEAT_URL, Category: Environment, sensitive: true.
  3. Paste the heartbeat URL from your Vigilmon monitor.

GitHub Actions trigger with heartbeat

# .github/workflows/nightly-drift-check.yml
name: Nightly Scalr Drift Check

on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily

jobs:
  drift-check:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger Scalr workspace run
        env:
          SCALR_TOKEN: ${{ secrets.SCALR_TOKEN }}
          SCALR_ACCOUNT: ${{ secrets.SCALR_ACCOUNT }}
          SCALR_WORKSPACE_ID: ${{ secrets.SCALR_WORKSPACE_ID }}
        run: |
          # Trigger and wait for completion
          bash scripts/scheduled-drift-check.sh

      - name: Send heartbeat on success
        if: success()
        run: curl -X POST "${{ secrets.VIGILMON_HEARTBEAT_URL }}"

Step 4: Alert Routing Strategy

Scalr fires alerts for infrastructure orchestration events: run failures, policy violations, workspace lock conflicts. Vigilmon fires alerts for service availability events: endpoint down, health check degraded, heartbeat missed. These overlap during deployments but come from independent systems.

| Failure mode | Source | Route to | |---|---|---| | Scalr run failure | Scalr | Slack #infra-deployments + on-call | | Policy check violation | Scalr | Infrastructure team + ticket | | Service endpoint down after deploy | Vigilmon | PagerDuty + Slack #prod-alerts | | Drift check job missed | Vigilmon | Slack #infra-ops-critical | | TLS certificate expiring | Vigilmon | DevOps team |

Configure Vigilmon alert channels under Alerts → Add channel:

  • Email: on-call distribution list.
  • PagerDuty webhook: primary on-call rotation for production endpoint outages.
  • Slack webhook: a #vigilmon-alerts channel with escalation rules.

Step 5: Investigating Vigilmon Alerts for Scalr-Managed Services

When Vigilmon fires a downtime alert for a service managed by a Scalr workspace, use this correlation checklist:

  1. Check Vigilmon probe regions — if all regions fail, the service itself is the problem. If one region fails, it may be DNS or routing.
  2. Review recent Scalr runs — did a workspace apply complete just before the alert fired? Post-deploy outages are typically caused by the most recent infrastructure change.
  3. Compare Terraform state revisions — Scalr stores the full state history. Compare the state before and after the failing run to identify what changed.
  4. Check policy runs — did a policy check pass but introduce a configuration that inadvertently broke connectivity? Security group or firewall rules are a common culprit.
  5. Look at dependent workspaces — if your service workspace shares infrastructure with a networking or DNS workspace, a change in a dependency workspace can break your service without any run in the service workspace itself.

Step 6: Public Status Page

Scalr run logs are internal infrastructure telemetry. Your customers and API consumers don't see them and shouldn't need to. Vigilmon's public status page gives them a transparent view of service availability.

  1. In Vigilmon → Status Pages → Create.
  2. Add your production service monitors and critical heartbeat monitors.
  3. Configure a custom domain (status.example.com) or use the Vigilmon subdomain.
  4. Add a status badge to your API documentation:
<img src="https://vigilmon.online/badge/<monitor-id>.svg" alt="API Status" />

What Vigilmon Adds to a Scalr Workflow

| Capability | Scalr | Vigilmon | |---|---|---| | Remote Terraform execution | Yes | No | | OPA policy enforcement | Yes | No | | Variable hierarchy management | Yes | No | | Drift detection between runs | Yes | No | | Run approval workflows | Yes | No | | External synthetic HTTP probing | No | Yes | | DNS resolution failure detection | No | Yes | | TLS/SSL certificate expiry alerts | No | Yes | | Scheduled job heartbeat monitoring | No | Yes | | Public status page | No | Yes | | Independent of CI/CD pipeline health | No | Yes |

Scalr gives you governance, access control, and policy enforcement for your Terraform workflows. Vigilmon gives you the independent, external view that tells you what users experience after those workflows complete. Together they close the loop between infrastructure orchestration and user-facing service reliability.

Add external uptime monitoring to your Scalr-managed services today — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles