Uptime monitoring for Cloud Custodian cloud governance

Cloud Custodian enforces your organization's cloud policies — tagging standards, idle resource cleanup, security group restrictions, compliance rules. When the Cloud Custodian workers stop running, policies stop executing silently. Resources accumulate without tags, security groups drift, and costs climb. Because Cloud Custodian runs as a policy engine rather than a user-facing service, no alarm goes off when it fails — until an audit or a runaway cost bill surfaces the gap.

This tutorial covers production-grade uptime monitoring for Cloud Custodian using Vigilmon. We will walk through:

• Monitoring the Cloud Custodian Mailer and policy worker processes
• Exposing a health endpoint from Cloud Custodian server mode
• Monitoring policy execution logs for silent failures
• Webhook alerts for DOWN/UP events

Prerequisites

• Cloud Custodian (c7n) installed and running policies against AWS, Azure, or GCP
• A free account at vigilmon.online

Part 1: Expose a health endpoint from Cloud Custodian server mode

Cloud Custodian can run in server mode (c7n-org run) to continuously execute policies. In server mode you can expose a simple HTTP health endpoint for external monitoring.

Option A — wrap Cloud Custodian in a health-check sidecar

The simplest pattern is a small sidecar HTTP server that monitors the Cloud Custodian process and responds to health check requests:

#!/usr/bin/env python3
# health_server.py — run alongside your c7n policy workers
import subprocess
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler

def is_c7n_running():
    result = subprocess.run(
        ["pgrep", "-f", "custodian"],
        capture_output=True,
        text=True
    )
    return result.returncode == 0

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            if is_c7n_running():
                self.send_response(200)
                self.send_header("Content-Type", "application/json")
                self.end_headers()
                self.wfile.write(b'{"status":"ok","worker":"running"}')
            else:
                self.send_response(503)
                self.send_header("Content-Type", "application/json")
                self.end_headers()
                self.wfile.write(b'{"status":"error","worker":"stopped"}')
        else:
            self.send_response(404)
            self.end_headers()

    def log_message(self, format, *args):
        pass  # suppress access log noise

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8090), HealthHandler)
    print("Health server listening on :8090")
    server.serve_forever()

Run it alongside your Cloud Custodian policies:

# Start health server
python3 health_server.py &

# Run Cloud Custodian policies
custodian run --output-dir=output policy.yml

Verify:

curl http://localhost:8090/health
# {"status":"ok","worker":"running"}

Option B — use a heartbeat file approach

Cloud Custodian writes output files after each policy execution. A sidecar can check the modification time of these files and return 503 if they haven't been updated recently:

#!/usr/bin/env python3
# heartbeat_health.py
import os
import time
from http.server import HTTPServer, BaseHTTPRequestHandler

OUTPUT_DIR = "/var/c7n/output"
MAX_STALE_SECONDS = 300  # alert if no policy ran in 5 minutes

def last_execution_age():
    latest = 0
    for root, dirs, files in os.walk(OUTPUT_DIR):
        for f in files:
            if f.endswith(".json"):
                mtime = os.path.getmtime(os.path.join(root, f))
                latest = max(latest, mtime)
    if latest == 0:
        return float("inf")
    return time.time() - latest

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            age = last_execution_age()
            if age < MAX_STALE_SECONDS:
                self.send_response(200)
                self.send_header("Content-Type", "application/json")
                self.end_headers()
                self.wfile.write(
                    f'{{"status":"ok","last_execution_age_seconds":{int(age)}}}'.encode()
                )
            else:
                self.send_response(503)
                self.send_header("Content-Type", "application/json")
                self.end_headers()
                self.wfile.write(
                    f'{{"status":"error","last_execution_age_seconds":{int(age)}}}'.encode()
                )
        else:
            self.send_response(404)
            self.end_headers()

    def log_message(self, format, *args):
        pass

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8090), HealthHandler)
    server.serve_forever()

Part 2: Docker Compose deployment

# docker-compose.yml
version: "3.8"

services:
  c7n-worker:
    image: cloudcustodian/c7n:latest
    command: >
      custodian run
        --output-dir=/output
        --cache-period=60
        /policies/policy.yml
    environment:
      AWS_DEFAULT_REGION: us-east-1
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    volumes:
      - ./policies:/policies:ro
      - c7n_output:/output
    restart: unless-stopped

  health-server:
    image: python:3.12-slim
    command: python /app/heartbeat_health.py
    ports:
      - "8090:8090"
    volumes:
      - ./heartbeat_health.py:/app/heartbeat_health.py:ro
      - c7n_output:/var/c7n/output:ro
    restart: unless-stopped

volumes:
  c7n_output:

Kubernetes deployment

# c7n-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: c7n-worker
  namespace: governance
spec:
  replicas: 1
  selector:
    matchLabels:
      app: c7n-worker
  template:
    metadata:
      labels:
        app: c7n-worker
    spec:
      containers:
        - name: c7n
          image: cloudcustodian/c7n:latest
          command: ["custodian", "run", "--output-dir=/output", "/policies/policy.yml"]
          volumeMounts:
            - name: output
              mountPath: /output
            - name: policies
              mountPath: /policies

        - name: health
          image: python:3.12-slim
          command: ["python", "/app/heartbeat_health.py"]
          ports:
            - containerPort: 8090
          volumeMounts:
            - name: output
              mountPath: /var/c7n/output
          livenessProbe:
            httpGet:
              path: /health
              port: 8090
            periodSeconds: 60

      volumes:
        - name: output
          emptyDir: {}
        - name: policies
          configMap:
            name: c7n-policies

Part 3: Set up HTTP monitoring in Vigilmon

Monitor Cloud Custodian worker health

1. Log in to vigilmon.online and click Add Monitor.
2. Choose HTTP(S) monitor.
3. Enter: https://custodian.example.com/health
4. Set interval to 5 minutes (Cloud Custodian policies run on schedules; a 5-minute interval matches typical execution windows).
5. Add a keyword check: must contain "status":"ok".
6. Add your alert channel.
7. Click Save.

Monitor Cloud Custodian Mailer

Cloud Custodian Mailer (c7n-mailer) handles policy notifications. If the Mailer worker stops, policy violations are silently swallowed — no emails, no Slack messages reach your teams.

Add a separate heartbeat monitor for the Mailer process using the same sidecar pattern:

# Start the Mailer
c7n-mailer --config mailer.yml &

# Monitor it on a separate port
MAILER_PID=$!

Or expose it as a separate service and add another Vigilmon monitor:

| Service | URL | Keyword | Interval | |---------|-----|---------|----------| | Policy worker | https://custodian.example.com/health | "status":"ok" | 5 min | | Mailer worker | https://custodian-mailer.example.com/health | "status":"ok" | 5 min |

Part 4: Monitoring policy execution via a scheduled job health endpoint

For policies run on a cron schedule (rather than in continuous server mode), use a heartbeat / dead man's switch pattern. After each successful policy execution, the script POSTs to Vigilmon's heartbeat endpoint. Vigilmon alerts if no heartbeat arrives within the expected window.

#!/bin/bash
# run-policies.sh

set -euo pipefail

echo "Running Cloud Custodian policies..."
custodian run --output-dir=/var/c7n/output /policies/policy.yml

if [ $? -eq 0 ]; then
  echo "Policies ran successfully"
  # Ping Vigilmon heartbeat URL to confirm successful execution
  curl -s -o /dev/null "https://vigilmon.online/api/heartbeat/your-heartbeat-id"
else
  echo "Policy execution failed" >&2
  exit 1
fi

Configure your cron job:

# /etc/cron.d/c7n-policies
*/15 * * * * c7n-user /opt/c7n/run-policies.sh >> /var/log/c7n-policies.log 2>&1

In Vigilmon, set the heartbeat window to 20 minutes — if no ping arrives within 20 minutes of the expected interval, Vigilmon sends a DOWN alert.

Part 5: SSL certificate monitoring

If your health endpoint is behind a public URL with TLS:

1. In Vigilmon, click Add Monitor.
2. Choose SSL monitor.
3. Enter: custodian.example.com
4. Set alert threshold to 14 days before expiry.
5. Add your alert channel.

Part 6: Webhook alerts

// webhook-receiver.ts
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhook/vigilmon', (req, res) => {
  const { monitor_name, status, url, response_code, checked_at } = req.body;

  if (status === 'down') {
    console.error('[VIGILMON] Cloud Custodian DOWN', {
      monitor: monitor_name,
      url,
      code: response_code,
      at: checked_at,
    });

    // Cloud Custodian going down is a compliance risk — page immediately
    createIncidentTicket({
      title: `Cloud Custodian worker is DOWN`,
      severity: 'high',
      url,
      details: `Monitor: ${monitor_name}\nChecked at: ${checked_at}`,
    });
  } else if (status === 'up') {
    console.info('[VIGILMON] Cloud Custodian recovered', { monitor: monitor_name });
  }

  res.sendStatus(204);
});

app.listen(3000);

Vigilmon sends this payload on DOWN and UP transitions:

{
  "monitor_id": "mon_abc123",
  "monitor_name": "Cloud Custodian Worker",
  "status": "down",
  "url": "https://custodian.example.com/health",
  "checked_at": "2026-06-30T09:00:00Z",
  "response_code": 503,
  "response_time_ms": 5001
}

Summary

Your Cloud Custodian deployment now has four layers of monitoring:

1. Process health endpoint — a sidecar confirms the Cloud Custodian worker process is running and has executed policies recently, polled every 5 minutes by Vigilmon.
2. Mailer worker monitor — a separate check ensures policy violation notifications are reaching your teams.
3. Heartbeat/dead man's switch — scheduled policy runs POST a ping to Vigilmon; missing a window triggers a DOWN alert.
4. Webhook alerts — DOWN events create high-severity incidents immediately, because a stopped Cloud Custodian is a compliance and cost risk.

Vigilmon handles check scheduling, alert routing, and uptime history so your cloud governance is itself governed.

Monitor your Cloud Custodian infrastructure free at vigilmon.online

#cloudcustodian #cloudgovernance #finops #devops #monitoring