tutorial

How to Monitor Pulumi: Cloud API Availability, State Storage Health, and Stack Operation Reliability with Vigilmon

Pulumi Cloud outages and state storage failures silently break infrastructure delivery. Learn how to monitor Pulumi Cloud API availability, state storage health, Automation API uptime, and stack operation failure rates with Vigilmon.

Pulumi is a modern infrastructure-as-code platform that lets you write cloud infrastructure in Python, TypeScript, Go, and other general-purpose languages. Unlike Terraform/OpenTofu, Pulumi stores state in Pulumi Cloud (or self-managed backends) and exposes a rich Automation API for programmatic stack operations. When Pulumi breaks — a Pulumi Cloud outage, a corrupted state file, or an Automation API timeout — your infrastructure delivery stops without obvious external errors.

Vigilmon gives you external monitoring for every Pulumi dependency: the Pulumi Cloud API, your state backend, Automation API health, and stack operation failure rates. This tutorial covers all four layers.

Pulumi's Failure Surfaces

Understanding what can fail helps you build the right monitors:

  • Pulumi Cloud API (api.pulumi.com) — stack metadata, webhooks, policy enforcement, and state storage for hosted backends all depend on this
  • Self-managed state backends (S3, GCS, Azure Blob) — if you self-host state, connectivity failures block all stack operations
  • Automation API — programmatic deployments via the Automation API can time out or fail silently in CI
  • Stack operation pipeline — high failure rates on pulumi up/pulumi preview indicate provider API issues or config drift

Step 1: Monitor Pulumi Cloud API Availability

If you use Pulumi Cloud as your state backend, its API is your highest-priority dependency.

# Pulumi Cloud API health — returns 200 when operational
curl -i https://api.pulumi.com/api/status

# With authentication (for organization-specific checks)
curl -H "Authorization: token $PULUMI_ACCESS_TOKEN" https://api.pulumi.com/api/user

Configure Vigilmon to probe https://api.pulumi.com/api/status directly as an HTTP monitor — no sidecar needed for public availability checking. Point it at this URL with a 5-minute interval and P1 alerting.

Pulumi Cloud Org and Stack Health Sidecar

For authenticated checks (stacks, webhooks, policy packs), expose a health endpoint that queries your org:

// health/pulumi.js
const express = require('express');
const fetch = require('node-fetch');

const app = express();
const PULUMI_API = process.env.PULUMI_API_URL || 'https://api.pulumi.com';
const PULUMI_TOKEN = process.env.PULUMI_ACCESS_TOKEN;
const PULUMI_ORG = process.env.PULUMI_ORG;

async function pulumiGet(path) {
  const res = await fetch(`${PULUMI_API}${path}`, {
    headers: { Authorization: `token ${PULUMI_TOKEN}` },
    signal: AbortSignal.timeout(10_000),
  });
  if (!res.ok) throw new Error(`HTTP ${res.status} from ${path}`);
  return res.json();
}

app.get('/health/pulumi', async (req, res) => {
  try {
    // Check API connectivity and token validity
    const user = await pulumiGet('/api/user');

    // Check org accessibility
    const stacks = await pulumiGet(`/api/stacks?organization=${PULUMI_ORG}&pageSize=1`);

    return res.status(200).json({
      status: 'ok',
      user: user.githubLogin || user.name,
      org: PULUMI_ORG,
      stacks_accessible: true,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3011, () => console.log('Pulumi health sidecar running on :3011'));

Step 2: Monitor Self-Managed State Storage

If you use S3, GCS, or Azure Blob for Pulumi state (via PULUMI_BACKEND_URL), monitor the backend directly:

const { S3Client, ListObjectsV2Command } = require('@aws-sdk/client-s3');

const s3 = new S3Client({ region: process.env.AWS_REGION });

app.get('/health/pulumi/state', async (req, res) => {
  const bucket = process.env.PULUMI_STATE_BUCKET;
  try {
    const listRes = await s3.send(new ListObjectsV2Command({
      Bucket: bucket,
      Prefix: process.env.PULUMI_STATE_PREFIX || '.pulumi/',
      MaxKeys: 5,
    }));

    // Check for lock files older than 1 hour (stuck stack operation)
    const locks = (listRes.Contents || []).filter(obj => obj.Key.includes('.json.lock'));
    const staleLocks = locks.filter(lock => {
      const ageMinutes = (Date.now() - lock.LastModified.getTime()) / 60_000;
      return ageMinutes > 60;
    });

    if (staleLocks.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'stale_stack_locks',
        locks: staleLocks.map(l => l.Key),
      });
    }

    return res.status(200).json({ status: 'ok', state_objects: listRes.KeyCount });
  } catch (err) {
    return res.status(503).json({ status: 'down', bucket, error: err.message });
  }
});

Step 3: Monitor Automation API Health

The Pulumi Automation API lets you embed pulumi up/pulumi preview directly in your applications and CI systems. If your platform runs Automation API operations, expose a health endpoint that verifies the runtime:

# health/pulumi_automation.py
from flask import Flask, jsonify
import pulumi.automation as auto
import os

app = Flask(__name__)
STACK_NAME = os.environ.get('PULUMI_HEALTH_STACK', 'prod')
PROJECT_NAME = os.environ.get('PULUMI_PROJECT')
PULUMI_ORG = os.environ.get('PULUMI_ORG')

@app.route('/health/pulumi/automation')
def automation_health():
    try:
        # List stacks — lightweight operation that validates API connectivity and local SDK
        stacks = auto.list_stacks(
            organization=PULUMI_ORG,
            project_name=PROJECT_NAME,
        )
        return jsonify(status='ok', stacks=len(stacks)), 200
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

if __name__ == '__main__':
    app.run(port=3012)

For TypeScript-based Automation API:

// health/pulumi-automation.ts
import express from 'express';
import { listStacks } from '@pulumi/pulumi/automation';

const app = express();

app.get('/health/pulumi/automation', async (req, res) => {
  try {
    const stacks = await listStacks({
      organization: process.env.PULUMI_ORG!,
      program: undefined,
    });
    res.status(200).json({ status: 'ok', stacks: stacks.length });
  } catch (err: any) {
    res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3012);

Step 4: Monitor Stack Operation Failure Rates

Stack operation failures (failed pulumi up runs) indicate provider API issues, config drift, or policy violations. Export failure rates to an HTTP endpoint:

# health/pulumi_ops.py — reads Pulumi Cloud update history
import requests, os
from flask import Flask, jsonify

app = Flask(__name__)
PULUMI_API = os.environ.get('PULUMI_API_URL', 'https://api.pulumi.com')
PULUMI_TOKEN = os.environ['PULUMI_ACCESS_TOKEN']
PULUMI_ORG = os.environ['PULUMI_ORG']
STACKS_TO_MONITOR = os.environ.get('PULUMI_MONITOR_STACKS', '').split(',')
FAILURE_THRESHOLD = float(os.environ.get('PULUMI_FAILURE_THRESHOLD', '0.25'))

def get_stack_updates(stack_name, limit=20):
    headers = {'Authorization': f'token {PULUMI_TOKEN}'}
    r = requests.get(
        f'{PULUMI_API}/api/stacks/{PULUMI_ORG}/default/{stack_name}/updates',
        headers=headers,
        params={'pageSize': limit},
        timeout=10,
    )
    if r.status_code != 200:
        raise Exception(f'API error {r.status_code}')
    return r.json().get('updates', [])

@app.route('/health/pulumi/operations')
def operations_health():
    issues = []
    for stack in STACKS_TO_MONITOR:
        if not stack:
            continue
        try:
            updates = get_stack_updates(stack.strip())
            total = len(updates)
            if total == 0:
                continue
            failed = sum(1 for u in updates if u.get('result') == 'failed')
            rate = failed / total
            if rate > FAILURE_THRESHOLD:
                issues.append({'stack': stack, 'failure_rate': round(rate, 2), 'failed': failed, 'total': total})
        except Exception as e:
            issues.append({'stack': stack, 'error': str(e)})

    if issues:
        return jsonify(status='degraded', reason='high_failure_rate', issues=issues), 503
    return jsonify(status='ok', stacks_checked=len(STACKS_TO_MONITOR)), 200

if __name__ == '__main__':
    app.run(port=3013)

Step 5: Configure Vigilmon Monitors for Pulumi

  1. Log in to vigilmon.online → Monitors → New Monitor → HTTP / HTTPS

Set up the following monitors:

| Monitor | URL | Interval | Priority | |---|---|---|---| | Pulumi Cloud API | https://api.pulumi.com/api/status | 2 minutes | P1 | | Org & stacks health | /health/pulumi | 5 minutes | P1 | | State storage (S3/GCS) | /health/pulumi/state | 2 minutes | P1 | | Automation API | /health/pulumi/automation | 5 minutes | P2 | | Stack operation failure rate | /health/pulumi/operations | 10 minutes | P2 |

For each monitor configure:

  • Expected status: 200
  • Response body contains: "status":"ok"
  • Response time threshold: 15 000ms for the Pulumi Cloud API check (Pulumi Cloud can be slow under load)
  • Alert channel: #infra-oncall Slack for P1 monitors

Step 6: Heartbeat Monitor for Pulumi Automation Jobs

For scheduled Pulumi operations (nightly drift checks, automated provisioning), configure a heartbeat to detect silent job failures:

  1. Vigilmon → Monitors → New Monitor → Heartbeat
  2. Name: pulumi-nightly-drift-check
  3. Expected interval: 24 hours
  4. Grace period: 2 hours
  5. Copy the heartbeat URL

Add the ping at the end of your scheduled Pulumi script:

#!/bin/bash
set -euo pipefail

pulumi login
pulumi stack select "$PULUMI_STACK"
pulumi preview --diff

# Signal Vigilmon: the scheduled job ran
curl -s "$VIGILMON_HEARTBEAT_URL" > /dev/null

Summary

Pulumi IaC platform reliability requires monitoring at every layer your stacks depend on:

| Monitor Type | What It Covers | |---|---| | HTTP: Pulumi Cloud API | Hosted state backend and API availability | | HTTP: self-managed state | S3/GCS backend access, stale lock detection | | HTTP: Automation API | SDK and programmatic operation health | | HTTP: stack operations | Failure rate trends across stacks | | Heartbeat | Scheduled drift detection job liveness |

Get started free at vigilmon.online — your Pulumi platform monitors are live in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles