tutorial

How to Monitor Brigade with Vigilmon (Event Gateway, Worker Queue + Alerts)

Brigade is a Kubernetes-native event-driven scripting platform that lets you write JavaScript or TypeScript event handlers that run as ephemeral worker pods ...

Brigade is a Kubernetes-native event-driven scripting platform that lets you write JavaScript or TypeScript event handlers that run as ephemeral worker pods in response to events — GitHub webhooks, cron schedules, CloudEvents, or any custom trigger. Teams use Brigade for CI/CD automation, GitOps workflows, scheduled batch jobs, and anything that needs event-driven execution on Kubernetes without standing up a full CI server.

But Brigade's distributed, ephemeral architecture makes monitoring non-trivial. A Brigade gateway can silently stop accepting events, a worker pod queue can back up under load, and a long-running script job can stall without any visible signal. In this tutorial you'll set up comprehensive monitoring for your Brigade deployment using Vigilmon — free tier, no credit card required.

Why Brigade needs external monitoring

Brigade consists of multiple components — the API server, event gateways, the scheduler, and ephemeral worker pods — each of which can fail independently. Brigade's own observability surface is limited to brig CLI queries and Kubernetes events.

The failure modes that external monitoring catches:

  • Event gateway down — the GitHub gateway or custom gateway stops accepting webhook payloads; GitHub shows 200 OK for delivery attempts but Brigade never processes the event
  • Worker queue depth spiking — a burst of events creates a backlog of pending worker pods; if the Kubernetes scheduler can't place them, they queue indefinitely
  • API server unreachable — the Brigade API server is the control plane for all event and job management; if it goes down, no new events are processed and the brig CLI fails
  • Worker pods stuck in Pending — resource constraints, node taints, or misconfigured resource requests cause worker pods to queue without ever starting
  • Substrate connectivity broken — Brigade's substrate (the Kubernetes cluster it targets) may become temporarily inaccessible; new events queue but workers can't launch
  • Script execution timeout — a Brigade script with a slow external dependency stalls the worker pod; it eventually times out, but the timeout window (default 24h) is too long for operational awareness

What you'll need

  • Brigade v2 deployed in a Kubernetes cluster (brig CLI configured)
  • At least one gateway configured (GitHub gateway, generic gateway, or cron gateway)
  • A free Vigilmon account — sign up takes 30 seconds

Step 1: Monitor the Brigade API server

The Brigade API server is the control plane. Without it, no events are processed and no workers can be scheduled.

  1. Go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://brigade.example.com/healthz
  4. Interval: 1 minute
  5. Expected status: 200
  6. Name: Brigade API Server Health

If your Brigade API server is exposed internally only, use a Vigilmon agent or probe script:

#!/usr/bin/env bash
# brigade-api-probe.sh — check Brigade API server health

BRIGADE_API="https://brigade.internal.example.com"
HEARTBEAT_URL="$VIGILMON_API_HEARTBEAT_URL"

STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$BRIGADE_API/healthz")

if [ "$STATUS" != "200" ]; then
  echo "Brigade API server unhealthy: HTTP $STATUS"
  exit 1
fi

curl -s "$HEARTBEAT_URL"
echo "Brigade API server healthy"

Schedule every 2 minutes with a 5-minute heartbeat grace period.

Step 2: Monitor the event gateway

Brigade's event gateways are the ingestion points for external events. Monitor each gateway you operate.

GitHub gateway

  1. New monitor → HTTP / HTTPS
  2. URL: https://brigade-github-gateway.example.com/healthz
  3. Interval: 1 minute
  4. Expected status: 200
  5. Name: Brigade GitHub Gateway Health

Generic/CloudEvents gateway

  1. New monitor → HTTP / HTTPS
  2. URL: https://brigade-gateway.example.com/healthz
  3. Interval: 1 minute
  4. Expected status: 200
  5. Name: Brigade Generic Gateway Health

If your gateway doesn't expose a /healthz endpoint, monitor the TCP port the gateway listens on:

  1. New monitor → TCP Port
  2. Host: brigade-gateway.example.com, Port: 443
  3. Name: Brigade Gateway TCP

Test webhook delivery end-to-end

The most reliable test for gateway health is whether a test webhook payload is accepted and processed. Set up a synthetic event probe:

#!/usr/bin/env bash
# brigade-gateway-probe.sh — send a test event and verify Brigade processes it

BRIGADE_API="https://brigade.internal.example.com"
BRIGADE_TOKEN="$BRIGADE_API_TOKEN"
PROJECT_ID="my-test-project"
HEARTBEAT_URL="$VIGILMON_GATEWAY_HEARTBEAT_URL"
TIMEOUT_SECONDS=120

# Create a test event via the Brigade API
EVENT_ID=$(curl -s \
  -H "Authorization: Bearer $BRIGADE_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  "$BRIGADE_API/v2/events" \
  -d "{
    \"projectID\": \"$PROJECT_ID\",
    \"source\": \"monitoring.probe\",
    \"type\": \"health.check\",
    \"payload\": \"{\\\"probe\\\": true}\"
  }" | jq -r '.id')

if [ -z "$EVENT_ID" ] || [ "$EVENT_ID" = "null" ]; then
  echo "Failed to create Brigade test event"
  exit 1
fi

echo "Created Brigade event: $EVENT_ID"

# Poll until the event's worker reaches a terminal state
START=$(date +%s)
while true; do
  NOW=$(date +%s)
  ELAPSED=$(( NOW - START ))

  if [ $ELAPSED -gt $TIMEOUT_SECONDS ]; then
    echo "TIMEOUT: Brigade worker did not complete within ${TIMEOUT_SECONDS}s"
    exit 1
  fi

  WORKER_PHASE=$(curl -s \
    -H "Authorization: Bearer $BRIGADE_TOKEN" \
    "$BRIGADE_API/v2/events/$EVENT_ID/worker" \
    | jq -r '.status.phase')

  echo "Worker phase: $WORKER_PHASE (${ELAPSED}s elapsed)"

  case "$WORKER_PHASE" in
    "SUCCEEDED")
      echo "Brigade probe event succeeded"
      curl -s "$HEARTBEAT_URL"
      exit 0
      ;;
    "FAILED"|"ABORTED"|"TIMED_OUT")
      echo "Brigade probe event FAILED with phase: $WORKER_PHASE"
      exit 1
      ;;
  esac

  sleep 5
done

Schedule every 10 minutes with a Vigilmon heartbeat grace period of 20 minutes. This probe exercises the complete Brigade event path: gateway → API server → scheduler → worker pod → script execution → result.

Step 3: Monitor worker queue depth

When Brigade's worker queue backs up, event processing latency grows and eventually events start failing. Monitor queue depth directly from the Brigade API:

#!/usr/bin/env bash
# brigade-queue-probe.sh — monitor pending worker count

BRIGADE_API="https://brigade.internal.example.com"
BRIGADE_TOKEN="$BRIGADE_API_TOKEN"
MAX_PENDING=10
HEARTBEAT_URL="$VIGILMON_QUEUE_HEARTBEAT_URL"

# Count workers in PENDING phase across all projects
PENDING=$(curl -s \
  -H "Authorization: Bearer $BRIGADE_TOKEN" \
  "$BRIGADE_API/v2/events?workerPhase=PENDING&limit=100" \
  | jq '.metadata.itemCount // 0')

RUNNING=$(curl -s \
  -H "Authorization: Bearer $BRIGADE_TOKEN" \
  "$BRIGADE_API/v2/events?workerPhase=RUNNING&limit=100" \
  | jq '.metadata.itemCount // 0')

echo "Brigade workers — Pending: $PENDING, Running: $RUNNING"

if [ "$PENDING" -gt "$MAX_PENDING" ]; then
  echo "QUEUE BACKLOG: $PENDING workers pending (threshold: $MAX_PENDING)"
  exit 1
fi

curl -s "$HEARTBEAT_URL"

Schedule every 3 minutes with a 10-minute heartbeat grace period.

For Kubernetes-side visibility into pending pods:

#!/usr/bin/env bash
# brigade-pods-probe.sh — check for brigade worker pods stuck in Pending

NAMESPACE="brigade"
MAX_PENDING_AGE_MINUTES=10
HEARTBEAT_URL="$VIGILMON_PODS_HEARTBEAT_URL"

# Find worker pods in Pending state older than threshold
STUCK_PODS=$(kubectl get pods \
  -n "$NAMESPACE" \
  -l "brigade.sh/role=worker" \
  --field-selector="status.phase=Pending" \
  -o json | jq --argjson max_age $(( MAX_PENDING_AGE_MINUTES * 60 )) '
    [.items[] |
      select(
        (now - (.metadata.creationTimestamp | fromdateiso8601)) > $max_age
      ) | .metadata.name
    ] | length
  ')

if [ "$STUCK_PODS" -gt 0 ]; then
  echo "STUCK: $STUCK_PODS Brigade worker pod(s) Pending for >${MAX_PENDING_AGE_MINUTES} minutes"
  kubectl get pods -n "$NAMESPACE" -l "brigade.sh/role=worker" \
    --field-selector="status.phase=Pending"
  exit 1
fi

echo "No stuck Brigade worker pods"
curl -s "$HEARTBEAT_URL"

Step 4: Monitor script execution latency

Brigade script execution latency is how long it takes from event received to worker completion. When latency spikes, downstream systems waiting on Brigade (deployment pipelines, notification systems) accumulate delays.

Instrument your Brigade project's worker script to report timing:

// brigade.js — add timing instrumentation
const { events } = require("@brigadecore/brigadier");

events.on("monitoring.probe", "health.check", async (event) => {
  const start = Date.now();

  // ... your actual work ...
  await doWork(event);

  const elapsed = Date.now() - start;
  console.log(`[METRICS] script_duration_ms=${elapsed}`);

  // Report heartbeat with timing
  const heartbeatUrl = event.project.secrets.VIGILMON_HEARTBEAT_URL;
  await fetch(heartbeatUrl);
});

events.process();

Then scrape the execution time from Brigade worker logs:

#!/usr/bin/env bash
# brigade-latency-probe.sh — check recent worker execution times

BRIGADE_API="https://brigade.internal.example.com"
BRIGADE_TOKEN="$BRIGADE_API_TOKEN"
MAX_DURATION_MS=30000  # Alert if script takes more than 30 seconds
HEARTBEAT_URL="$VIGILMON_LATENCY_HEARTBEAT_URL"

# Get logs from the most recent SUCCEEDED worker
LATEST_SUCCEEDED=$(curl -s \
  -H "Authorization: Bearer $BRIGADE_TOKEN" \
  "$BRIGADE_API/v2/events?workerPhase=SUCCEEDED&limit=1" \
  | jq -r '.items[0].id')

if [ -z "$LATEST_SUCCEEDED" ] || [ "$LATEST_SUCCEEDED" = "null" ]; then
  echo "No recent succeeded workers found"
  exit 1
fi

DURATION=$(curl -s \
  -H "Authorization: Bearer $BRIGADE_TOKEN" \
  "$BRIGADE_API/v2/events/$LATEST_SUCCEEDED/worker/logs?container=brigadier" \
  | jq -r '.[] | select(.message | contains("script_duration_ms")) | .message' \
  | grep -oP 'script_duration_ms=\K[0-9]+' | tail -1)

if [ -z "$DURATION" ]; then
  echo "Could not parse execution duration from worker logs"
  exit 1
fi

echo "Most recent script duration: ${DURATION}ms"

if [ "$DURATION" -gt "$MAX_DURATION_MS" ]; then
  echo "LATENCY: ${DURATION}ms exceeds threshold ${MAX_DURATION_MS}ms"
  exit 1
fi

curl -s "$HEARTBEAT_URL"

Step 5: Monitor substrate connectivity

Brigade's substrate is the Kubernetes cluster it launches worker pods in. If the substrate becomes temporarily inaccessible — due to API server issues, network partition, or expired credentials — Brigade can't launch any workers.

Monitor substrate connectivity with a Kubernetes API probe:

#!/usr/bin/env bash
# brigade-substrate-probe.sh — verify Brigade can reach Kubernetes substrate

BRIGADE_API="https://brigade.internal.example.com"
BRIGADE_TOKEN="$BRIGADE_API_TOKEN"
HEARTBEAT_URL="$VIGILMON_SUBSTRATE_HEARTBEAT_URL"

# Brigade exposes a substrate connectivity check via its API
SUBSTRATE_STATUS=$(curl -s \
  -H "Authorization: Bearer $BRIGADE_TOKEN" \
  "$BRIGADE_API/v2/substrate/nodes?limit=1" \
  | jq -r '.items | length')

if [ -z "$SUBSTRATE_STATUS" ] || [ "$SUBSTRATE_STATUS" = "null" ]; then
  echo "Brigade cannot reach substrate or returned unexpected response"
  exit 1
fi

echo "Brigade substrate accessible: $SUBSTRATE_STATUS node(s) visible"
curl -s "$HEARTBEAT_URL"

Additionally, monitor the Kubernetes API server that Brigade targets:

  1. New monitor → HTTP / HTTPS
  2. URL: https://k8s-cluster.example.com:6443/healthz
  3. Interval: 1 minute
  4. Expected status: 200
  5. Name: Brigade Substrate (K8s API)

Step 6: Monitor cron gateway schedules

If you use Brigade's cron gateway to trigger scheduled events, set up Vigilmon heartbeats to verify each scheduled job runs on time:

// brigade.js — cron job that reports to Vigilmon
const { events } = require("@brigadecore/brigadier");

events.on("brigade.sh/cron", "backup", async (event) => {
  // ... perform backup work ...
  await runBackup();

  // Report to Vigilmon that this cron fired and succeeded
  const heartbeatUrl = event.project.secrets.VIGILMON_BACKUP_HEARTBEAT_URL;
  await fetch(heartbeatUrl);
});

events.process();

For a cron gateway event that fires daily at 03:00:

  1. In Vigilmon: Monitors → New Monitor → Heartbeat
  2. Name: Brigade Daily Backup Job
  3. Grace period: 25 hours

If the Brigade cron gateway stops firing — because the gateway crashed, the cron schedule was misconfigured, or Brigade can't reach the substrate — Vigilmon alerts you within 25 hours.

Step 7: Set up alerts

Configure Vigilmon alerts for all Brigade monitors:

| Monitor | Alert threshold | Notification | |---|---|---| | Brigade API Server Health | 2 minutes down | Slack #platform-alerts, PagerDuty | | GitHub Gateway Health | 2 minutes down | Slack #ci-health | | Generic Gateway Health | 2 minutes down | Slack #platform-alerts | | End-to-End Probe heartbeat | 20-minute silence | Slack #ci-health | | Queue Depth heartbeat | 10-minute silence | Slack #platform-alerts | | Pending Pods heartbeat | 10-minute silence | Slack #k8s-ops | | Script Latency heartbeat | 30-minute silence | Slack #ci-health | | Substrate heartbeat | 10-minute silence | PagerDuty | | Cron Job heartbeats | Per-schedule grace | Slack #platform-alerts |

For the Brigade API server and substrate monitors, escalate to PagerDuty immediately — without these, no Brigade workloads can run.

Step 8: Create a Brigade status page

Create a Vigilmon status page that summarizes Brigade health for your team:

  1. Go to Status Pages → New
  2. Name: Brigade Platform Status
  3. Add monitors:
    • Brigade API Server
    • GitHub Gateway
    • Worker Queue Depth
    • Substrate Connectivity
  4. Share the URL in your team's wiki or Slack channel description

This gives developers a quick reference when their Brigade-triggered CI jobs hang.

Interpreting Vigilmon alerts for Brigade

| Alert | Likely cause | First action | |---|---|---| | Brigade API Server down | Pod crashed, OOM, deployment rollout | kubectl get pods -n brigade; kubectl logs -n brigade -l app=brigade-apiserver | | GitHub Gateway down | Gateway pod crash, ingress misconfiguration | Check gateway pod; test with a manual webhook delivery from GitHub | | End-to-End Probe silent | Full pipeline failure, probe script error | Run brigade-gateway-probe.sh manually; check brig events list | | Queue Depth silent | High event volume, scheduler issue, node pressure | brig events list --worker-phase PENDING; check node capacity | | Pending Pods silent | Node resource pressure, PodDisruptionBudget, taint | kubectl describe pod -n brigade <pending-pod>; check node kubectl describe node | | Script Latency silent | External dependency slow, probe log parse failure | Check last worker logs via brig job logs | | Substrate silent | K8s API server issue, network partition, expired kubeconfig | kubectl cluster-info; check Brigade's kubeconfig secret | | Cron Job silent | Cron gateway crash, Brigade API down, script failure | Check cron gateway pod; brig events list for missing cron events |

What's next

With Brigade monitoring in place:

  • Worker log aggregation — ship Brigade worker pod logs to a centralized log platform and set up alerts on script ERROR output, complementing Vigilmon's uptime perspective
  • Per-project probes — if you have multiple Brigade projects with different SLAs, create separate end-to-end probes and Vigilmon heartbeats per project
  • Slack event notifications — use Vigilmon webhooks to post Brigade status changes directly to the Slack channel where your team triages CI issues
  • Worker resource tuning — use pending pod data from Vigilmon alerts to right-size Brigade worker resource requests, reducing the frequency of pod scheduling failures

Brigade makes event-driven automation on Kubernetes elegant. Vigilmon makes the event path — from gateway ingestion through worker execution — observable, so you know immediately when events stop flowing or workers stop completing.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles