SPIRE (the SPIFFE Runtime Environment) is the production-grade implementation of the SPIFFE standard for workload identity. It issues short-lived X.509 SVIDs (SPIFFE Verifiable Identity Documents) and JWT-SVIDs that replace long-lived secrets in zero-trust architectures. If SPIRE goes down, workloads can no longer authenticate to each other—and depending on your SVID TTL, a silent SPIRE outage can cascade into a service-wide auth failure within minutes.
This tutorial covers how to monitor SPIRE Server health, agent registration sync, SVID rotation rates, and trust bundle distribution using Vigilmon, so you have external assurance that your zero-trust identity plane is operational before your workloads discover it isn't.
Why SPIRE needs external monitoring
SPIRE's failure modes are subtle:
- SPIRE Server crashes silently — systemd or the Kubernetes operator may not restart it cleanly, leaving agents unable to attest new workloads
- Agent loses sync — a SPIRE Agent that cannot reach the Server stops renewing SVIDs; workloads continue to function until their current SVID expires (often 1 hour)
- Trust bundle distribution stalls — if the trust bundle is not propagated to downstream SPIFFE Federation peers, cross-cluster authentication fails
- SVID rotation rate drops — an agent that is alive but failing to rotate SVIDs at the expected rate will eventually cause auth failures silently
- CA rotation breaks renewal — upstream CA rotation can invalidate the signing chain; agents appear healthy but new SVIDs fail verification
Kubernetes liveness probes catch complete crashes. They do not catch degraded-but-alive states like stalled rotation or failed sync.
What you'll need
- A running SPIRE deployment (bare-metal systemd, Docker Compose, or Kubernetes)
spire-serverCLI access- A free Vigilmon account
Step 1: Expose SPIRE Server health
SPIRE Server exposes a gRPC Health Checking Protocol endpoint by default. Wrap it in an HTTP probe for easy monitoring:
# Check server health via spire-server CLI
spire-server healthcheck
# Server is healthy.
For HTTP-based external monitoring, deploy a thin health bridge:
# spire-health-bridge.py
import subprocess, json
from flask import Flask, jsonify
app = Flask(__name__)
def run_spire_cmd(args):
result = subprocess.run(
["spire-server"] + args,
capture_output=True, text=True, timeout=10
)
return result.returncode, result.stdout, result.stderr
@app.get("/health")
def health():
code, stdout, stderr = run_spire_cmd(["healthcheck"])
if code != 0:
return jsonify({"status": "unhealthy", "error": stderr.strip()}), 503
return jsonify({"status": "healthy"}), 200
@app.get("/agents")
def agents():
code, stdout, stderr = run_spire_cmd(["agent", "list", "-output", "json"])
if code != 0:
return jsonify({"status": "error", "error": stderr.strip()}), 503
try:
data = json.loads(stdout)
agents = data.get("agents", [])
return jsonify({"status": "ok", "agent_count": len(agents)}), 200
except Exception as e:
return jsonify({"status": "parse_error", "error": str(e)}), 503
@app.get("/bundle")
def bundle():
code, stdout, stderr = run_spire_cmd(["bundle", "show", "-output", "json"])
if code != 0:
return jsonify({"status": "error", "error": stderr.strip()}), 503
return jsonify({"status": "ok", "bundle_present": True}), 200
if __name__ == "__main__":
app.run(host="0.0.0.0", port=9090)
Run this bridge on the SPIRE Server node (it needs local socket access):
python3 spire-health-bridge.py &
Step 2: Monitor agent registration sync
SPIRE Agents register workloads via selector matching. An agent that has lost sync with the Server will have a stale registration list. Add a sync check to your health bridge:
import time
AGENT_SYNC_WARN_SECONDS = 300 # alert if no agent checked in within 5 minutes
@app.get("/sync")
def sync_check():
code, stdout, stderr = run_spire_cmd(["agent", "list", "-output", "json"])
if code != 0:
return jsonify({"status": "error"}), 503
data = json.loads(stdout)
agents = data.get("agents", [])
now = time.time()
stale = []
for agent in agents:
# banned_at or last_seen_at depending on SPIRE version
last_seen = agent.get("x509svid_expires_at", 0)
if last_seen and (last_seen - now) < AGENT_SYNC_WARN_SECONDS:
stale.append(agent.get("id", "unknown"))
if stale:
return jsonify({
"status": "stale_agents",
"stale": stale,
"count": len(stale)
}), 503
return jsonify({"status": "ok", "agent_count": len(agents)}), 200
Step 3: Track SVID rotation rate with Prometheus metrics
SPIRE Server and Agent both expose Prometheus metrics when configured. Add to your SPIRE Server config:
# server.conf
telemetry {
Prometheus {
host = "0.0.0.0"
port = 9988
}
}
# agent.conf
telemetry {
Prometheus {
host = "0.0.0.0"
port = 9989
}
}
Key metrics to scrape:
# SVIDs issued per minute (should be > 0 continuously)
rate(spire_server_svid_ttl_count[5m])
# Agent bundle refreshes
rate(spire_agent_bundle_refreshes_total[5m])
# Trust bundle age — alert if > TTL
spire_server_bundle_refresh_managed_updates_total
Add a Prometheus rule to expose an aggregate health endpoint:
# spire-recording-rules.yaml
groups:
- name: spire
rules:
- alert: SpireSVIDRotationStalled
expr: rate(spire_server_svid_ttl_count[10m]) == 0
for: 10m
labels:
severity: critical
annotations:
summary: SPIRE SVID rotation has stalled
For Vigilmon, wrap this into an HTTP endpoint:
import requests
PROMETHEUS_URL = "http://localhost:9090"
@app.get("/svid-rotation")
def svid_rotation():
resp = requests.get(
f"{PROMETHEUS_URL}/api/v1/query",
params={"query": "rate(spire_server_svid_ttl_count[5m])"},
timeout=5,
)
data = resp.json()
results = data.get("data", {}).get("result", [])
if not results:
return jsonify({"status": "no_data"}), 503
rate_val = float(results[0]["value"][1])
if rate_val == 0:
return jsonify({"status": "rotation_stalled", "rate": rate_val}), 503
return jsonify({"status": "ok", "rate_per_second": rate_val}), 200
Step 4: Monitor trust bundle distribution
For SPIFFE Federation setups, verify that downstream trust bundles are current:
# Check federation bundle status
spire-server federation list -output json | jq '.federations[].bundle_endpoint_url'
Add a federation health endpoint to the bridge:
@app.get("/federation")
def federation():
code, stdout, stderr = run_spire_cmd(
["federation", "list", "-output", "json"]
)
if code != 0:
return jsonify({"status": "error", "error": stderr}), 503
data = json.loads(stdout)
feds = data.get("federations", [])
# Check that all federation bundles were refreshed recently
failed = [f["trust_domain"] for f in feds if f.get("bundle_endpoint_profile") == ""]
if failed:
return jsonify({"status": "federation_error", "failed": failed}), 503
return jsonify({"status": "ok", "federation_count": len(feds)}), 200
Step 5: Configure Vigilmon monitors
Create the following monitors in Vigilmon:
SPIRE Server health
- Type: HTTP
- URL:
http://spire-server-host:9090/health - Expected: 200
- Interval: 1 minute
Agent sync
- Type: HTTP
- URL:
http://spire-server-host:9090/sync - Expected: 200
- Interval: 2 minutes
SVID rotation
- Type: HTTP
- URL:
http://spire-server-host:9090/svid-rotation - Expected: 200
- Interval: 5 minutes
Trust bundle
- Type: HTTP
- URL:
http://spire-server-host:9090/bundle - Expected: 200
- Interval: 10 minutes
Set alert destinations to PagerDuty or your on-call rotation for SPIRE—an SVID rotation stall is a P1 incident if your SVID TTL is ≤1 hour.
Vigilmon integration summary
| Signal | Monitor type | Alert condition |
|--------|-------------|-----------------|
| SPIRE Server alive | HTTP /health | Non-200 / timeout |
| Agent sync | HTTP /sync | Stale agents |
| SVID rotation rate | HTTP /svid-rotation | Rate == 0 |
| Trust bundle present | HTTP /bundle | Non-200 |
| Federation health | HTTP /federation | Non-200 |
Next steps
- Add Vigilmon webhook → PagerDuty to auto-page the on-call engineer when SVID rotation stalls
- Use Vigilmon's incident history to measure SPIRE availability SLA against your zero-trust posture commitments
- Expose a public status page component showing "Identity Infrastructure" so security and platform teams share a single source of truth