tutorial

Monitoring OpenBao with Vigilmon: Seal Status, Auth Method Health, Secret Engine Availability & HA Cluster Monitoring

How to monitor OpenBao with Vigilmon — seal status checks, auth method health validation, secret engine availability, token TTL monitoring, HA cluster health, and automated alerting for secrets infrastructure.

OpenBao is the Linux Foundation open-source fork of HashiCorp Vault, providing enterprise-grade secrets management, PKI, dynamic credentials, and encryption-as-a-service. When OpenBao's active node seals unexpectedly — due to a hardware failure, lost Shamir key shares, or misconfigured auto-unseal — every application reading secrets from it fails with a 503 until an operator manually unseals it. When a standby HA node loses its connection to the storage backend, it cannot promote itself during a failover and the cluster is left without a functional active node. When a PKI secret engine's CA certificate nears expiry, every certificate issued from it is about to become untrusted across your infrastructure. Vigilmon gives you external visibility into OpenBao's seal status, API health, and HA cluster endpoints, so you catch seal events and cluster degradation before applications start failing.

What You'll Build

  • A seal status monitor that fires the moment OpenBao becomes sealed
  • An API health monitor confirming the OpenBao server is responding to requests
  • HA cluster leader check to validate a standby-promotion path exists
  • Secret engine and auth method availability probes
  • SSL certificate monitoring for OpenBao's API and PKI-issued certificates
  • An alerting runbook mapping OpenBao failure modes to Vigilmon monitor states

Prerequisites

  • OpenBao 2.x installed (single node or HA cluster with Raft or Consul storage)
  • OpenBao auto-unseal configured (AWS KMS, Azure Key Vault, etc.) or documented manual unseal procedure
  • Prometheus configured to scrape OpenBao's /v1/sys/metrics endpoint
  • OpenBao's API accessible over HTTPS (standard port 8200)
  • A free account at vigilmon.online

Step 1: Understand OpenBao's Health Surface

OpenBao exposes a /v1/sys/health endpoint that returns machine-readable status without requiring authentication. This is the primary external health surface:

# Check OpenBao health — returns different HTTP codes by state
curl -s https://openbao.example.com:8200/v1/sys/health | jq .

# HTTP 200: initialized, unsealed, active
# HTTP 429: unsealed, standby (HA mode)
# HTTP 472: disaster recovery secondary
# HTTP 501: not initialized
# HTTP 503: sealed or uninitialized

# Seal status (requires no authentication)
curl -s https://openbao.example.com:8200/v1/sys/seal-status | jq '{sealed, t, n, progress}'
# sealed: false = healthy; sealed: true = immediate incident

For HA clusters, check leader status:

# Leader endpoint — confirms which node is active
curl -s https://openbao.example.com:8200/v1/sys/leader | jq '{ha_enabled, is_self, leader_address, performance_standby}'

Key Prometheus metrics:

# Scrape OpenBao metrics (requires metrics policy token or unauthenticated access configured)
curl -s -H "X-Vault-Token: $OPENBAO_TOKEN" \
  https://openbao.example.com:8200/v1/sys/metrics?format=prometheus | grep openbao_

# Key metrics
openbao_core_unsealed           # 1 = unsealed, 0 = sealed
openbao_core_ha_cluster_leader  # 1 = this node is the HA leader
openbao_expire_num_leases       # Total active leases (watch for runaway lease growth)
openbao_token_count             # Active token count

Step 2: Monitor Seal Status — The Critical Check

A sealed OpenBao instance returns HTTP 503 from /v1/sys/health. This is the most critical monitor to configure because a sealed server means zero secrets are accessible:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://openbao.example.com:8200/v1/sys/health.
  3. Check interval: 30 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "sealed":false.
  7. Label: OpenBao seal status (active node).
  8. Click Save.

HA standby nodes return HTTP 429, not 200. For a standby-aware check that accepts both active and standby states:

  1. Add Monitor → HTTP.
  2. URL: https://openbao-standby.example.com:8200/v1/sys/health?standbyok=true.
  3. Expected status: 200.
  4. Keyword: "sealed":false.
  5. Label: OpenBao seal status (standby).
  6. Click Save.

The ?standbyok=true parameter makes the health endpoint return 200 for both active and standby nodes, allowing a single keyword check to validate that the node is unsealed regardless of its HA role.

Alert after: 1 consecutive failure. A sealed OpenBao means all application secret reads are failing right now.

Step 3: Monitor the HA Leader Endpoint

In an HA cluster, a valid leader must exist at all times. Monitor the leader endpoint to detect split-brain or leaderless states:

  1. Add Monitor → HTTP.
  2. URL: https://openbao.example.com:8200/v1/sys/leader.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "ha_enabled":true.
  7. Label: OpenBao HA leader check.
  8. Click Save.

When the cluster has no leader — during a network partition or after a failed failover — this endpoint still returns 200 but with leader_address empty. Add a secondary keyword check on a non-empty leader_address field if your monitoring supports multiple keyword assertions.

Step 4: Monitor Auth Method Health

OpenBao's auth methods (AppRole, Kubernetes, LDAP, JWT/OIDC) are mounted at /v1/auth/<method>/ paths. Validate that the most critical auth methods are responding:

# Check Kubernetes auth method mount health (no auth required for the sys/mounts listing)
curl -s -H "X-Vault-Token: $OPENBAO_TOKEN" \
  https://openbao.example.com:8200/v1/sys/auth | jq 'keys'

# Verify a specific auth method path is accessible
curl -s -o /dev/null -w "%{http_code}" \
  https://openbao.example.com:8200/v1/auth/kubernetes/config
# Returns 200 (configured) or 404 (not configured)

For external monitoring with Vigilmon, the /v1/sys/mounts endpoint is a reliable proxy for auth method availability:

  1. Add Monitor → HTTP.
  2. URL: https://openbao.example.com:8200/v1/sys/mounts.
  3. Check interval: 5 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "kubernetes/" (or your primary auth method path).
  7. Label: OpenBao auth method availability.
  8. Click Save.

This endpoint requires a valid token. Create a dedicated monitoring policy and AppRole with read-only sys/mounts access to avoid using a privileged token for health checks.

Step 5: Monitor Secret Engine Availability and PKI Certificate Expiry

Secret engines are the core of OpenBao's value. Monitor the KV and PKI engines specifically:

# List all secret engine mounts
curl -s -H "X-Vault-Token: $OPENBAO_TOKEN" \
  https://openbao.example.com:8200/v1/sys/mounts | jq 'to_entries[] | {path: .key, type: .value.type}'

# PKI engine — check CA certificate expiry
curl -s -H "X-Vault-Token: $OPENBAO_TOKEN" \
  https://openbao.example.com:8200/v1/pki/cert/ca | jq '.data.expiration'
# Returns Unix timestamp — compare against current time for days-to-expiry

For Prometheus-based PKI expiry monitoring:

# prometheus-openbao-alerts.yaml
groups:
  - name: openbao_security
    rules:
      - alert: OpenBaoPKICACertExpiringSoon
        expr: (openbao_secret_pki_ca_cert_expiry_seconds - time()) / 86400 < 30
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "OpenBao PKI CA certificate expiring in less than 30 days"
          description: "PKI mount {{ $labels.mount }} CA certificate expires in {{ $value | humanizeDuration }}."

      - alert: OpenBaoHighLeaseCount
        expr: openbao_expire_num_leases > 50000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "OpenBao lease count high"
          description: "OpenBao has {{ $value }} active leases — check for dynamic secret rotation issues."

      - alert: OpenBaoTokenCountHigh
        expr: openbao_token_count > 10000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "OpenBao token count elevated"
          description: "OpenBao has {{ $value }} active tokens — check for token TTL configuration drift."

Step 6: Monitor Token TTL Configuration

OpenBao token TTLs control how long application credentials are valid. Monitoring for tokens near expiry at the system level requires Prometheus metrics. For application-level token health, create a Vigilmon monitor against each application's health endpoint that internally validates its own token:

# Application health endpoint should validate its own OpenBao token
# GET /health should return 200 only if the secret backend is accessible
curl https://myapp.example.com/health
# A 503 here may mean the app's OpenBao token expired
  1. Add Monitor → HTTP.
  2. URL: https://myapp.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Label: myapp (OpenBao-backed health check).
  7. Click Save.

Correlate alerts from this monitor with the OpenBao seal status monitor. If OpenBao is healthy but the app health check fails, the application's token may have expired or its AppRole login may be failing.

Step 7: SSL Certificate Monitoring for the OpenBao API and PKI

OpenBao's API TLS certificate and any PKI-issued certificates for internal services should both be monitored:

  1. Add Monitor → SSL Certificate.
  2. Domain: openbao.example.com.
  3. Port: 8200.
  4. Alert when expiry is within: 30 days.
  5. Alert again: 14 days, 7 days, 3 days.
  6. Click Save.

Also monitor any intermediate CA-issued certificates that PKI has signed for internal services — these often have 1-year TTLs and expire before operators notice.

Step 8: Configure Alerting for Seal and Cluster Events

In Vigilmon under Settings → Notifications, configure alert channels and map them to the correct response runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | Seal status /v1/sys/health | HTTP 503 or "sealed":true keyword | Execute unseal procedure immediately; check auto-unseal KMS connectivity; page on-call | | HA leader endpoint | Non-200 or empty leader_address | Check Raft/Consul quorum; verify network connectivity between HA nodes; check storage backend | | Auth method availability | Non-200 or keyword missing | Specific auth method may be disabled or misconfigured; check mount audit logs | | Application health (OpenBao-backed) | Non-200 | Correlate with seal status; may indicate expired app token | | SSL certificate | < 30 days to expiry | Renew OpenBao API TLS certificate; all API clients will reject expired cert |

Paging rule: The seal status monitor should trigger your highest-priority alert channel (PagerDuty, SMS, phone call). A sealed OpenBao means no application can read secrets — every minute of seal time is downtime across your entire infrastructure.

Common OpenBao Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Node sealed due to auto-unseal KMS unreachable | Seal status monitor fires (HTTP 503) within 30 seconds | | Active HA node crashes, no standby promotes | HA leader monitor fires on empty leader_address | | Raft quorum lost (2 of 3 nodes down) | All health endpoints return 503; seal status monitor fires | | PKI CA certificate expired | SSL monitors fire on all PKI-issued certs; Prometheus PKI alert fires | | Storage backend (Consul/etcd) becomes read-only | Writes fail; reads still work; health endpoint shows degraded state | | AppRole secret ID TTL expires | Application health monitor fires (app fails to login); OpenBao itself is healthy | | Token TTL too short, apps see frequent 403s | Application health monitor fires; correlate with OpenBao audit logs | | Lease renewal flood triggers rate limiting | High lease count Prometheus alert fires; applications may see throttled responses | | HA standby can't connect to storage | Standby health endpoint returns 503; standby-specific seal status monitor fires | | API TLS cert expired | SSL monitor fires 30 days before; all API clients reject connection on expiry |

OpenBao is the keystone of your secrets infrastructure — when it seals, every application in your stack that reads credentials, certificates, or dynamic secrets immediately begins failing. External monitoring is the only way to detect a seal event before your alerting stack itself goes dark (since your alerting credentials may come from OpenBao). Vigilmon gives you an out-of-band view: seal status checks, HA leader validation, auth method availability probes, and SSL certificate tracking that operates completely independently of your internal infrastructure. When OpenBao can't tell you something is wrong, Vigilmon can.

Start monitoring your OpenBao secrets infrastructure in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles