Monitoring HashiCorp Vault with Vigilmon

HashiCorp Vault is the backbone of secrets management in many self-hosted and enterprise environments — it issues dynamic credentials, rotates secrets, and controls privileged access across your infrastructure. When Vault goes down or gets stuck in a sealed state, every service that depends on it stops working. Vigilmon gives you continuous visibility into Vault's availability, seal status, cluster health, and storage backends before those failures cascade.

What You'll Set Up

• Vault HTTP API availability monitoring on port 8200
• Seal/unseal status polling with alerting on sealed state
• Active node leader health and standby node status checks
• Secret engine endpoint health monitors
• Dynamic secrets lease renewal service health
• Audit log backend connectivity monitoring
• Token renewal service health checks
• Performance replication status monitoring
• Storage backend connectivity checks (Consul, etcd, or Raft integrated storage)

Prerequisites

• HashiCorp Vault 1.12+ running in server mode (dev mode excluded)
• Vault accessible over HTTP/HTTPS on port 8200 (or your configured listener address)
• A free Vigilmon account

Step 1: Monitor the Vault HTTP API

The Vault HTTP API is the primary surface for all Vault operations. Monitoring its availability is the most critical check you can add.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Vault health endpoint: https://vault.yourdomain.com:8200/v1/sys/health
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Vault's /v1/sys/health endpoint returns different HTTP status codes depending on cluster state:

| Status Code | Meaning | |---|---| | 200 | Initialized, unsealed, active node | | 429 | Unsealed, standby node | | 472 | Disaster recovery mode replication secondary | | 473 | Performance standby | | 501 | Not initialized | | 503 | Sealed |

By default, Vigilmon will alert on any non-200 response. To accept standby nodes as healthy, configure your monitor to accept both 200 and 429. If you run HA Vault, add a separate monitor for each node.

Step 2: Seal/Unseal Status Monitoring

A sealed Vault cannot serve any secrets — it's as if the entire service is down. Vault becomes sealed after a restart, a key shard quorum is lost, or a storage backend failure occurs.

Add a dedicated seal-status monitor:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://vault.yourdomain.com:8200/v1/sys/seal-status
3. Set Expected HTTP status to 200.
4. Enable Response body keyword check and require the response to contain "sealed":false.
5. Set Check interval to 1 minute.
6. Click Save.

The response looks like this for an unsealed Vault:

{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 3,
  "n": 5,
  "progress": 0,
  "nonce": "",
  "version": "1.15.0",
  "migration": false,
  "cluster_name": "vault-cluster",
  "cluster_id": "abc123"
}

If "sealed":true is present in the response body, Vigilmon fires an alert. Pair this with your paging channel — a sealed Vault is an incident requiring immediate operator intervention.

Step 3: Active Node Leader and Standby Node Health

In a Vault HA cluster, the active node handles all write operations while standby nodes forward or redirect requests. Monitor both:

Active node leader check:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://vault-active.yourdomain.com:8200/v1/sys/health
3. Set Expected HTTP status to 200.
4. Label it Vault Active Leader.

Standby node check:

1. Add another HTTP monitor.
2. URL: https://vault-standby-1.yourdomain.com:8200/v1/sys/health
3. Set Expected HTTP status to 429 (standby nodes return 429 by design).
4. Label it Vault Standby Node 1.

You can also monitor the HA status endpoint to verify leader election is healthy:

https://vault.yourdomain.com:8200/v1/sys/leader

This returns the current leader address and whether the node is a performance standby. Vigilmon's body keyword check can verify the "ha_enabled":true field is present.

Step 4: Secret Engine Endpoint Health

Vault's secret engines (KV, PKI, database, AWS, etc.) each have their own mount paths. A misconfigured or disabled engine silently breaks dependent applications.

For a KV v2 secrets engine mounted at secret/:

1. Add an HTTP monitor for https://vault.yourdomain.com:8200/v1/secret/config (returns mount configuration).
2. Set Expected HTTP status to 200.
3. Add a Vault token in the request headers if your Vault requires authentication for this path.

To add a custom header in Vigilmon:

1. Open the monitor settings.
2. Under Custom headers, add:
   • Key: X-Vault-Token
   • Value: A periodic token with read-only policy scoped to the health check path.

For a PKI engine at pki/:

https://vault.yourdomain.com:8200/v1/pki/ca

This endpoint returns the CA certificate without authentication and confirms the PKI engine is mounted and serving.

Step 5: Dynamic Secrets Lease Renewal Service

Dynamic secrets (database credentials, AWS IAM keys) are issued with TTLs and must be renewed before they expire. The lease renewal service must be running for dynamic secrets to stay valid.

Monitor the lease count endpoint:

1. Add an HTTP monitor for https://vault.yourdomain.com:8200/v1/sys/leases/count.
2. Set Expected HTTP status to 200.
3. Add your Vault admin token header.

If the lease service is unhealthy, requests to this endpoint time out or return errors. Combine this with a cron heartbeat from an application that regularly renews a test lease:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 5 minutes.
3. Copy the heartbeat URL and add it to a small renewal health-check script:

#!/bin/bash
# Run every 5 minutes via cron
VAULT_ADDR="https://vault.yourdomain.com:8200"
VAULT_TOKEN="your-health-check-token"

# Renew a known test lease
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "X-Vault-Token: ${VAULT_TOKEN}" \
  "${VAULT_ADDR}/v1/auth/token/renew-self")

if [ "$STATUS" = "200" ]; then
  curl -s https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID
fi

Step 6: Audit Log Backend Connectivity

Vault's audit log backend records every request and response. If all configured audit backends fail, Vault stops serving requests (by default) to prevent unaudited access.

Monitor your audit log backend:

Syslog / file backend: Add a TCP port monitor for the syslog server if you're using a remote syslog backend.

1. Click Add Monitor → TCP Port.
2. Enter your syslog server address and port (typically 514 for UDP syslog or 6514 for TLS syslog).

Vault audit hash check:

https://vault.yourdomain.com:8200/v1/sys/audit

This endpoint (requires admin token) lists enabled audit devices. Monitor its response to verify audit backends remain configured.

Step 7: Token Renewal Service

Vault service tokens must be renewed before their TTLs expire, or dependent services lose access. Add a heartbeat that confirms your token renewal automation is working:

1. In your token renewal service (Vault Agent, Consul Template, or a custom script), add a Vigilmon ping on each successful renewal:

import hvac
import requests

client = hvac.Client(url='https://vault.yourdomain.com:8200', token=VAULT_TOKEN)
result = client.auth.token.renew_self()

if result['auth']['client_token']:
    requests.get('https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID', timeout=5)

2. Set the heartbeat interval to slightly less than your renewal frequency so a missed renewal triggers an alert before the token expires.

Step 8: Performance Replication and Storage Backend

Performance replication status:

For Vault Enterprise with performance replication:

1. Add an HTTP monitor for https://vault.yourdomain.com:8200/v1/sys/replication/status.
2. Use body keyword check to verify "mode":"primary" or "mode":"secondary" as appropriate, and "state":"stream-wals" to confirm replication is active.

Storage backend connectivity:

The storage backend (Consul, etcd, or Raft) is Vault's durable state store — if it fails, Vault seals itself.

Consul backend:

1. Add an HTTP monitor for http://consul.yourdomain.com:8500/v1/health/service/vault.
2. Verify the response contains "Status":"passing".

Raft integrated storage:

https://vault.yourdomain.com:8200/v1/sys/storage/raft/configuration

This endpoint (admin token required) lists Raft peers. Monitor it to verify the Raft cluster is configured and responding.

etcd backend: Add a TCP port monitor on port 2379 (etcd client port) for each etcd peer.

Step 9: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your preferred channel (Slack, PagerDuty, email, or webhook).
2. For seal-status and storage backend monitors, set Consecutive failures before alert to 1 — these are immediate incidents.
3. For standby node and replication monitors, set Consecutive failures before alert to 2 to absorb brief network hiccups.
4. Create a separate high-priority alert channel for seal/unseal events and route it to your on-call rotation.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Vault API health | /v1/sys/health | Server down, sealed state | | Seal status | /v1/sys/seal-status | Vault sealed after restart or key loss | | Active leader | /v1/sys/leader | Leader election failure | | Standby node | Node /v1/sys/health | Standby unavailable for failover | | Secret engine | Engine config endpoint | Disabled or misconfigured engine | | Lease renewal | Cron heartbeat | Renewal service failure | | Audit backend | /v1/sys/audit or syslog TCP | Audit backend loss causing request block | | Token renewal | Cron heartbeat | Token expiry cascade | | Replication status | /v1/sys/replication/status | Replication lag or failure | | Storage backend | Consul health / Raft config / etcd TCP | Vault auto-seal from storage loss |

HashiCorp Vault is a critical path dependency for every service that uses dynamic credentials or centralized secrets. With Vigilmon monitoring seal status, leader health, secret engine availability, and storage backend connectivity, you get the earliest possible warning before a Vault incident silently breaks your infrastructure.