tutorial

How to Monitor Vault Agent with Vigilmon (Secret Lease Renewal, Auth Connectivity, Cache Health)

Vault Agent runs as a sidecar injecting secrets into your workloads — but lease renewal failures, stale caches, and broken auth are invisible until your app starts receiving expired credentials. Here's how to add external monitoring with Vigilmon.

HashiCorp Vault Agent is the bridge between Vault's secret engine and your workloads. It runs as a sidecar — or as a standalone daemon — and takes care of auto-auth (acquiring a Vault token automatically), secret template rendering (writing secrets into files that your app reads), and caching (serving cached secrets to co-located processes without every request hitting Vault). When it works, it is invisible. When it breaks, the failure modes are subtle: your application sees what looks like a valid secret file, but the contents are stale because the Agent's lease renewal silently failed three hours ago.

Vault Agent has no built-in alerting. It logs to stdout and exposes a limited proxy interface, but nothing fires a page when a lease is twelve hours from expiry, when the auth backend becomes unreachable, or when a template has not been re-rendered in longer than expected. This tutorial walks through building a lightweight health proxy around Vault Agent and wiring it into Vigilmon for external monitoring.

Why Vault Agent needs external monitoring

Vault Agent can fail in ways that are invisible to process monitors, liveness probes, and log scanners unless you know exactly what to look for:

  • Lease renewal silently stops. If the Vault server is briefly unreachable during a renewal window, the Agent retries internally — but if retries exhaust before connectivity restores, the lease expires. Your app continues reading the secret file with no indication the credentials inside it are no longer valid.
  • Auth token expiry. Auto-auth acquires an initial token and re-authenticates when it expires. If the auth backend (Kubernetes, AWS IAM, AppRole) becomes unreachable at renewal time, the Agent loses its token and can no longer fetch new secrets. Existing rendered files stay on disk; nothing signals the problem.
  • Template rendering breaks without crashing. A template that references a secret path the Agent's policy no longer covers will silently fail to render updates. The Agent process stays running; the file on disk contains the last successfully rendered version.
  • Cache serves stale entries after Vault rotation. Vault Agent's in-memory cache does not always invalidate on server-side secret rotation. Applications fetching credentials through the Agent proxy can receive cached values that no longer match Vault's current state.
  • Injector webhook becomes unreachable. In Kubernetes, the Vault Agent Injector runs as a MutatingWebhookConfiguration. If the injector pod is down and the webhook is set to Fail, every pod admission request fails. This breaks deployments cluster-wide, silently, until someone tries to roll out a change.

Each of these failures leaves the Agent process alive, logs noisy but not obviously broken, and process-level health checks green. External monitoring is the only reliable detection path.

What you'll need

  • A running HashiCorp Vault cluster (OSS or Enterprise)
  • Vault Agent deployed as a sidecar or standalone process, with auto-auth and at least one template configured
  • A Vigilmon account — free tier at vigilmon.online
  • Python 3.9+ and pip on the host running the health proxy (or a container image you control)

Step 1: Check Vault Agent health endpoint

Vault Agent runs an API listener that proxies requests to the Vault server. By default it listens on 127.0.0.1:8200 (configurable via the listener stanza in the Agent config). You can use this listener to verify the Agent is alive and able to reach Vault.

First, confirm your Agent config has a listener block:

# vault-agent.hcl
auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    config = {
      role = "my-app-role"
    }
  }

  sink "file" {
    config = {
      path = "/home/vault/.vault-token"
    }
  }
}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = true
}

template {
  source      = "/vault/templates/db-creds.ctmpl"
  destination = "/vault/secrets/db-creds"
}

With that listener running, test reachability:

# Verify the Agent listener is up and proxying correctly
curl -s http://127.0.0.1:8200/v1/sys/health | jq .
# Expected: {"initialized":true,"sealed":false,"standby":false,...}

# Check the Agent's own token validity
curl -s \
  --header "X-Vault-Token: $(cat /home/vault/.vault-token)" \
  http://127.0.0.1:8200/v1/auth/token/lookup-self \
  | jq '{renewable: .data.renewable, ttl: .data.ttl, expire_time: .data.expire_time}'

If the lookup-self call returns a TTL of 0 or an error, the Agent's token has expired. If the sys/health call hangs or returns a non-200 status, the Agent cannot reach Vault.

Wrap these checks in a small shell script for quick manual verification:

#!/bin/bash
# vault-agent-check.sh

AGENT_ADDR="${VAULT_AGENT_ADDR:-http://127.0.0.1:8200}"
TOKEN_FILE="${VAULT_TOKEN_FILE:-/home/vault/.vault-token}"
MIN_TTL_SECONDS="${MIN_TOKEN_TTL:-3600}"

if [ ! -f "$TOKEN_FILE" ]; then
  echo "CRITICAL: token file not found at $TOKEN_FILE"
  exit 2
fi

TOKEN=$(cat "$TOKEN_FILE")

HEALTH=$(curl -sf "${AGENT_ADDR}/v1/sys/health")
if [ $? -ne 0 ]; then
  echo "CRITICAL: Agent listener unreachable at ${AGENT_ADDR}"
  exit 2
fi

TTL=$(curl -sf \
  --header "X-Vault-Token: ${TOKEN}" \
  "${AGENT_ADDR}/v1/auth/token/lookup-self" \
  | jq -r '.data.ttl // 0')

if [ "$TTL" -eq 0 ]; then
  echo "CRITICAL: token TTL is 0 — token has expired"
  exit 2
fi

if [ "$TTL" -lt "$MIN_TTL_SECONDS" ]; then
  echo "WARNING: token TTL is ${TTL}s — below threshold of ${MIN_TTL_SECONDS}s"
  exit 1
fi

echo "OK: Agent reachable, token TTL ${TTL}s"
exit 0

This script is useful in runbooks, but Vigilmon cannot call a shell script directly. The next step wraps this logic in an HTTP endpoint that Vigilmon can probe.

Step 2: Build a health proxy for secret lease monitoring

This FastAPI service exposes three health endpoints that Vigilmon will probe: one for token TTL, one for template render freshness, and one for the Agent cache.

pip install fastapi uvicorn httpx

# vault_agent_health.py
import os
import time
from pathlib import Path

import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

AGENT_ADDR = os.environ.get("VAULT_AGENT_ADDR", "http://127.0.0.1:8200")
TOKEN_FILE = os.environ.get("VAULT_TOKEN_FILE", "/home/vault/.vault-token")
TEMPLATE_PATHS = os.environ.get("VAULT_TEMPLATE_PATHS", "").split(",")
# Alert if a template file has not been modified in this many seconds
TEMPLATE_MAX_AGE_SECONDS = int(os.environ.get("TEMPLATE_MAX_AGE_SECONDS", "7200"))
# Alert if token TTL drops below this many seconds
MIN_TOKEN_TTL_SECONDS = int(os.environ.get("MIN_TOKEN_TTL_SECONDS", "3600"))


def read_token() -> str | None:
    try:
        return Path(TOKEN_FILE).read_text().strip()
    except OSError:
        return None


@app.get("/health/token")
async def token_health():
    """Check that the Agent's auto-auth token is valid and has sufficient TTL."""
    token = read_token()
    if not token:
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "token_file_missing",
            "token_file": TOKEN_FILE,
        })

    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            resp = await client.get(
                f"{AGENT_ADDR}/v1/auth/token/lookup-self",
                headers={"X-Vault-Token": token},
            )
    except httpx.RequestError as exc:
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "agent_unreachable",
            "error": str(exc),
        })

    if resp.status_code != 200:
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "token_lookup_failed",
            "vault_status": resp.status_code,
        })

    data = resp.json().get("data", {})
    ttl = data.get("ttl", 0)
    renewable = data.get("renewable", False)
    expire_time = data.get("expire_time")

    if ttl == 0:
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "token_expired",
            "expire_time": expire_time,
        })

    if ttl < MIN_TOKEN_TTL_SECONDS:
        return JSONResponse(status_code=503, content={
            "status": "warning",
            "reason": "token_ttl_low",
            "ttl_seconds": ttl,
            "threshold_seconds": MIN_TOKEN_TTL_SECONDS,
            "renewable": renewable,
            "expire_time": expire_time,
        })

    return JSONResponse(status_code=200, content={
        "status": "ok",
        "ttl_seconds": ttl,
        "renewable": renewable,
        "expire_time": expire_time,
    })


@app.get("/health/templates")
async def template_health():
    """Check that rendered secret template files are fresh."""
    if not TEMPLATE_PATHS or TEMPLATE_PATHS == [""]:
        return JSONResponse(status_code=200, content={
            "status": "ok",
            "message": "No template paths configured",
        })

    now = time.time()
    stale = []
    missing = []

    for path in TEMPLATE_PATHS:
        path = path.strip()
        if not path:
            continue
        p = Path(path)
        if not p.exists():
            missing.append(path)
            continue
        age = now - p.stat().st_mtime
        if age > TEMPLATE_MAX_AGE_SECONDS:
            stale.append({"path": path, "age_seconds": int(age)})

    if missing:
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "template_files_missing",
            "missing": missing,
        })

    if stale:
        return JSONResponse(status_code=503, content={
            "status": "warning",
            "reason": "template_files_stale",
            "stale": stale,
            "max_age_seconds": TEMPLATE_MAX_AGE_SECONDS,
        })

    return JSONResponse(status_code=200, content={
        "status": "ok",
        "templates_checked": len(TEMPLATE_PATHS),
    })


@app.get("/health/cache")
async def cache_health():
    """Check that the Vault Agent cache endpoint is responding."""
    token = read_token()
    if not token:
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "token_file_missing",
        })

    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            # The Agent cache proxies /v1/sys/health as a lightweight connectivity test
            resp = await client.get(
                f"{AGENT_ADDR}/v1/sys/health",
                headers={"X-Vault-Token": token},
            )
    except httpx.RequestError as exc:
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "cache_proxy_unreachable",
            "error": str(exc),
        })

    if resp.status_code not in (200, 429, 473, 501, 503):
        # Vault's /sys/health returns non-200 for standby/sealed states,
        # but all are valid responses from the proxy — a connection error is the real failure.
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "unexpected_vault_status",
            "vault_status": resp.status_code,
        })

    vault_data = resp.json()
    return JSONResponse(status_code=200, content={
        "status": "ok",
        "vault_initialized": vault_data.get("initialized"),
        "vault_sealed": vault_data.get("sealed"),
        "vault_standby": vault_data.get("standby"),
    })

Run the health proxy:

VAULT_AGENT_ADDR=http://127.0.0.1:8200 \
VAULT_TOKEN_FILE=/home/vault/.vault-token \
VAULT_TEMPLATE_PATHS=/vault/secrets/db-creds,/vault/secrets/api-key \
TEMPLATE_MAX_AGE_SECONDS=7200 \
MIN_TOKEN_TTL_SECONDS=3600 \
uvicorn vault_agent_health:app --host 0.0.0.0 --port 8210

Verify the endpoints respond correctly before wiring them into Vigilmon:

curl -s http://localhost:8210/health/token | jq .
# {"status":"ok","ttl_seconds":86234,"renewable":true,"expire_time":"2026-07-03T10:22:00Z"}

curl -s http://localhost:8210/health/templates | jq .
# {"status":"ok","templates_checked":2}

curl -s http://localhost:8210/health/cache | jq .
# {"status":"ok","vault_initialized":true,"vault_sealed":false,"vault_standby":false}

If you are deploying Vault Agent as a Kubernetes sidecar, expose the proxy as a container port and add it as a liveness probe target alongside your Vigilmon monitor:

# vault-agent-sidecar patch
containers:
  - name: vault-agent-health
    image: your-registry/vault-agent-health:latest
    ports:
      - containerPort: 8210
    env:
      - name: VAULT_AGENT_ADDR
        value: "http://127.0.0.1:8200"
      - name: VAULT_TOKEN_FILE
        value: "/home/vault/.vault-token"
      - name: VAULT_TEMPLATE_PATHS
        value: "/vault/secrets/db-creds,/vault/secrets/api-key"
    livenessProbe:
      httpGet:
        path: /health/token
        port: 8210
      initialDelaySeconds: 15
      periodSeconds: 30

Step 3: Monitor the Vault cluster reachability via Agent

Beyond token and template health, you want to know whether the Agent can actively reach the Vault cluster. A network partition or Vault maintenance window that the Agent cannot recover from will stop all lease renewals.

Test Agent-to-Vault connectivity with a direct curl through the Agent listener:

# Test Agent can proxy a simple Vault read (adjust the path to match your setup)
curl -sf \
  --header "X-Vault-Token: $(cat /home/vault/.vault-token)" \
  http://127.0.0.1:8200/v1/sys/seal-status \
  | jq '{sealed: .sealed, cluster_name: .cluster_name}'

Add a dedicated connectivity endpoint to the health proxy:

@app.get("/health/vault-reachability")
async def vault_reachability():
    """Test that the Agent can actively reach the Vault cluster."""
    token = read_token()
    headers = {"X-Vault-Token": token} if token else {}

    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            resp = await client.get(
                f"{AGENT_ADDR}/v1/sys/seal-status",
                headers=headers,
            )
    except httpx.RequestError as exc:
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "vault_unreachable_via_agent",
            "error": str(exc),
        })

    body = resp.json()
    if body.get("sealed"):
        return JSONResponse(status_code=503, content={
            "status": "critical",
            "reason": "vault_sealed",
            "cluster_name": body.get("cluster_name"),
        })

    return JSONResponse(status_code=200, content={
        "status": "ok",
        "sealed": body.get("sealed"),
        "cluster_name": body.get("cluster_name"),
        "version": body.get("version"),
    })

Test this endpoint manually before adding it to Vigilmon:

curl -s http://localhost:8210/health/vault-reachability | jq .
# {"status":"ok","sealed":false,"cluster_name":"vault-cluster-prod","version":"1.17.0"}

This is the definitive test: if this endpoint returns 503, your Agent cannot reach Vault — no renewal or new secret fetch will succeed regardless of what process monitors say.

Step 4: Add Vigilmon monitors

With the health proxy running and all four endpoints returning 200, set up one Vigilmon monitor per endpoint.

Token health monitor:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to https://your-host.example.com:8210/health/token
  4. Set the check interval to 30 seconds
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Name it [vault-agent] token TTL
  7. Save the monitor

Template freshness monitor:

Repeat the process for /health/templates:

  • URL: https://your-host.example.com:8210/health/templates
  • Interval: 30 seconds
  • Name: [vault-agent] template render

Cache health monitor:

  • URL: https://your-host.example.com:8210/health/cache
  • Interval: 30 seconds
  • Name: [vault-agent] cache proxy

Vault reachability monitor:

  • URL: https://your-host.example.com:8210/health/vault-reachability
  • Interval: 30 seconds
  • Name: [vault-agent] vault reachability

A 30-second interval gives you detection within one probe cycle — fast enough to catch a renewal failure before the lease actually expires, assuming your minimum TTL threshold is set to at least a few hours.

If the health proxy is not publicly reachable (e.g., it is behind a firewall or on a private network), use an SSH tunnel or place the proxy on a port accessible via your internal reverse proxy, then configure TLS on it before pointing Vigilmon at the public URL.

Step 5: Set up alerts

In Vigilmon, configure alert channels before or after adding monitors. Go to Alert Channels → New Channel and choose your delivery method — email, Slack webhook, PagerDuty, or generic webhook.

For Vault Agent, the failure hierarchy maps directly to on-call urgency:

Vault sealed or reachability lost — wake the on-call engineer immediately. No secret can be fetched or renewed while Vault is sealed or partitioned from the Agent. Assign this monitor to your PagerDuty channel or highest-priority Slack channel.

Token TTL below threshold — this is a warning with a lead time measured in hours (if you have set MIN_TOKEN_TTL_SECONDS to 3600 or higher). Assign to a Slack channel. If the token is renewable and the Agent is healthy, it should renew on its own — but you want to know if it does not.

Template files stale or missing — applications reading secret files may be using expired credentials. Assign to Slack and email. Treat it as critical if the template has not rendered in more than twice the expected re-render interval.

Cache proxy down — the Agent listener has stopped responding. Applications making API calls through the proxy will receive connection errors. Assign to PagerDuty.

To set up a Slack webhook channel:

  1. Go to Alert Channels → New Channel → Webhook
  2. Paste your Slack Incoming Webhook URL
  3. Assign the channel to all four Vault Agent monitors
  4. Under each monitor, set Alert on first failure (do not wait for consecutive failures — lease windows are tight)

Recommended alert thresholds

| Metric | Warning threshold | Critical threshold | |---|---|---| | Token TTL | Less than 4 hours | Less than 1 hour | | Template file age | Greater than 2x expected render interval | File missing from disk | | /health/vault-reachability response code | N/A | Non-200 | | /health/cache response code | N/A | Non-200 | | Health proxy response time | Greater than 2000ms | Greater than 5000ms (proxy itself is overloaded) | | Vault sealed status | N/A | "sealed":true in response body |

Adjust the TTL thresholds based on your lease duration. If your Vault policy issues tokens with a max TTL of 24 hours, a warning at 4 hours gives you enough time to investigate and intervene before the token expires. If your leases are short-lived (1 hour), tighten the thresholds accordingly and rely on Vault's renewability — but verify that auto-renewal is actually working by watching the token TTL trend across successive probe responses in Vigilmon's response history.

Conclusion

Vault Agent is a well-engineered piece of infrastructure, but its failure modes are designed to be transparent to the workloads it serves — which means they are also transparent to your monitoring tooling unless you deliberately expose them. The health proxy built in this tutorial surfaces the four signals that matter: token validity, template freshness, cache availability, and Vault cluster reachability. Wiring those signals into Vigilmon gives you external, multi-region detection with sub-minute alerting — so a silent lease renewal failure becomes a paged incident before it becomes a production outage.

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles