tutorial

How to Monitor Secrets Store CSI Driver with Vigilmon

Secrets Store CSI Driver is a Kubernetes Container Storage Interface driver that lets pods mount secrets from external stores — HashiCorp Vault, AWS Secrets ...

Secrets Store CSI Driver is a Kubernetes Container Storage Interface driver that lets pods mount secrets from external stores — HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager — directly as volumes. It runs as a DaemonSet on every node and uses SecretProviderClass custom resources to define how secrets are fetched and mounted.

But the driver is infrastructure: a DaemonSet that must be healthy on every node, a gRPC interface between the kubelet and the driver, and live connections to external secret backends. When any of these break, pods fail to start and workloads go dark — often with cryptic FailedMount errors that surface only when something tries to restart. In this tutorial you'll set up uptime and response-time monitoring for your Secrets Store CSI Driver deployment using Vigilmon — free tier, no credit card required.

Why Secrets Store CSI Driver needs external monitoring

Internal Kubernetes health checks won't tell you when the CSI driver's dependencies fail:

  • External secret backend unreachable — the driver can't fetch secrets from Vault or AWS Secrets Manager; new pods and pod restarts stall silently with FailedMount
  • Driver DaemonSet has missing pods — a node reboots and the CSI driver pod doesn't come back; all workloads on that node lose secret-mount capability
  • SecretProviderClass misconfiguration — a policy change or rotation in the upstream store breaks the SecretProviderClass spec; mounted secrets go stale without alerting
  • Vault token renewal fails — the driver's Vault token expires; cached mounts may serve stale secrets or break on pod restart
  • Sync-as-Secret controller down — if you use the optional Kubernetes Secret sync feature, a controller failure means secrets drift between the CSI volume and the Kubernetes Secret object

External monitoring from Vigilmon catches these before your engineers start debugging why a deployment rollout is stuck.

What you'll need

  • Secrets Store CSI Driver installed (helm upgrade --install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver)
  • A provider installed (e.g., HashiCorp Vault provider or AWS provider)
  • A free Vigilmon account

Step 1: Expose a health endpoint for the CSI driver

The Secrets Store CSI Driver exposes a health gRPC endpoint on each node at /tmp/csi-secrets-store.sock. For HTTP-based monitoring, deploy a sidecar health exporter or a separate probe service:

# csi-health-probe.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: csi-driver-health-probe
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-driver-health-probe
  template:
    metadata:
      labels:
        app: csi-driver-health-probe
    spec:
      tolerations:
        - operator: Exists
      containers:
        - name: probe
          image: node:20-alpine
          command:
            - sh
            - -c
            - |
              cat > /server.mjs << 'JS'
              import http from 'http';
              import { execSync } from 'child_process';
              http.createServer((req, res) => {
                if (req.url !== '/health') { res.writeHead(404); res.end(); return; }
                try {
                  // Verify driver DaemonSet pod is running on this node
                  execSync('ls /tmp/csi-secrets-store.sock', { timeout: 3000 });
                  res.writeHead(200, { 'Content-Type': 'application/json' });
                  res.end(JSON.stringify({ status: 'ok', driver: 'socket-present' }));
                } catch (e) {
                  res.writeHead(503, { 'Content-Type': 'application/json' });
                  res.end(JSON.stringify({ status: 'error', message: e.message }));
                }
              }).listen(9095, () => console.log('CSI health probe :9095'));
              JS
              node /server.mjs
          ports:
            - containerPort: 9095
          volumeMounts:
            - name: csi-socket-dir
              mountPath: /tmp
      volumes:
        - name: csi-socket-dir
          hostPath:
            path: /tmp
---
apiVersion: v1
kind: Service
metadata:
  name: csi-driver-health
  namespace: kube-system
spec:
  type: ClusterIP
  selector:
    app: csi-driver-health-probe
  ports:
    - port: 9095
      targetPort: 9095

kubectl apply -f csi-health-probe.yaml
# Expose via LoadBalancer or Ingress for external monitoring
kubectl expose service csi-driver-health \
  --name=csi-driver-health-external \
  --type=LoadBalancer \
  --namespace=kube-system
kubectl get svc csi-driver-health-external -n kube-system
# NAME                        TYPE           EXTERNAL-IP     PORT(S)
# csi-driver-health-external  LoadBalancer   203.0.113.40    9095:32100/TCP

Step 2: Monitor the secret backend health endpoint

Add a health check to your secret backend. For a self-hosted HashiCorp Vault:

# Vault exposes a built-in health endpoint
curl https://vault.yourdomain.com/v1/sys/health
# Returns 200 when active, 429 when standby, 501 when uninitialized

# Check the Vault Kubernetes auth method is configured
vault auth list | grep kubernetes

For AWS Secrets Manager, create a canary check Lambda or a sidecar that verifies secret retrieval:

# Verify the CSI provider can reach AWS Secrets Manager
aws secretsmanager describe-secret --secret-id your-canary-secret --region us-east-1

Wrap the backend check in a lightweight HTTP endpoint so Vigilmon can probe it:

# vault-health-sidecar.yaml
- name: vault-health
  image: node:20-alpine
  command:
    - sh
    - -c
    - |
      node -e "
      const http = require('http');
      const https = require('https');
      http.createServer((req, res) => {
        if (req.url !== '/health') { res.writeHead(404); res.end(); return; }
        https.get('https://vault.yourdomain.com/v1/sys/health', r => {
          const ok = r.statusCode === 200 || r.statusCode === 429;
          res.writeHead(ok ? 200 : 503, {'Content-Type':'application/json'});
          res.end(JSON.stringify({ status: ok ? 'ok' : 'degraded', vault: r.statusCode }));
        }).on('error', e => {
          res.writeHead(503, {'Content-Type':'application/json'});
          res.end(JSON.stringify({ status: 'error', message: e.message }));
        });
      }).listen(9096);
      "
  ports:
    - containerPort: 9096

Step 3: Set up HTTP monitoring in Vigilmon

With your endpoints exposed, add them to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | CSI Driver Health (node probe) | http://203.0.113.40:9095/health | 200 | | Vault Backend Health | https://vault.yourdomain.com/v1/sys/health | 200 | | Secrets Store CSI Webhook | https://csi-webhook.yourdomain.com/healthz | 200 |

  1. Set the check interval to 1 minute
  2. Under Expected response, set status code 200 and optionally match "status":"ok" in the response body
  3. Save each monitor

Vigilmon probes from multiple geographic regions. When the Vault backend is unreachable from your cluster's region but shows healthy from your laptop, Vigilmon will surface the discrepancy immediately.

Step 4: Monitor the TCP layer

The CSI driver communicates with the kubelet over a Unix domain socket, and with Vault over HTTPS (typically 8200). Add a TCP monitor for the Vault port:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your Vault host and port (e.g., vault.yourdomain.com / 8200)
  4. Save the monitor

A TCP monitor catches TLS handshake failures and port-level outages that the HTTP health endpoint might report differently depending on Vault's seal state.

Step 5: Configure alert channels

When the CSI driver or its secret backend goes down, every pod restart and new deployment in the cluster will fail to mount secrets. You need immediate notification.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call address
  3. Assign the channel to all CSI driver monitors

Webhook alerts for incident automation

  1. Go to Alert Channels → Add Channel → Webhook
  2. Use a webhook URL that routes to your incident management system (PagerDuty, Opsgenie, Slack)
  3. The payload Vigilmon sends:
{
  "monitor_name": "Vault Backend Health",
  "status": "down",
  "url": "https://vault.yourdomain.com/v1/sys/health",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Wire this into a runbook that pauses all Kubernetes rollouts until the backend is restored — preventing a cascade of FailedMount errors across your cluster.

Step 6: Correlate Vigilmon alerts with CSI driver diagnostics

When you receive a CSI driver or backend downtime alert, run these checks in order:

# 1. Check the CSI driver DaemonSet — all nodes should have a Ready pod
kubectl get pods -n kube-system -l app=secrets-store-csi-driver -o wide

# 2. Look at driver logs for mount failures
kubectl logs -n kube-system -l app=secrets-store-csi-driver --tail=50 \
  | grep -i "error\|fail\|secret"

# 3. Check SecretProviderClass objects for errors
kubectl get secretproviderclass --all-namespaces
kubectl describe secretproviderclass my-provider-class -n production

# 4. Look at SecretProviderClassPodStatus for per-pod mount state
kubectl get secretproviderclasspodstatus --all-namespaces

# 5. Verify Vault connectivity from inside the cluster
kubectl run vault-test --rm -it --image=curlimages/curl -- \
  curl -s https://vault.yourdomain.com/v1/sys/health

# 6. Check the Vault Kubernetes auth configuration
kubectl exec -n kube-system deploy/vault-agent -- \
  vault auth list

If Vigilmon shows the backend down but your Vault pods show as Running, the issue is likely at the network layer between the cluster and Vault — a NetworkPolicy change or a service account binding problem.

Step 7: Create a status page for secrets infrastructure

Secret mount failures are often misreported as application bugs. A status page lets all teams self-diagnose before escalating:

  1. Go to Status Pages → New Status Page
  2. Name it: "Secrets Infrastructure"
  3. Add your monitors: CSI Driver Health, Vault Backend Health
  4. Share the URL with your platform and application teams

When a deployment fails with FailedMount, teams can check the status page first instead of filing an incident ticket.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on CSI driver node probe | Driver DaemonSet gaps, socket failures | | HTTP monitor on Vault/backend health | Backend outages, seal state changes | | TCP monitor on Vault port | TLS failures, port-level outages | | Email + webhook alert channels | Immediate notification before pod restarts fail | | Status page | Team self-service during secret mount incidents |

The goal is to make your secrets infrastructure as observable as the workloads that depend on it. When the Secrets Store CSI Driver or its backend goes dark, you want an alert before developers start debugging FailedMount errors — not after.

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles