Monitoring External Secrets Operator with Vigilmon: Secret Sync Status, Provider Health & Webhook Alerts

External Secrets Operator (ESO) bridges your Kubernetes cluster and external secret stores — AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault, and more. It continuously fetches secrets from these providers and syncs them into Kubernetes Secret objects that your applications consume. When ESO is healthy, the sync is invisible. When it fails, your pods can't start, applications crash on secret reads, and the error surfaces far from the root cause — a developer sees ImagePullBackOff or CrashLoopBackOff rather than "ESO lost connectivity to Vault." Vigilmon monitors ESO from outside the cluster so you know about sync failures within minutes, before they cascade into application incidents.

What You'll Build

• HTTP monitors for ESO's webhook and controller health endpoints
• A heartbeat monitor to verify secret sync is completing successfully
• Provider-specific connectivity checks
• Alert channels that notify your platform and security teams together

Prerequisites

• External Secrets Operator installed in a Kubernetes cluster (v0.9+)
• At least one SecretStore or ClusterSecretStore configured and syncing
• An Ingress exposing ESO health endpoints over HTTPS
• A free account at vigilmon.online

Why ESO Monitoring Requires an External Perspective

ESO failure modes don't always surface as Kubernetes pod failures:

Provider token expiry is silent. If the IAM role, Vault token, or service account credentials ESO uses to authenticate expire, the controller keeps running — it just can't fetch new secret values. Existing Secret objects persist, so currently-running pods are fine. But the next time ESO tries to refresh a rotating secret, it fails silently. You won't notice until a pod restarts and can't find its credentials.

Network partitions between ESO and the provider are invisible to the cluster. ESO pods report Running and Ready even when they can't reach AWS Secrets Manager or Vault. Kubernetes has no way to know the external connectivity has failed.

Webhook downtime blocks ExternalSecret creation. Like Kyverno and Crossplane, ESO registers a validating/mutating webhook for ExternalSecret and SecretStore resources. If the webhook is unreachable, creating or updating ExternalSecret objects fails with admission errors — which means new deployments that depend on secrets can't launch.

Sync interval drift goes undetected. ESO refreshes secrets on a configurable interval (often 1 hour). If the controller is OOM-killed and restarts frequently, the effective refresh interval can stretch far beyond the configured value without triggering any alert.

Step 1: Identify ESO Health Endpoints

ESO exposes health endpoints on its controller and webhook pods:

# Controller health endpoints (port 8081)
kubectl exec -n external-secrets deploy/external-secrets -- \
  wget -qO- http://localhost:8081/healthz

# Webhook health (port 10250)
kubectl exec -n external-secrets deploy/external-secrets-webhook -- \
  wget -qO- http://localhost:10250/healthz

Both return HTTP 200 when healthy. Check the deployments running in your cluster:

kubectl get deployments -n external-secrets
# Typically: external-secrets, external-secrets-webhook, external-secrets-cert-controller

Step 2: Expose Health Endpoints via Ingress

Create Services and an Ingress for external monitoring:

# eso-health-ingress.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: eso-controller-health
  namespace: external-secrets
spec:
  selector:
    app.kubernetes.io/name: external-secrets
    app.kubernetes.io/component: controller
  ports:
    - port: 8081
      targetPort: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: eso-webhook-health
  namespace: external-secrets
spec:
  selector:
    app.kubernetes.io/name: external-secrets
    app.kubernetes.io/component: webhook
  ports:
    - port: 10250
      targetPort: 10250
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: eso-health
  namespace: external-secrets
spec:
  rules:
    - host: eso-health.internal.yourdomain.com
      http:
        paths:
          - path: /controller/healthz
            pathType: Prefix
            backend:
              service:
                name: eso-controller-health
                port:
                  number: 8081
          - path: /webhook/healthz
            pathType: Prefix
            backend:
              service:
                name: eso-webhook-health
                port:
                  number: 10250

Apply and verify:

kubectl apply -f eso-health-ingress.yaml
curl https://eso-health.internal.yourdomain.com/controller/healthz
# Expected: {"status":"ok"}

Step 3: Add Vigilmon HTTP Monitors

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Create the following monitors:

| Monitor Name | URL | Severity | |---|---|---| | ESO controller /healthz | https://eso-health.yourdomain.com/controller/healthz | High | | ESO webhook /healthz | https://eso-health.yourdomain.com/webhook/healthz | Critical | | ESO webhook TLS cert | https://eso-webhook.yourdomain.com | Critical |

Settings for each monitor:

• Check interval: 1 minute
• Timeout: 10 seconds
• Expected status code: 200
• Alert after: 2 consecutive failures

For the webhook TLS cert monitor, enable SSL certificate expiry alerts with a 14-day warning threshold. ESO's webhook TLS certificate is managed by its cert-controller — if that component fails, the certificate eventually expires and the webhook stops accepting connections.

Step 4: Heartbeat Monitor for Secret Sync

The most important thing to verify isn't that ESO's pods are alive — it's that secrets are actually being synced successfully. Create a CronJob that checks ExternalSecret sync status across your namespaces:

# eso-sync-heartbeat.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: eso-sync-heartbeat
  namespace: external-secrets
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: eso-sync-heartbeat
          containers:
            - name: heartbeat
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: eso-heartbeat-url
                - name: CHECK_NAMESPACES
                  value: "production,staging"
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  FAILED=0
                  for NS in $(echo "$CHECK_NAMESPACES" | tr ',' ' '); do
                    # Count ExternalSecrets that are NOT Ready
                    NOT_READY=$(kubectl get externalsecrets -n "$NS" \
                      -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
                      2>/dev/null | grep -v "True" | grep -v "^$" | wc -l)
                    if [ "$NOT_READY" -gt 0 ]; then
                      echo "WARN: $NOT_READY ExternalSecrets not Ready in namespace $NS"
                      kubectl get externalsecrets -n "$NS" \
                        -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.conditions[?(@.type=="Ready")].status}{" "}{.status.conditions[?(@.type=="Ready")].message}{"\n"}{end}' \
                        | grep -v "True"
                      FAILED=$((FAILED + NOT_READY))
                    fi
                  done
                  if [ "$FAILED" -gt 0 ]; then
                    echo "FAIL: $FAILED ExternalSecrets are not syncing correctly"
                    exit 1
                  fi
                  wget -qO- "$HEARTBEAT_URL"
                  echo "ESO sync heartbeat sent — all ExternalSecrets are Ready"

Set up RBAC and the secret:

# Create a ClusterRole with read access to ExternalSecrets
kubectl create clusterrole eso-heartbeat-reader \
  --verb=get,list \
  --resource=externalsecrets.external-secrets.io

kubectl create serviceaccount eso-sync-heartbeat -n external-secrets

kubectl create clusterrolebinding eso-heartbeat-reader \
  --clusterrole=eso-heartbeat-reader \
  --serviceaccount=external-secrets:eso-sync-heartbeat

kubectl create secret generic vigilmon-secrets \
  --from-literal=eso-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  -n external-secrets

In Vigilmon, create the heartbeat monitor:

1. Add Monitor → Heartbeat
2. Name: ESO secret sync check
3. Expected interval: 15 minutes
4. Grace period: 5 minutes

If any ExternalSecret fails to sync — whether due to provider connectivity loss, token expiry, or a misconfigured secret path — the heartbeat won't fire, and Vigilmon opens an incident.

Step 5: Provider Connectivity Checks

ESO supports many secret providers. Add a dedicated connectivity check for your primary provider:

AWS Secrets Manager

# Verify ESO can reach AWS by checking ClusterSecretStore status
kubectl get clustersecretstores -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

Create a separate heartbeat that specifically validates the ClusterSecretStore is Ready:

# In the heartbeat script, add:
STORE_READY=$(kubectl get clustersecretstores \
  -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
  | grep -c "True")
STORE_TOTAL=$(kubectl get clustersecretstores --no-headers | wc -l)
if [ "$STORE_READY" -lt "$STORE_TOTAL" ]; then
  echo "FAIL: Only $STORE_READY/$STORE_TOTAL ClusterSecretStores are Ready"
  exit 1
fi

HashiCorp Vault

Add a separate Vigilmon HTTP monitor directly against Vault's health endpoint (if Vault is self-hosted within your infrastructure):

URL: https://vault.internal.yourdomain.com/v1/sys/health
Expected status: 200
Response body contains: "initialized":true

This monitors Vault independently of ESO — if Vault goes down, you know it before ESO starts failing sync.

Step 6: Configure Alert Channels

Route ESO alerts to the right teams:

Webhook downtime — page on-call (blocks new deployments):

1. Alert Channels → Add Channel → PagerDuty
2. Assign to: ESO webhook /healthz, ESO webhook TLS cert

Sync failures and controller issues — notify platform and security (Slack):

1. Alert Channels → Add Channel → Slack Webhook
2. Paste your #secrets-alerts or #platform-alerts webhook URL
3. Assign to: ESO controller /healthz, ESO secret sync check heartbeat

Complete Monitoring Coverage

| Component | Monitor Type | What It Detects | |---|---|---| | ESO controller | HTTP /healthz | Controller pod crash | | ESO webhook | HTTP /healthz | Webhook pod crash (blocks ExternalSecret creation) | | Webhook TLS cert | SSL expiry | Certificate expiry causes admission failures | | ExternalSecret sync | Heartbeat | Sync failures, provider connectivity loss | | ClusterSecretStore | Heartbeat (embedded) | Provider auth expiry, network issues | | Vault (if self-hosted) | HTTP /v1/sys/health | Upstream secret store availability |

Conclusion

External Secrets Operator is a critical piece of security infrastructure — it's the reason your pods don't need to embed credentials directly. That criticality means its failure modes deserve the same monitoring rigor as your primary API services. With Vigilmon, you get external monitoring that validates not just that ESO's pods are running, but that secrets are actually being synced to the applications that depend on them.

Set up your first ESO monitor free at vigilmon.online. The heartbeat-based sync check gives you meaningful coverage in under 10 minutes of setup.