Sealed Secrets Monitoring with Vigilmon

Bitnami Sealed Secrets is a Kubernetes controller and CLI tool that encrypts Kubernetes Secrets using asymmetric cryptography, allowing you to safely commit encrypted secrets to version control. The controller running inside the cluster holds the private key and decrypts SealedSecret resources into standard Secret objects when they are applied.

This architecture makes GitOps workflows safe — your secret values never live in plaintext in your repository. But it also introduces critical infrastructure that must be monitored: if the Sealed Secrets controller goes down or the decryption key becomes unavailable, applications that rely on those secrets will fail to start or lose access to credentials at runtime.

This guide covers how to monitor Sealed Secrets infrastructure using Vigilmon.

Why Sealed Secrets Infrastructure Needs Monitoring

The Sealed Secrets controller is a critical path dependency in Kubernetes clusters using GitOps:

• Controller pod unavailability — if the sealed-secrets controller pod is crashlooping or evicted, no new SealedSecret resources can be decrypted
• Certificate rotation failures — Sealed Secrets automatically rotates its sealing key; if rotation fails or the new cert is not propagated, kubeseal commands start failing across CI/CD pipelines
• Decryption errors at deploy time — a new deployment that includes a SealedSecret will leave the application's Secret missing if the controller cannot decrypt it
• Backup of the sealing private key — if the controller's private key is lost (namespace deleted, accidental key rotation without backup), all sealed secrets become permanently undecryptable
• Controller version drift — old controller versions can fall behind the encryption format used by kubeseal, causing silent failures

Each of these scenarios blocks deployments or causes application outages. Monitoring gives you early warning before a developer pushes a release with broken secrets.

What to Monitor

1. Controller Pod Health via Kubernetes Health Endpoint

The Sealed Secrets controller exposes metrics and a health endpoint. The simplest approach is to expose it via a Service and monitor it directly:

# sealed-secrets-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: sealed-secrets-health
  namespace: kube-system
spec:
  selector:
    app.kubernetes.io/name: sealed-secrets
  ports:
    - port: 8080
      targetPort: 8080
      name: http

Then expose the service externally or via an ingress for Vigilmon to reach:

# ingress for health check (restrict to monitoring IP range)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sealed-secrets-health-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "monitoring-cidr/32"
spec:
  rules:
    - host: sealed-secrets-health.internal.your-domain.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: sealed-secrets-health
                port:
                  number: 8080

In Vigilmon:

1. Monitors → New Monitor
2. Type: HTTP
3. URL: https://sealed-secrets-health.internal.your-domain.com/healthz
4. Interval: 1 minute
5. Expected status: 200

2. Heartbeat from CI/CD kubeseal Validation

The most operationally relevant check is whether kubeseal can actually encrypt a secret using the cluster's current certificate. Add this to your CI pipeline:

#!/bin/bash
# ci-sealed-secrets-health.sh
set -euo pipefail

# Attempt to seal a test secret
echo '{"apiVersion":"v1","kind":"Secret","metadata":{"name":"test","namespace":"default"},"data":{"key":"dmFsdWU="}}' \
  | kubeseal \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    --format=yaml \
    > /tmp/sealed-test.yaml

# Verify output is a valid SealedSecret
if grep -q "SealedSecret" /tmp/sealed-test.yaml; then
  echo "[sealed-secrets] kubeseal test passed"
  curl -s "https://heartbeat.vigilmon.online/$VIGILMON_KUBESEAL_HEARTBEAT_ID"
else
  echo "[sealed-secrets] kubeseal produced unexpected output"
  exit 1
fi

Configure a Heartbeat Monitor in Vigilmon with an expected interval matching your CI run frequency (e.g., every 30 minutes if you run a scheduled pipeline check).

Setting Up Vigilmon Monitors

Monitor 1: Controller HTTP Health Check

1. Log in to Vigilmon → Monitors → New Monitor
2. Type: HTTP
3. Name: Sealed Secrets Controller - Health
4. URL: https://sealed-secrets-health.internal.your-domain.com/healthz
5. Method: GET
6. Expected status: 200
7. Interval: 1 minute
8. Alert after: 2 consecutive failures (avoid noise from transient pod restarts)

Monitor 2: kubeseal CI Heartbeat

1. Type: Heartbeat
2. Name: Sealed Secrets - kubeseal CI validation
3. Expected interval: 30 minutes
4. Grace period: 10 minutes

Add the heartbeat URL to your CI pipeline as the final step in the sealed-secrets validation job.

Monitor 3: Key Rotation Verification Heartbeat

Sealed Secrets rotates the sealing key on a schedule (default: every 30 days). Monitor that rotation succeeds:

#!/bin/bash
# Run this in a scheduled Kubernetes Job after each rotation window
set -euo pipefail

# Fetch current certificate
kubeseal --controller-name=sealed-secrets --controller-namespace=kube-system --fetch-cert > /tmp/current-cert.pem

# Verify cert is valid and not expired within 48 hours
openssl x509 -in /tmp/current-cert.pem -noout -checkend 172800
CERT_STATUS=$?

if [ $CERT_STATUS -eq 0 ]; then
  echo "[rotation] certificate is valid"
  curl -s "https://heartbeat.vigilmon.online/$VIGILMON_ROTATION_HEARTBEAT_ID"
else
  echo "[rotation] certificate expiring within 48 hours — rotation may have failed"
  exit 1
fi

In Vigilmon:

• Type: Heartbeat
• Name: Sealed Secrets - Key Rotation Verification
• Expected interval: 7 days
• Grace period: 24 hours

Monitoring the Private Key Backup

Sealed Secrets' private key is stored as a Kubernetes Secret in kube-system. If this is lost, all encrypted secrets are permanently unrecoverable. Monitor your backup process:

#!/bin/bash
# backup-sealed-secrets-key.sh - run daily
set -euo pipefail

BACKUP_DEST="s3://your-cluster-backups/sealed-secrets/$(date +%Y-%m-%d)/"

# Export the sealing key
kubectl get secret \
  -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -o yaml > /tmp/sealed-secrets-key-backup.yaml

# Encrypt and upload
age -r "$BACKUP_RECIPIENT_KEY" /tmp/sealed-secrets-key-backup.yaml \
  | aws s3 cp - "${BACKUP_DEST}sealed-secrets-key.yaml.age"

# Signal success
curl -s "https://heartbeat.vigilmon.online/$VIGILMON_BACKUP_HEARTBEAT_ID"

# Clean up plaintext
rm -f /tmp/sealed-secrets-key-backup.yaml

In Vigilmon:

• Type: Heartbeat
• Name: Sealed Secrets - Private Key Backup
• Expected interval: 24 hours
• Grace period: 2 hours

Alert Configuration

| Monitor | Type | Interval | Grace | Alert Channel | |---------|------|----------|-------|---------------| | Controller health | HTTP | 1 min | — | PagerDuty + Slack | | kubeseal CI validation | Heartbeat | 30 min | 10 min | Slack | | Key rotation | Heartbeat | 7 days | 24 h | Email + Slack | | Private key backup | Heartbeat | 24 h | 2 h | PagerDuty |

Configure notification channels in Vigilmon under Settings → Notifications. For controller health and key backup failures, route alerts to PagerDuty since these block all deployments that use secrets.

Kubernetes Probe Configuration

Add liveness and readiness probes to the controller deployment if your cluster does not use the default Bitnami Helm chart:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Kubernetes will restart the pod on liveness failures, but Vigilmon's external monitoring catches the window between failure and recovery — and catches cases where the pod is running but decryption is failing for other reasons.

Status Page for Secrets Infrastructure

Create a Vigilmon status page that groups all secrets-related monitors:

1. Status Pages → New Page → name it "Secrets Infrastructure"
2. Add monitors: Sealed Secrets Controller, kubeseal CI, Key Rotation, Key Backup
3. Share with your platform and security teams

This provides a clear, non-cluster view of secrets infrastructure health — useful during incident response when cluster access may be disrupted.

Summary

Sealed Secrets is a simple, elegant solution for GitOps-safe secret management, but the controller and its key material are critical infrastructure. Vigilmon gives you:

• HTTP monitors for controller liveness
• Heartbeat monitors for CI validation, key rotation, and backup jobs
• Alert routing to PagerDuty for deployment-blocking failures
• Status pages for team-wide secrets infrastructure visibility

Get started at vigilmon.online.