Monitoring Stash with Vigilmon: Operator Health, Backup Sessions & Webhook TLS Alerts

Stash by AppsCode is a Kubernetes-native backup solution that protects workloads — Deployments, StatefulSets, DaemonSets, and databases — through a sidecar-based backup model and operator-managed backup sessions. It coordinates backup schedules, repository management, and restore workflows entirely through Kubernetes custom resources. When the Stash operator goes offline, backup sessions stop being created, restore operations cannot proceed, and the webhook server that validates backup configurations begins rejecting new resource submissions. Vigilmon gives you external visibility into every layer of Stash's availability: the operator process, backup session endpoints, webhook server, and TLS certificate health.

What You'll Build

• A monitor on Stash's operator health endpoint to detect controller failures
• A webhook server availability check to catch TLS and admission webhook failures
• A backup session freshness monitor to verify backups are completing on schedule
• SSL certificate monitoring for the webhook server certificate
• An alerting setup that separates operator health from backup job failures

Prerequisites

• Stash 0.28+ deployed in your Kubernetes cluster (typically in the kube-system or stash namespace)
• The Stash operator and webhook server running with their respective Services
• Access to kubectl and cluster admin permissions for initial endpoint verification
• A free account at vigilmon.online

Step 1: Understand Stash's Health and Webhook Endpoints

Stash runs two key services: the operator (which processes BackupSession, RestoreSession, and Repository resources) and the webhook server (which validates and mutates Stash custom resources submitted to the Kubernetes API).

The operator exposes a health check endpoint:

# Port-forward to the Stash operator service
kubectl port-forward -n kube-system svc/stash-operator 8443:443 &

curl -k https://localhost:8443/healthz
# Returns: ok

The webhook server listens on port 443 and responds to Kubernetes admission webhook calls. Its availability can be verified externally if the service is exposed:

curl -k https://stash-webhook.example.com/webhook
# Returns a 400 or 422 for unauthenticated requests — confirming the server is alive

A 400 Bad Request or 422 Unprocessable Entity from the webhook server is the expected response for unauthenticated probes — it proves the server is running and processing requests even though it correctly rejects unsigned admission review payloads.

Step 2: Create a Vigilmon HTTP Monitor for the Operator Health Endpoint

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://stash-operator.example.com/healthz (or your port-forwarded internal URL).
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Click Save.

This monitor catches:

• Stash operator pod crashes or OOM kills from large repository reconciliation workloads
• Kubernetes API server connectivity failures that prevent BackupSession processing
• RBAC misconfiguration after a Helm upgrade removes necessary ClusterRole permissions
• CrashLoopBackOff conditions after version upgrades with incompatible CRD schemas
• Namespace-scoped failures when the operator's service account is deleted or modified

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Stash operator is down, all BackupSessions for all workloads in the cluster stop being created — backup failures accumulate silently across every protected namespace.

Step 3: Monitor the Webhook Server Availability

Stash's webhook server intercepts Kubernetes API calls to validate BackupConfiguration, RestoreSession, and other Stash resources. If the webhook server is unreachable, Kubernetes refuses to create or update any Stash resource — engineers see opaque errors like failed calling webhook when applying backup manifests:

curl -k -o /dev/null -w "%{http_code}" https://stash-webhook.example.com/webhook
# Returns: 400 (expected for unauthenticated probe — server is alive)

1. Add Monitor → HTTP.
2. URL: https://stash-webhook.example.com/webhook.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 400 (the expected rejection code for an unauthenticated probe).
6. Label: Stash webhook server.
7. Click Save.

Configure Vigilmon to treat any non-5xx response as healthy — a 400 from the webhook server is confirmation it's alive. A 502 or 503 from the ingress, or a connection timeout, indicates the webhook server pod is down. When the webhook server is unavailable for more than a few seconds, kubectl apply -f backup.yaml starts failing for all engineers in the cluster.

Step 4: Monitor Backup Session Freshness

Stash creates a BackupSession resource for each scheduled backup run. To verify that backup sessions are completing successfully, configure Stash to send a POST hook to a heartbeat endpoint after each successful session:

# Example Stash BackupConfiguration hook (simplified)
# hooks.postBackup.httpPost.url: https://backup-heartbeat.example.com/stash/heartbeat

curl https://backup-heartbeat.example.com/stash/heartbeat
# Returns: {"last_backup": "2026-07-01T02:00:00Z", "phase": "Succeeded"}

1. Add Monitor → HTTP.
2. URL: https://backup-heartbeat.example.com/stash/heartbeat.
3. Check interval: 6 hours (or match your BackupConfiguration schedule).
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: Succeeded.
7. Label: Stash backup session freshness.
8. Click Save.

Alternatively, expose a simple endpoint from your backup monitoring stack that reads the phase of the most recent BackupSession for your critical workloads and returns its status in the response body.

Step 5: Monitor Repository Integrity via Stash's Audit Endpoint

Stash tracks repository health through its Repository CRD. While direct repository integrity checks require cluster-internal access, you can expose a summary endpoint via a sidecar or monitoring aggregator:

curl https://stash-status.example.com/repository-health
# Returns: {"status": "ok", "repositories": 4, "integrity_checks_passed": 4}

1. Add Monitor → HTTP.
2. URL: https://stash-status.example.com/repository-health.
3. Check interval: 12 hours.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: Stash repository integrity.
8. Click Save.

Step 6: Monitor SSL Certificates for the Webhook Server

Stash's webhook server uses a TLS certificate that Kubernetes admission webhook infrastructure validates. An expired or invalid certificate causes kubectl apply operations on Stash resources to fail across the entire cluster:

openssl s_client -connect stash-webhook.example.com:443 2>/dev/null | openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: stash-webhook.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

Stash generates its own webhook server certificate during installation and stores it in a Kubernetes Secret. This certificate is not managed by cert-manager by default, so it will not auto-renew. Check the expiry date with kubectl get secret -n kube-system stash-apiserver-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates and set Vigilmon's alert threshold to match your certificate validity period.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Operator /healthz | Non-200 or keyword missing | Check pod: kubectl get pods -n kube-system -l app=stash; inspect for OOM or crash | | Webhook server | Connection error or 5xx | Webhook pod down; kubectl apply on Stash resources will fail cluster-wide | | Backup freshness | Missing Succeeded keyword | BackupSession failed; check: kubectl get backupsessions -A for Failed phase | | Repository integrity | Non-200 or ok missing | Repository corruption; run manual stash check for affected repositories | | SSL certificate | < 30 days to expiry | Rotate webhook certificate; update webhook configuration with new CA bundle |

Alert after: 1 consecutive failure for the operator and webhook monitors. 2 consecutive failures for backup freshness and repository integrity.

Common Stash Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Operator pod OOM killed during large restore | /healthz returns connection error; alert within 60 s | | Webhook server certificate expired | SSL monitor alerts; all kubectl apply on Stash resources fail | | BackupConfiguration schedule misconfigured | Operator healthy; backup freshness monitor alerts when heartbeat goes stale | | RBAC change removes operator permissions | Operator alive; BackupSessions stop being created; backup freshness alerts | | Repository backend unreachable (S3 outage) | Operator healthy; BackupSession phase becomes Failed; freshness monitor alerts | | Kubernetes API server restart | Webhook server temporarily unavailable; usually resolves in < 2 min | | Stash version upgrade breaks CRD schema | Operator crashes; /healthz fails; webhook may reject existing resources | | Webhook admission timeout | Kubernetes marks webhook as failed-open or failed-closed; backup configs rejected | | Storage class deleted, PVC binding fails | Operator healthy; RestoreSession never completes; requires application-level monitoring | | Namespace stuck in termination | Operator healthy; backup jobs for that namespace stop; backup freshness alerts |

Stash is the backup control plane for your Kubernetes workloads — when the operator fails, backup sessions stop accumulating, and when the webhook server goes down, every engineer applying Stash configurations hits cryptic admission errors. Vigilmon gives you external visibility into the operator health, webhook availability, backup freshness, repository integrity, and TLS certificate status so you catch Stash failures before they become recovery incidents.

Start monitoring Stash in under 5 minutes — register free at vigilmon.online.