Monitoring Stash by AppsCode with Vigilmon: Operator Health, Backup Heartbeats, Repository Integrity & SSL Certificates

Stash by AppsCode is the Kubernetes-native backup and recovery solution that protects workloads, volumes, and databases through declarative BackupConfiguration and RestoreSession custom resources — integrating with AWS S3, GCS, Azure Blob, and local repositories via Restic under the hood. When the Stash operator fails or backup jobs stop running, your clusters operate without a safety net: silent backup failures are only discovered at restore time, when data loss has already occurred. Vigilmon gives you external visibility into the Stash operator health, backup job completion, repository accessibility, restore readiness, and SSL certificates so you know your backup pipeline is working before you need to rely on it.

What You'll Build

• An HTTP monitor on the Stash operator health endpoint
• Heartbeat monitors verifying backup jobs complete on schedule
• A TCP monitor confirming the backup repository storage endpoint is reachable
• SSL certificate monitoring for the Stash webhook server
• An alerting runbook for Stash failure modes

Prerequisites

• A running Stash operator in your Kubernetes cluster (v0.20+)
• At least one BackupConfiguration with a scheduled backup running
• A free account at vigilmon.online

Step 1: Verify the Stash Operator Health

The Stash operator runs as a deployment in the kube-system or a dedicated namespace and exposes health endpoints on port 8443. Verify it is running:

kubectl get pods -n kube-system -l app.kubernetes.io/name=stash

Expected output:

NAME                     READY   STATUS    RESTARTS   AGE
stash-xxx                2/2     Running   0          5d

The Stash operator also deploys a webhook server that validates backup configurations on admission. Expose the health endpoint for external monitoring:

curl -k https://stash-webhook-svc.kube-system.svc:8443/healthz

For external Vigilmon monitoring, expose this through a Kubernetes Ingress or use a Node-level health probe.

Step 2: Create a Vigilmon HTTP Monitor for the Stash Operator

If you expose the Stash operator metrics or health endpoint via an Ingress:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://stash-health.example.com/healthz.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Label: Stash Operator Health.
7. Click Save.

Alert after 1 consecutive failure — when the Stash operator is down, no new backup or restore jobs will be created for any workload in the cluster.

Step 3: Monitor Backup Completion with Heartbeats

The most critical Stash signal is whether scheduled backups are completing successfully. A backup that fails silently — due to a full disk, expired cloud credentials, or a network partition — leaves your workloads unprotected until the next restore attempt reveals the gap. Set up a Vigilmon heartbeat for each critical backup schedule:

Step 3a — Create a Heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat.
2. Name: Stash PostgreSQL daily backup.
3. Expected interval: 24 hours.
4. Grace period: 2 hours.
5. Copy the ping URL (e.g., https://vigilmon.online/ping/abc123).

Step 3b — Deploy a backup verification CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: stash-backup-heartbeat-pg
  namespace: default
spec:
  schedule: "30 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: default
          containers:
            - name: check
              image: bitnami/kubectl:latest
              command:
                - sh
                - -c
                - |
                  # Find the latest BackupSession for this BackupConfiguration
                  PHASE=$(kubectl get backupsession \
                    -n default \
                    -l stash.appscode.com/backup-configuration=my-postgres-backup \
                    --sort-by=.metadata.creationTimestamp \
                    -o jsonpath='{.items[-1].status.phase}')
                  if [ "$PHASE" = "Succeeded" ]; then
                    curl -fsS -m 10 https://vigilmon.online/ping/abc123
                  fi
          restartPolicy: OnFailure

Replace my-postgres-backup with your BackupConfiguration name. The heartbeat only pings Vigilmon when the latest BackupSession phase is Succeeded — a Failed or Running phase means the backup did not complete cleanly.

Step 4: Monitor Backup Repository Storage Accessibility

Stash stores backup data in remote repositories (S3, GCS, Azure, or NFS). A repository that becomes inaccessible — due to a misconfigured IAM policy, network partition, or storage account suspension — causes all backup jobs to fail. Monitor the storage endpoint directly:

For S3-compatible storage:

1. Add Monitor → TCP.
2. Host: s3.amazonaws.com (or your MinIO/Ceph endpoint hostname).
3. Port: 443.
4. Check interval: 5 minutes.
5. Label: Stash S3 Repository Storage.
6. Click Save.

For GCS:

1. Add Monitor → TCP.
2. Host: storage.googleapis.com.
3. Port: 443.
4. Check interval: 5 minutes.
5. Label: Stash GCS Repository Storage.
6. Click Save.

A TCP failure here means backup writes will fail for all BackupConfiguration resources using that repository.

Step 5: Monitor the Stash Webhook Server SSL Certificate

Stash's admission webhook server uses TLS to validate BackupConfiguration, RestoreSession, and other CRD objects at admission time. If the webhook server's certificate expires, Kubernetes will reject all Stash resource creation and updates — locking out your entire backup configuration:

1. Add Monitor → SSL Certificate.
2. Domain: stash-health.example.com (your Stash webhook or health Ingress domain).
3. Alert when expiry is within: 30 days.
4. Alert again at: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

Step 6: Monitor Restore Readiness with a Canary Restore Heartbeat

Backups are only valuable if they can be restored. A common Stash failure mode is that backups complete successfully but the data is corrupted or the credentials for restoration have rotted. Set up a weekly canary restore heartbeat:

Step 6a — Create a Heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat.
2. Name: Stash canary restore.
3. Expected interval: 7 days.
4. Grace period: 12 hours.
5. Copy the ping URL.

Step 6b — Schedule a weekly restore dry-run:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: stash-restore-canary
  namespace: default
spec:
  schedule: "0 5 * * 0"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: default
          containers:
            - name: check
              image: bitnami/kubectl:latest
              command:
                - sh
                - -c
                - |
                  # Check if most recent RestoreSession succeeded
                  PHASE=$(kubectl get restoresession \
                    -n default \
                    --sort-by=.metadata.creationTimestamp \
                    -o jsonpath='{.items[-1].status.phase}')
                  if [ "$PHASE" = "Succeeded" ]; then
                    curl -fsS -m 10 https://vigilmon.online/ping/<your-heartbeat-id>
                  fi
          restartPolicy: OnFailure

Trigger a test RestoreSession to a disposable namespace weekly and ping Vigilmon only on success. This validates your entire backup-to-restore pipeline end to end.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Stash operator health | Non-200 or timeout | kubectl get pods -n kube-system -l app.kubernetes.io/name=stash | | Backup heartbeat | No ping within window | Check BackupSession: kubectl get backupsession -n default | | Repository storage TCP | Connection refused | Check cloud storage endpoint health and IAM/firewall rules | | Canary restore heartbeat | No ping within 7 days | Check RestoreSession logs; verify repository credentials | | SSL certificate | < 30 days | Renew webhook cert; check cert-manager TLS issuance |

Common Stash Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Stash operator pod crash | HTTP health monitor fires within 60 s | | BackupSession fails silently | Backup heartbeat expires after grace period | | S3 IAM policy revoked | Repository TCP accessible but backup heartbeat stops | | Webhook TLS certificate expired | SSL monitor alerts at 30-day threshold | | Restore credentials rotted | Canary restore heartbeat expires after 7 days | | Storage endpoint network partition | Repository TCP monitor fires | | BackupConfiguration misconfiguration | Operator returns error; backup heartbeat stops |

Stash's declarative backup model gives you policy-driven protection for every workload — but silent failures in backup jobs are the most dangerous kind, discovered only at disaster recovery time when the data you expected to restore does not exist. Vigilmon's layered monitoring of the Stash operator health, backup completion heartbeats, repository accessibility, restore canary, and SSL certificates gives you continuous verification that your backup pipeline is working exactly as configured, every day.

Start monitoring Stash in under 5 minutes — register free at vigilmon.online.