Monitoring Kanister with Vigilmon: Operator Health, ActionSet Status & Blueprint Execution Alerts

Kanister is a Kubernetes-native framework for application-level data management — it orchestrates backup, restore, and disaster recovery workflows through Blueprints and ActionSets, giving platform teams fine-grained control over how stateful applications protect their data. When the Kanister operator goes down, all scheduled backup workflows stop silently: ActionSets queue up in a pending state with no one to execute them, Blueprints stop firing, and the first sign of failure is a missing backup discovered during a recovery attempt. Vigilmon gives you external visibility into Kanister's operator health, controller availability, and backup job completion so your backup infrastructure is monitored the same way your application is.

What You'll Build

• A monitor on Kanister's operator health endpoint to detect controller crashes
• An HTTP check on the controller's availability endpoint
• A blueprint execution verification monitor using Kanister's webhook integration
• SSL certificate monitoring for your Kanister API endpoint if exposed externally
• An alerting setup that distinguishes operator failures from backup job failures

Prerequisites

• Kanister 0.100+ deployed in your Kubernetes cluster (typically in the kanister namespace)
• The Kanister operator service exposed via a Kubernetes Service (ClusterIP or LoadBalancer)
• Access to kubectl for verifying endpoint availability
• A free account at vigilmon.online

Step 1: Understand Kanister's Health Endpoints

The Kanister operator exposes a liveness health endpoint at /healthz on its HTTP port (default 8000). This endpoint returns a 200 OK when the operator process is alive and its internal control loops are running:

# Port-forward to access locally (if not exposed externally)
kubectl port-forward -n kanister svc/kanister-operator 8000:8000 &

curl http://localhost:8000/healthz
# Returns: ok

If the operator is exposed via an Ingress or LoadBalancer:

curl https://kanister.example.com/healthz
# Returns: ok

A 200 with body ok confirms the operator is running and its Kubernetes API watches are active. A connection error or 503 indicates the pod has been OOM killed, the Deployment has scaled to zero, or the Service selector is mismatched after an upgrade.

Kanister also exposes a readiness endpoint at /readyz which additionally verifies that the operator's Kubernetes client has successfully established watches on ActionSet and Blueprint custom resources:

curl http://localhost:8000/readyz
# Returns: ok

Step 2: Create a Vigilmon HTTP Monitor for the Operator Health Endpoint

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://kanister.example.com/healthz (or your internal URL if port-forwarded and proxied).
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Click Save.

This monitor catches:

• Kanister operator pod crashes or unexpected restarts
• OOM kills due to large ActionSet workloads or memory leaks
• Kubernetes API server connectivity issues that prevent the operator from watching resources
• Namespace deletion or RBAC changes that break the operator's service account
• Image pull failures after a version upgrade that leave the operator in a CrashLoopBackOff

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Kanister operator is down, every scheduled backup Blueprint stops executing — backup failures accumulate silently until the next restore attempt reveals missing snapshots.

Step 3: Monitor Controller Availability via the Readiness Endpoint

While /healthz confirms the process is alive, /readyz verifies that Kanister's internal controllers have successfully synced with the Kubernetes API and are ready to process ActionSets:

curl https://kanister.example.com/readyz
# Returns: ok

1. Add Monitor → HTTP.
2. URL: https://kanister.example.com/readyz.
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: Kanister controller readiness.
8. Click Save.

When /readyz fails but /healthz passes, the operator process is alive but its Kubernetes informers have lost their watches — typically caused by a temporary Kubernetes API server restart or a network partition between the Kanister pod and the API server. The operator will usually self-recover within a few minutes, but if it doesn't, kubectl rollout restart deployment kanister-operator -n kanister forces a clean reconnection.

Step 4: Monitor Blueprint Execution via HTTP Alerting Hooks

Kanister supports output hooks that can post execution results to external endpoints. You can configure a Kanister Blueprint to send an HTTP POST to a Vigilmon-compatible webhook or a custom healthcheck endpoint after each successful backup run, enabling backup freshness monitoring.

The pattern is to create a lightweight health endpoint in your infrastructure (for example, a simple HTTP endpoint that returns 200 when a file written by the backup job exists) and monitor it with Vigilmon:

# Example: after a successful Kanister ActionSet, write a heartbeat file
# then expose it via a simple HTTP server
curl https://backup-heartbeat.example.com/last-backup
# Returns JSON: {"timestamp": "2026-07-01T02:00:00Z", "status": "success"}

1. Add Monitor → HTTP.
2. URL: https://backup-heartbeat.example.com/last-backup.
3. Check interval: 6 hours (match your backup schedule).
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: success.
7. Label: Kanister backup freshness.
8. Click Save.

Alternatively, use Vigilmon's Custom Check interval to align with your Blueprint schedule — if backups run nightly, a 24-hour check window with a 1-hour tolerance gives you early warning before the next recovery window.

Step 5: Monitor ActionSet API Availability via Kubernetes API Proxy

If your cluster exposes the Kubernetes API externally (e.g., via a bastion or API gateway), you can check ActionSet availability by querying the custom resource endpoint:

curl -k -H "Authorization: Bearer $TOKEN" \
  https://k8s-api.example.com/apis/cr.kanister.io/v1alpha1/actionsets
# Returns a JSON list of ActionSets

For clusters where the Kubernetes API is not externally accessible, skip this step and rely on the operator health and readiness monitors from Steps 2–3.

Step 6: Monitor SSL Certificates

If your Kanister operator or backup heartbeat endpoint uses HTTPS, certificate monitoring prevents silent HTTPS failures from masking underlying backup problems:

openssl s_client -connect kanister.example.com:443 2>/dev/null | openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: kanister.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

Kanister deployments that use cert-manager for TLS are vulnerable to the same auto-renewal failure modes as any other cert-manager user: Let's Encrypt rate limits, ACME challenge failures due to NetworkPolicy, and annotation drift during Helm upgrades. A 30-day alert window gives you time to investigate cert-manager events before actual expiry.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | /healthz | Non-200 or keyword missing | Check operator pod: kubectl get pods -n kanister; inspect logs for OOM or CrashLoopBackOff | | /readyz | Non-200 or keyword missing | Operator alive but informers disconnected; restart deployment if it doesn't self-recover in 5 min | | Backup freshness | Missing success keyword | ActionSet failed or was not created; check ActionSet status with kubectl get actionsets -n kanister | | SSL certificate | < 30 days to expiry | Renew certificate; check cert-manager events |

Alert after: 1 consecutive failure for operator health. 2 consecutive failures for readiness and backup freshness monitors.

Common Kanister Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Operator pod OOM killed | /healthz returns connection error; alert within 60 s | | Kubernetes API server restart | /readyz temporarily fails; usually self-recovers | | Blueprint misconfiguration | Operator healthy; ActionSet moves to Failed state; backup freshness monitor alerts | | Helm upgrade breaks RBAC | Operator starts but cannot watch ActionSets; /readyz fails | | Backup schedule drift (time zone change) | Backup freshness monitor alerts when expected heartbeat is missed | | Namespace deletion removes operator | Both /healthz and /readyz fail simultaneously | | Storage backend unreachable | Operator healthy; ActionSet fails at execution; backup freshness monitor alerts | | SSL certificate expiry | SSL monitor alerts at 30-day threshold; HTTPS health checks stop resolving | | CrashLoopBackOff after version upgrade | /healthz intermittently fails during restart loops | | ActionSet stuck in running state | Operator healthy; freshness heartbeat never updates; backup freshness monitor alerts |

Kanister is the orchestration layer for your Kubernetes backup strategy — when the operator fails, every application-level backup Blueprint stops executing, and the first indication of the failure is often a missing snapshot discovered during an incident. Vigilmon gives you external visibility into the operator's liveness, readiness, and backup job completion so you know immediately when the backup control plane is degraded, long before the next recovery test reveals missing data.

Start monitoring Kanister in under 5 minutes — register free at vigilmon.online.