How to Monitor k8up with Vigilmon (HTTP + TCP + Alerts)

k8up is VSHN's open-source Kubernetes backup operator built on Restic. It integrates directly with Kubernetes via custom resource definitions — Schedule, Backup, Restore, Archive, and PreBackupPod CRDs in the k8up.io API group — and manages Restic backup jobs that run as one-shot pods targeting S3-compatible storage. The k8up operator pod runs persistently in the k8up-system namespace while backup jobs spin up on demand, execute, and exit. When either layer fails silently, you lose backup coverage with no native Kubernetes alert.

In this tutorial you'll set up comprehensive uptime monitoring for k8up — operator health and S3 backend reachability — using Vigilmon — free tier, no credit card required.

Why k8up needs external monitoring

k8up separates the persistent operator from short-lived backup job pods. Kubernetes health checks only tell you a container is running, not that backups are actually completing:

• Operator pod crash leaves all scheduled backups silently skipped — the Schedule CRD still exists, kubectl shows nothing wrong, but no backup jobs are created because the controller is gone
• S3 target unreachable: backup pods start, fail, no Kubernetes alert to namespace owner — job pods run to completion with a non-zero exit code; the failure stays in job logs that nobody reads unless explicitly monitoring the S3 endpoint
• PreBackupPod fails: app-consistent backup silently degrades to crash-consistent — the PreBackupPod CRD is designed to quiesce an application before backup; if it errors out, k8up falls back to a crash-consistent snapshot with no alert
• RBAC change removes operator's PVC access: backup jobs fail with permission denied — a ServiceAccount or ClusterRole update can strip the operator's read access to PVCs; jobs fail immediately but the Schedule still shows as active
• Restic repository lock stuck: backups queue but never run until manually unlocked — a previous backup job that died mid-run leaves a Restic lock on the repository; subsequent jobs block waiting for the lock and never complete

External monitoring from multiple geographic regions catches failures spanning the operator layer, S3 backend, and network path simultaneously.

What you'll need

• A Kubernetes cluster with k8up installed (via Helm from https://charts.k8up.io)
• S3-compatible object storage for Restic repositories (AWS S3, MinIO, Exoscale, etc.)
• A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose k8up operator health endpoints

The k8up operator exposes Prometheus metrics and a /healthz liveness endpoint on port 8080. Expose it outside the cluster via a NodePort service so Vigilmon can reach it.

Create a NodePort service for the k8up operator

# k8up-operator-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: k8up-operator-health
  namespace: k8up-system
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: k8up
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080
      nodePort: 30880
      protocol: TCP

Apply it:

kubectl apply -f k8up-operator-health-svc.yaml

Verify the operator is running and the health endpoint is reachable:

# Confirm the operator pod is running
kubectl get pods -n k8up-system

# Test the health endpoint
curl http://<node-ip>:30880/healthz
# ok

# Check Prometheus metrics are exposed
curl -s http://<node-ip>:30880/metrics | head -20

The /healthz endpoint returns ok when the operator is functioning. If the pod crashes or becomes unresponsive, this endpoint stops responding — giving Vigilmon a clear signal that backup scheduling is broken.

Step 2: Set up HTTP monitoring in Vigilmon

With the health endpoint exposed, add the k8up operator to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS as the type
3. Set the URL to http://<node-ip>:30880/healthz
4. Set the check interval to 1 minute
5. Under Expected response, set:
   • Status code: 200
   • Response body contains: ok
6. Save the monitor

The multi-region consensus advantage

Vigilmon probes your endpoints from multiple geographic regions simultaneously. Because k8up operator health depends on both the Kubernetes control plane and the network path to your cluster, a single probe failure could reflect a transient routing issue rather than a real operator crash. Vigilmon's multi-region consensus requires multiple independent probes to agree before triggering an alert — accurate detection without false positives.

Step 3: Monitor your TCP layer

TCP monitoring catches infrastructure failures before they manifest as HTTP errors or backup job failures. Add TCP monitors for the operator port, your S3 storage endpoint, and any NFS mounts used as PVC sources.

In Vigilmon:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your node IP and port 30880 (k8up operator)
4. Save the monitor
5. Add a TCP monitor for <s3-endpoint> on port 443 (S3 storage HTTPS)
6. Add a TCP monitor for <nfs-server-ip> on port 2049 (NFS mount, if applicable)

Confirm each endpoint is reachable from your cluster:

# Test k8up operator TCP port
nc -zv <node-ip> 30880
# Connection to <node-ip> 30880 port [tcp/*] succeeded!

# Test S3 endpoint reachability (AWS S3, MinIO, Exoscale, etc.)
nc -zv <s3-endpoint> 443
# Connection to <s3-endpoint> 443 port [tcp/https] succeeded!

# Test NFS server port (if PVCs are backed by NFS)
nc -zv <nfs-server-ip> 2049
# Connection to <nfs-server-ip> 2049 port [tcp/nfs] succeeded!

The S3 TCP monitor on port 443 is your earliest warning that backup targets are unreachable — it fires before any Restic job even starts, giving you time to investigate before a backup window is missed.

Step 4: Configure alert channels

A crashed k8up operator means every scheduled backup silently stops running. An unreachable S3 endpoint means every backup job fails. You need to know immediately.

Email alerts

1. In Vigilmon, go to Alert Channels → Add Channel → Email
2. Enter your on-call or platform-team email address
3. Assign the channel to your k8up operator HTTP monitor, operator TCP monitor, S3 TCP monitor, and NFS TCP monitor

Webhook alerts (Slack, PagerDuty, etc.)

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your webhook URL
3. Assign the channel to your monitors

Example payload Vigilmon sends when a monitor goes down:

{
  "monitor_name": "k8up operator /healthz",
  "status": "down",
  "url": "http://203.0.113.50:30880/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 60
}

When you receive a k8up alert, correlate it with the cluster using k8up-specific kubectl commands:

# Check the k8up operator pod status
kubectl get pods -n k8up-system

# Check Schedule CRD status across all namespaces
kubectl get schedules.k8up.io -A

# Check recent Backup job objects and their status
kubectl get backups.k8up.io -A

# Inspect a specific backup that failed
kubectl describe backup.k8up.io <backup-name> -n <namespace>

# Check Backup job pods — look for Error or OOMKilled exit reasons
kubectl get pods -A -l k8up.io/owned-by=k8up --field-selector=status.phase=Failed

# Check for stuck Restic repository locks (shows in operator logs)
kubectl logs -n k8up-system deploy/k8up --tail=100 | grep -i lock

# List Restore objects across all namespaces
kubectl get restores.k8up.io -A

Step 5: Create a public status page

k8up backs up workloads owned by multiple teams. A public status page lets application owners verify that backup infrastructure is healthy without needing kubectl access.

1. In Vigilmon, go to Status Pages → New Status Page
2. Give it a name: "Backup Infrastructure — k8up"
3. Add your monitors: k8up operator health, S3 backup storage TCP, NFS mount TCP
4. Group them:
   • k8up Operator — operator HTTP health, operator TCP
   • S3 Backup Storage — S3 endpoint TCP 443
   • Backup Jobs — NFS PVC source TCP (if applicable)
5. Publish the page

Share the status page URL in your runbook and with teams that depend on k8up for their backup coverage.

Putting it all together

Here's a summary of the monitors to create for a k8up deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30880/healthz | HTTP | Operator crash, OOMKill, pod restart loops | | <node-ip>:30880 | TCP | Operator port reachability | | <s3-endpoint>:443 | TCP | S3 storage unreachable — catches backup failures before jobs start | | <nfs-server-ip>:2049 | TCP | NFS mount source unreachable — PVC read failures during backup |

Verify the full k8up state in your cluster:

# List all k8up CRD types installed in the cluster
kubectl get crd | grep k8up.io

# Check Schedule objects and their next run time
kubectl get schedules.k8up.io -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.backend.s3.bucket,GENERATION:.metadata.generation'

# Check the status of all Backup objects
kubectl get backups.k8up.io -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STARTED:.status.started,FINISHED:.status.finished,CONDITIONS:.status.conditions[*].type'

# Check Archive and Restore CRDs
kubectl get archives.k8up.io -A
kubectl get restores.k8up.io -A

# Check PreBackupPod objects (application-consistent backup hooks)
kubectl get prebackuppods.k8up.io -A

# View k8up operator logs for scheduling and Restic errors
kubectl logs -n k8up-system deploy/k8up --tail=50

What's next

• Heartbeat monitors on successful backup jobs — configure your backup Schedule to call a Vigilmon heartbeat URL on completion; if a backup window passes without a heartbeat, Vigilmon alerts you that the job never ran
• Multi-bucket coverage — if different namespaces or Schedule objects target different S3 buckets or endpoints, add a TCP monitor for each S3 endpoint so a single bucket failure doesn't go undetected
• Monitoring PreBackupPod health — add a dedicated monitor for any service port that PreBackupPod probes use; if the application sidecar that k8up contacts to quiesce the app is unreachable, your backup silently drops to crash-consistent without warning

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.