tutorial

How to Monitor Portworx Backup (PX-Backup) with Vigilmon (HTTP + TCP + Alerts)

Portworx Backup (PX-Backup) is Pure Storage's enterprise Kubernetes backup solution for multi-cluster environments. The pxcentral control plane runs a centra...

Portworx Backup (PX-Backup) is Pure Storage's enterprise Kubernetes backup solution for multi-cluster environments. The pxcentral control plane runs a central management service with a REST API and UI on port 10001, while a stork scheduler and backup agent are deployed to each managed cluster to coordinate application-consistent snapshots. PX-Backup supports AWS S3, Azure Blob, GCS, and NFS as backup destinations — but when the control plane crashes, a stork agent on a managed cluster becomes unreachable, or a cloud storage backend goes down, backups fail silently with no native Kubernetes alert to cluster owners.

In this tutorial you'll set up comprehensive uptime monitoring for PX-Backup — including the control plane API, stork agents, etcd backing store, and cloud storage backends — using Vigilmon — free tier, no credit card required.

Why Portworx Backup needs external monitoring

PX-Backup spans a central control plane and distributed stork agents across every managed cluster. Kubernetes pod health checks only tell you a container is running, not that backup scheduling and data transfer are functional:

  • PX-Backup control plane pod crashes — all backup scheduling and policy enforcement stops across every managed cluster simultaneously; existing backup jobs drain but no new jobs are dispatched and no operator is alerted
  • Stork agent on a managed cluster becomes unreachable — that cluster's scheduled backups silently skip with no alert to the cluster owner; missed backups accumulate undetected until an actual restore is needed
  • Cloud storage location unreachable (S3/Azure/GCS) — backups initiate and begin data transfer, then fail mid-write when the storage endpoint drops; this creates incomplete backup artifacts with data corruption risk and no automatic retry notification
  • PX-Backup API goes unhealthy on port 10001 — automation scripts and CI/CD pipelines integrating with the PX-Backup REST API silently fail; scheduled GitOps-driven backup policies stop being applied without any error surfaced to the pipeline
  • etcd backing PX-Backup becomes degraded — backup state diverges from actual data on storage; job history, restore points, and policy assignments may be lost or return stale data to operators querying the API

External monitoring from multiple geographic regions catches failures spanning the full PX-Backup path — control plane to stork agent to cloud storage — simultaneously.

What you'll need

  • A Kubernetes cluster with PX-Backup installed in the pxcentral namespace (via the PX-Central Helm chart or operator)
  • At least one managed cluster with the stork agent deployed
  • A cloud storage backup location configured (S3, Azure Blob, GCS, or NFS)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose PX-Backup health endpoints

Expose the PX-Backup API health endpoint

PX-Backup exposes its REST API and health endpoint on port 10001. If you already have a LoadBalancer service for PX-Backup, use that external IP directly. To expose it via NodePort for monitoring:

# px-backup-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: px-backup-health
  namespace: pxcentral
spec:
  type: NodePort
  selector:
    app: px-backup
  ports:
    - name: api
      port: 10001
      targetPort: 10001
      nodePort: 30001
      protocol: TCP

Apply it:

kubectl apply -f px-backup-health-svc.yaml

Verify the health endpoint responds:

curl http://<node-ip>:30001/v1/health
# {"status":"ok"}

# Alternative health path used by some PX-Backup versions:
curl http://<node-ip>:30001/health
# ok

Check PX-Backup control plane pod status

Confirm the px-backup deployment is healthy before adding it to Vigilmon:

# Check the px-backup deployment in the pxcentral namespace
kubectl get deployment -n pxcentral px-backup

# Check all pxcentral pods
kubectl get pods -n pxcentral -o wide

# Confirm the API port is listening
kubectl exec -n pxcentral deploy/px-backup -- \
  sh -c "wget -qO- http://localhost:10001/v1/health"

Check stork agents on managed clusters

Stork runs in the kube-system namespace on each managed cluster. Check its status on each cluster:

# List stork pods on a managed cluster (run with that cluster's kubeconfig)
kubectl get pods -n kube-system -l name=stork -o wide

# Check stork deployment readiness
kubectl get deployment -n kube-system stork

# View stork logs for backup errors
kubectl logs -n kube-system deploy/stork --tail=50

Step 2: Set up HTTP monitoring in Vigilmon

With the health endpoint exposed, add PX-Backup to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to http://<node-ip>:30001/v1/health
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: ok
  6. Save the monitor

If the PX-Backup UI is also exposed (typically on a separate port or via ingress), add a second HTTP monitor for the UI endpoint to catch web frontend failures independently of the API.

The multi-region consensus advantage

Vigilmon probes your endpoints from multiple geographic regions simultaneously. Because PX-Backup health depends on both the Kubernetes control plane and external cloud storage backends, a single probe failure could reflect a transient network path issue. Vigilmon's multi-region consensus requires multiple independent probes to agree before triggering an alert — accurate detection without false positives.

Step 3: Monitor your TCP layer

TCP monitoring catches PX-Backup API, etcd, and cloud storage reachability failures before they manifest as HTTP errors or backup job timeouts.

In Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your node IP and port 30001 (PX-Backup API NodePort)
  4. Save the monitor
  5. Add a TCP monitor for <pxcentral-node-ip>:2379 (etcd port)
  6. Add a TCP monitor for s3.amazonaws.com:443 (or your S3-compatible endpoint on 443) to verify cloud storage TCP reachability
  7. Add TCP monitors for stork agent ports on each managed cluster if stork exposes a dedicated health port

The etcd TCP monitor on port 2379 is critical — etcd degradation causes PX-Backup to serve stale or incorrect backup state even when the API pod itself appears healthy.

# Confirm etcd TCP port from the pxcentral namespace
kubectl exec -n pxcentral deploy/px-backup -- \
  sh -c "nc -zv <etcd-ip> 2379 && echo 'etcd reachable'"

# Confirm S3 storage TCP reachability from the cluster
kubectl run px-debug --image=busybox --rm -it --restart=Never -- \
  sh -c "nc -zv s3.amazonaws.com 443 && echo 'S3 reachable'"

Step 4: Configure alert channels

A crashed PX-Backup control plane silently stops all backup scheduling across every managed cluster. An unreachable stork agent means an entire cluster stops being backed up. You need to know immediately.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your on-call or platform-engineering email address
  3. Assign the channel to your PX-Backup API, etcd, stork, and cloud storage monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload Vigilmon sends when a monitor goes down:

{
  "monitor_name": "px-backup /v1/health",
  "status": "down",
  "url": "http://203.0.113.50:30001/v1/health",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 60
}

When you receive a PX-Backup alert, correlate it with the cluster:

# Check the px-backup deployment status
kubectl get deployment -n pxcentral px-backup
kubectl describe deployment -n pxcentral px-backup

# Check all pxcentral pods for crash loops or OOMKill
kubectl get pods -n pxcentral
kubectl describe pod -n pxcentral -l app=px-backup

# Check px-backup logs for API errors
kubectl logs -n pxcentral deploy/px-backup --tail=50

# Check stork pods on managed clusters
kubectl get pods -n kube-system -l name=stork

# Check backup location status via PX-Backup API
curl -X GET http://<node-ip>:30001/v1/backuplocation \
  -H "Authorization: Bearer <token>"

# Check etcd health
kubectl exec -n pxcentral deploy/px-backup -- \
  sh -c "ETCDCTL_API=3 etcdctl --endpoints=<etcd-ip>:2379 endpoint health"

Step 5: Create a public status page

PX-Backup is typically a shared platform service used by multiple development and operations teams. A public status page lets everyone check backup infrastructure health without needing kubectl access or API credentials.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Portworx Backup (PX-Backup) Infrastructure"
  3. Add your monitors: PX-Backup API HTTP, etcd TCP, stork TCP, cloud storage TCP
  4. Group them:
    • PX-Backup Control Plane — API HTTP monitor, API TCP monitor, etcd TCP monitor
    • Stork Agents — TCP monitors for stork on each managed cluster
    • Cloud Backup Storage — S3/Azure Blob/GCS TCP monitors for each configured backup location
  5. Publish the page

Share the status page URL in your runbook and with teams that own workloads protected by PX-Backup policies.

Putting it all together

Here's a summary of the monitors to create for a PX-Backup deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30001/v1/health | HTTP | PX-Backup API health, crash loops, OOMKill | | <node-ip>:30001 | TCP | PX-Backup API port reachability | | <etcd-ip>:2379 | TCP | etcd reachability — backup state loss and divergence | | s3.amazonaws.com:443 (or equivalent) | TCP | Cloud backup storage TCP reachability | | <managed-cluster-node>:<stork-port> | TCP | Stork agent reachability per managed cluster |

Verify the full PX-Backup state in your cluster:

# List all pxcentral pods and their status
kubectl get pods -n pxcentral -o wide

# Check the px-backup deployment rollout status
kubectl rollout status deployment/px-backup -n pxcentral

# List stork pods across managed clusters
kubectl get pods -n kube-system -l name=stork -o wide

# Query backup jobs via PX-Backup API
curl -X GET http://<node-ip>:30001/v1/backup \
  -H "Authorization: Bearer <token>"

# Check configured backup locations
curl -X GET http://<node-ip>:30001/v1/backuplocation \
  -H "Authorization: Bearer <token>"

# View PX-Backup events for recent failures
kubectl get events -n pxcentral --sort-by='.lastTimestamp' | tail -20

What's next

  • Monitor each managed cluster's stork agent separately — add an individual TCP monitor per stork agent so a single unreachable cluster is isolated in your alerts rather than masked by the overall PX-Backup control plane being healthy
  • Heartbeat monitors on backup job completion webhooks — configure PX-Backup to POST to a Vigilmon heartbeat URL on successful backup completion; if the heartbeat stops arriving, Vigilmon alerts you that backups are no longer completing even when all services appear up
  • Alerting on storage location capacity — add HTTP monitors against your S3/GCS/Azure management APIs to track storage bucket availability and catch quota or permissions errors before they block backup writes

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles