tutorial

How to Monitor Portworx with Vigilmon (Node Health, Volume Replication + Alerts)

Portworx is the cloud-native storage layer for stateful Kubernetes workloads — databases, message queues, and AI/ML pipelines that can't tolerate data loss o...

Portworx is the cloud-native storage layer for stateful Kubernetes workloads — databases, message queues, and AI/ML pipelines that can't tolerate data loss or storage unavailability. Its distributed replication engine and CSI driver integration mean that when Portworx degrades, stateful pods enter CrashLoopBackOff and persistent volumes become unavailable across the cluster.

This tutorial covers the full Portworx observability stack: internal metrics from the Portworx Prometheus endpoint, alert rules for node health and volume replication, and Vigilmon external uptime checks that verify your Portworx storage services are reachable from outside the cluster.

Why Portworx needs external monitoring

Portworx ships with built-in Prometheus metrics and integrates with Grafana through the PX-Central Operator. But internal monitoring has a fundamental limitation: it runs inside the same Kubernetes cluster it monitors. When Portworx itself is the problem — node failure, storage pool exhaustion, CSI driver crash — internal monitoring may be the first casualty.

External monitoring catches what internal tooling misses:

  • Portworx node eviction — a storage node gets evicted from the Kubernetes cluster; its volumes become degraded and Portworx begins cross-node replication immediately, but the eviction itself isn't visible in Prometheus until after the fact
  • kvdb (etcd) connectivity loss — Portworx loses connectivity to its internal key-value store (built-in etcd or external); volume operations stall while the cluster appears healthy to application pods
  • Storage pool exhaustion — a storage pool hits 100% capacity; new volume provisions fail with cryptic CSI errors; no Kubernetes event fires until the pod PVC gets stuck in Pending
  • CSI driver pod crash — the portworx-api DaemonSet pod on a node crashes; PVC mounts on that node fail for new pods while running pods keep their existing mounts
  • PX-Central Lighthouse unreachable — your management console is down; you can't diagnose or remediate without it

Vigilmon monitors Portworx from outside Kubernetes — confirming reachability from the same position as your users and external systems.

What you'll need

  • Kubernetes cluster with Portworx installed (v3.0 or later recommended)
  • kubectl access with cluster-admin
  • pxctl CLI available (runs inside the Portworx pod or as a node binary)
  • A free Vigilmon account — no credit card required

Step 1: Verify Portworx node and volume status

Check the Portworx cluster status using pxctl via the DaemonSet pod:

# Get the Portworx pod on any node
PX_POD=$(kubectl get pods -n kube-system -l name=portworx -o jsonpath='{.items[0].metadata.name}')

# Cluster status
kubectl exec -n kube-system "$PX_POD" -- /opt/pwx/bin/pxctl status
# Status: PX is operational
# License: Enterprise
#
# Node ID: a1b2c3d4-...
#   IP: 10.0.1.10
#   Local Storage Pool: 1 pool
#   POOL  IO_PRIORITY  RAID_LEVEL  USABLE  USED  STATUS  ZONE  REGION
#   0     HIGH         raid5       500 GiB  10 GiB  Online  us-east-1a  us-east-1

# Check all nodes
kubectl exec -n kube-system "$PX_POD" -- /opt/pwx/bin/pxctl cluster list
# ID             IP          CPU    MEM TOTAL  MEM FREE  CONTAINERS  VERSION  STATUS
# a1b2c3d4...   10.0.1.10   2.1%   16 GB      12 GB     5           3.0.0    Online

# Check volume replication
kubectl exec -n kube-system "$PX_POD" -- /opt/pwx/bin/pxctl volume list
# ID           NAME        SIZE   HA  SHARED  ENCRYPTED  STATUS
# 123456789   postgres-pv  100 GiB  3   no      no         up - attached on 10.0.1.10

Volume with HA: 3 means 3-way replication across nodes. If STATUS shows degraded, replication is compromised.

Step 2: Scrape Portworx Prometheus metrics

Portworx exposes Prometheus metrics on port 9001 of each storage node:

# Port-forward to verify
kubectl port-forward -n kube-system "$PX_POD" 9001:9001 &
curl -s http://localhost:9001/metrics | grep "^px_" | head -20

Key metrics:

px_cluster_status_cluster_size         # total nodes in cluster
px_cluster_status_cluster_quorum       # 1 if quorum reached, 0 if not
px_node_status_node_status             # per-node status (1=online)
px_volume_status_vol_status            # per-volume status (1=up, 0=down)
px_volume_stats_replication_status     # 1=fully replicated, 0=degraded
px_volume_capacity_total_bytes         # volume total capacity
px_volume_capacity_usage_bytes         # volume used bytes
px_pool_stats_pool_status             # storage pool status
px_pool_capacity_total_bytes           # pool total capacity
px_pool_capacity_usage_bytes           # pool used bytes
px_network_io_bytes_total              # network I/O per interface

Prometheus alert rules for Portworx:

- alert: PortworxNodeOffline
  expr: px_node_status_node_status == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Portworx node {{ $labels.node }} is offline"

- alert: PortworxClusterQuorumLoss
  expr: px_cluster_status_cluster_quorum == 0
  for: 30s
  labels:
    severity: critical
  annotations:
    summary: "Portworx cluster has lost quorum — volume operations may stall"

- alert: PortworxVolumeReplicationDegraded
  expr: px_volume_stats_replication_status == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Portworx volume {{ $labels.volumename }} replication is degraded"

- alert: PortworxVolumeDown
  expr: px_volume_status_vol_status == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Portworx volume {{ $labels.volumename }} is down"

- alert: PortworxStoragePoolCapacityHigh
  expr: |
    px_pool_capacity_usage_bytes / px_pool_capacity_total_bytes > 0.80
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Portworx storage pool on {{ $labels.node }} is above 80% capacity"

Step 3: Expose the Portworx API endpoint for external checks

Portworx exposes a REST API on port 9001. Create a NodePort Service so Vigilmon can reach it from outside the cluster:

apiVersion: v1
kind: Service
metadata:
  name: portworx-api-external
  namespace: kube-system
spec:
  selector:
    name: portworx
  ports:
    - name: metrics
      port: 9001
      targetPort: 9001
      nodePort: 30901
  type: NodePort

kubectl apply -f portworx-api-external.yaml

Test the health endpoint from outside the cluster:

curl http://<NODE_IP>:30901/v1/cluster/versions
# {"Version":"3.0.0","PxVersion":"3.0.0",...}

For the CSI driver status, check the DaemonSet availability via the Kubernetes API or use the /health endpoint:

# Check CSI driver DaemonSet desired vs available
kubectl get daemonset portworx -n kube-system -o jsonpath='{.status.desiredNumberScheduled},{.status.numberAvailable}'
# 3,3  (all nodes running)

Step 4: Add Portworx checks in Vigilmon

Log in to Vigilmon and create monitors for each Portworx endpoint:

Portworx API version endpoint (per node):

  1. Click Add Monitor → HTTP(S)
  2. URL: http://<NODE_IP>:30901/v1/cluster/versions
  3. Check interval: 60 seconds
  4. Expected keyword: PxVersion
  5. Configure alert channels (Slack, PagerDuty, email, webhook)

Prometheus metrics endpoint (spot-check):

  1. URL: http://<NODE_IP>:30901/metrics
  2. Expected keyword: px_cluster_status_cluster_quorum

Add a monitor for each node you want to verify independently. Portworx distributes storage across all nodes, so a single-node check isn't sufficient — you want at least 3 monitors for a 3-node cluster.

PX-Central Lighthouse (if deployed):

PX-Central provides a management UI and REST API:

# Get Lighthouse URL
kubectl get svc px-lighthouse -n kube-system

Add a Vigilmon HTTP monitor for the Lighthouse login URL with expected keyword matching the page title or a known string in the HTML.

Step 5: Validate alert paths end-to-end

Test that Vigilmon fires before your on-call receives a PagerDuty page via Kubernetes events:

# Simulate a node going offline by cordoning and draining
kubectl cordon <STORAGE_NODE>

# Watch Portworx detect the change
kubectl exec -n kube-system "$PX_POD" -- /opt/pwx/bin/pxctl cluster list
# The cordoned node should show status change within 30-60 seconds

Portworx will begin rebalancing replication to remaining nodes. Vigilmon should alert within 60–120 seconds if the node's API endpoint becomes unreachable.

Restore the node:

kubectl uncordon <STORAGE_NODE>

Recommended monitoring checklist

| Signal | Source | Threshold | |--------|--------|-----------| | Node online status | Portworx Prometheus | Any node offline | | Cluster quorum | Portworx Prometheus | Quorum = 0 | | Volume replication status | Portworx Prometheus | Any volume degraded | | Storage pool utilization | Portworx Prometheus | > 80% | | Volume availability | Portworx Prometheus | Any volume down | | API endpoint reachability | Vigilmon (external) | Any downtime per node | | PX-Central Lighthouse | Vigilmon (external) | Any downtime |

Conclusion

Portworx underpins your most critical stateful Kubernetes workloads. A degraded storage node or split replication set can take down databases and message queues that your entire application stack depends on. Pairing Portworx's built-in Prometheus metrics with Vigilmon's external uptime monitoring gives you defense in depth: internal alerts catch volume-level degradation early, while Vigilmon confirms the storage API and management plane are reachable from outside the cluster — the ground truth your on-call needs during an incident.

Set up your first Portworx monitor at vigilmon.online in under five minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles