tutorial

How to Monitor Local Path Provisioner with Vigilmon (HTTP + TCP + Alerts)

Local Path Provisioner is Rancher's lightweight dynamic storage provisioner that creates host-path persistent volumes on Kubernetes nodes — ideal for develop...

Local Path Provisioner is Rancher's lightweight dynamic storage provisioner that creates host-path persistent volumes on Kubernetes nodes — ideal for development clusters, edge deployments, and single-node Kubernetes installations like K3s. Because it stores data directly on node filesystems, there is no storage backend to fall back to: if the provisioner pod dies or the node filesystem fills up, new PVCs simply never get bound.

In this tutorial you'll set up comprehensive uptime monitoring for Local Path Provisioner using Vigilmon — free tier, no credit card required.

Why Local Path Provisioner needs external monitoring

Local Path Provisioner's simplicity is also its risk surface. There is no distributed storage layer to absorb failures:

  • Provisioner pod crash — if the single provisioner pod restarts during a PV create or delete operation, the PVC stays Pending indefinitely with no automatic retry; workloads stall silently
  • Node disk full — the provisioner creates directories on the host filesystem but does not enforce disk quotas; a full filesystem causes all new PVC binds on that node to fail with no Kubernetes-level alert
  • Config map corruption — Local Path Provisioner reads its configuration (storage path, node affinity rules) from a ConfigMap; accidental deletion or misconfiguration breaks volume provisioning cluster-wide
  • Node eviction after data write — because volumes are node-local, any rescheduling of a pod to a different node loses all previously written data; there is no cross-node replication
  • Provisioner image pull failure — in air-gapped or restricted environments, the provisioner image may become unavailable; after a pod restart the provisioner stays in ImagePullBackOff and no new PVCs bind

External monitoring gives you early warning before provisioning failures block deployments or corrupt state in running workloads.

What you'll need

  • A Kubernetes cluster with Local Path Provisioner installed (common in K3s, Rancher Desktop, and RKE2 clusters)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose Local Path Provisioner health endpoints

Expose the provisioner health endpoint

Local Path Provisioner exposes a health check endpoint on port 2112 by default. Expose it via NodePort so Vigilmon can reach it from outside the cluster:

# local-path-provisioner-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: local-path-provisioner-health
  namespace: local-path-storage
spec:
  type: NodePort
  selector:
    app: local-path-provisioner
  ports:
    - name: healthz
      port: 2112
      targetPort: 2112
      nodePort: 30112
      protocol: TCP

Apply it:

kubectl apply -f local-path-provisioner-health-svc.yaml

Verify the health endpoint:

curl http://<node-ip>:30112/healthz
# ok

Check provisioner pod and ConfigMap state

Before adding external monitors, verify the provisioner is running and its configuration is intact:

# Confirm provisioner pod is running
kubectl get pods -n local-path-storage

# Confirm the config map is present
kubectl get configmap -n local-path-storage local-path-config -o yaml

# Confirm the StorageClass is provisioning
kubectl get storageclass local-path

Expose a node filesystem metrics endpoint (optional)

If you run node-exporter on your cluster nodes, expose disk usage metrics on a NodePort for additional capacity monitoring:

# node-exporter-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: node-exporter-external
  namespace: monitoring
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: node-exporter
  ports:
    - name: metrics
      port: 9100
      targetPort: 9100
      nodePort: 30910
      protocol: TCP

kubectl apply -f node-exporter-svc.yaml
curl http://<node-ip>:30910/metrics | grep node_filesystem_avail_bytes

Step 2: Set up HTTP monitoring in Vigilmon

With the health endpoint exposed, add Local Path Provisioner to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to http://<node-ip>:30112/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: ok
  6. Save the monitor

If you exposed node-exporter, add a second HTTP monitor for http://<node-ip>:30910/metrics to confirm the metrics endpoint is reachable.

The multi-region consensus advantage

Vigilmon probes your endpoints from multiple geographic regions simultaneously. A single probe failure could reflect a transient network issue rather than a provisioner failure. Vigilmon's multi-region consensus requires multiple independent probes to agree before triggering an alert — giving you accurate detection without false positives.

Step 3: Monitor your TCP layer

TCP monitoring catches port-level failures before they surface as HTTP errors and confirms the provisioner is accepting connections at the network layer.

In Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your node IP and port 30112 (provisioner health)
  4. Save the monitor

If your cluster nodes expose the Kubernetes API server externally, also add a TCP monitor for port 6443 — a failed API server causes kubectl operations to hang, which blocks any manual recovery of stuck PVCs.

# Confirm provisioner health port is open from outside the cluster
nc -zv <node-ip> 30112
# Connection to <node-ip> 30112 port [tcp/*] succeeded!

Step 4: Configure alert channels

A failed Local Path Provisioner pod prevents any new PVC from binding. In development clusters this stalls all new deployments. In lightweight production clusters (K3s edge nodes) this can halt critical services. You need to know immediately.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your on-call or platform-team email address
  3. Assign the channel to your Local Path Provisioner health monitor

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload Vigilmon sends when a monitor goes down:

{
  "monitor_name": "local-path-provisioner /healthz",
  "status": "down",
  "url": "http://203.0.113.50:30112/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 60
}

When you receive a Local Path Provisioner alert, correlate it with the cluster:

# Check provisioner pod status
kubectl get pods -n local-path-storage

# Check provisioner logs for errors
kubectl logs -n local-path-storage deploy/local-path-provisioner --tail=50

# Check for PVCs stuck in Pending
kubectl get pvc -A | grep Pending

# Check recent PV provisioning events
kubectl get events -n local-path-storage --sort-by='.lastTimestamp'

# Check available disk space on nodes where local-path stores data
kubectl get nodes -o custom-columns='NAME:.metadata.name,STORAGE:.status.allocatable.ephemeral-storage'

Step 5: Create a public status page

In K3s and Rancher environments, local storage is often the only storage available. A public status page lets developers check whether provisioner issues are causing their deployment failures.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Storage Infrastructure — Local Path Provisioner"
  3. Add your monitors: provisioner health, node TCP
  4. Group them: Provisioner, Node Health
  5. Publish the page

Share the status page URL with your development teams so they can self-diagnose PVC binding failures.

Putting it all together

Here's a summary of the monitors to create for a Local Path Provisioner deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30112/healthz | HTTP | Provisioner pod health, crash loops, OOMKill | | <node-ip>:30112 | TCP | Provisioner port reachability | | http://<node-ip>:30910/metrics | HTTP | Node exporter reachability (optional) | | <node-ip>:6443 | TCP | Kubernetes API server — blocks recovery when down |

Verify the full Local Path Provisioner state in your cluster:

# Check provisioner pod and replica status
kubectl get deploy -n local-path-storage local-path-provisioner

# List all local-path PVs and their binding state
kubectl get pv -o custom-columns='NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]'

# Confirm local-path is the default StorageClass
kubectl get storageclass

# Check recent provisioner activity
kubectl get events -n local-path-storage --field-selector reason=ProvisioningSucceeded,ProvisioningFailed --sort-by='.lastTimestamp'

What's next

  • Disk capacity alerting — pair Vigilmon's uptime monitoring with node-exporter alerts on node_filesystem_avail_bytes to catch disk-full conditions before provisioning fails
  • Heartbeat monitoring — if you have batch jobs writing to local-path PVCs, Vigilmon's heartbeat monitors alert you when a job stops reporting in
  • Multiple node coverage — in multi-node clusters, add a health endpoint monitor for each node's provisioner pod to pinpoint which node is affected

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles