tutorial

How to Monitor CSI Driver NFS with Vigilmon (HTTP + TCP + Alerts)

CSI Driver NFS lets Kubernetes workloads mount NFS server volumes directly as persistent volumes without a separate provisioner — the CSI driver handles volu...

CSI Driver NFS lets Kubernetes workloads mount NFS server volumes directly as persistent volumes without a separate provisioner — the CSI driver handles volume creation, attachment, and mounting in-process. But the CSI driver's node plugin runs as a DaemonSet on every node, and when any node's plugin fails, pods scheduled to that node silently lose their ability to mount NFS volumes.

In this tutorial you'll set up comprehensive uptime monitoring for CSI Driver NFS — and the NFS servers behind it — using Vigilmon — free tier, no credit card required.

Why CSI Driver NFS needs external monitoring

CSI Driver NFS is distributed: the controller plugin handles PV lifecycle and the node plugin handles per-node mounting. Internal Kubernetes health checks only tell you each pod is alive, not that the NFS data path is functional:

  • NFS server goes down — the CSI driver node pods are Running and healthy, but all NFS mounts stall; pods that try to mount or remount volumes hang indefinitely
  • Node plugin pod evicted — a node runs low on memory and evicts the CSI driver DaemonSet pod; pods scheduled to that node can't mount new NFS volumes
  • CSI registration socket stale — after a node restart, the kubelet's CSI registration socket directory has stale entries; the driver isn't invoked for mount requests on that node
  • Controller plugin crash — the controller pod restarts during a PV create/delete operation; the operation is orphaned and the PVC stays Terminating or Pending
  • NFS server version mismatch — an NFS server upgrade changes the protocol version; mount operations fail silently at the kernel level while the CSI pod reports healthy

External monitoring from multiple geographic regions is the safety net that catches failures spanning the NFS data path and CSI driver layers simultaneously.

What you'll need

  • A Kubernetes cluster with CSI Driver NFS installed (via Helm or manifest)
  • One or more NFS servers providing volumes to the cluster
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose CSI Driver NFS health endpoints

Expose the controller plugin health endpoint

The CSI Driver NFS controller plugin exposes a CSI health check on port 29652 and a liveness probe on port 29652. Expose it via NodePort:

# csi-nfs-controller-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: csi-nfs-controller-health
  namespace: kube-system
spec:
  type: NodePort
  selector:
    app: csi-nfs-controller
  ports:
    - name: healthz
      port: 29652
      targetPort: 29652
      nodePort: 30652
      protocol: TCP

Apply it:

kubectl apply -f csi-nfs-controller-health-svc.yaml

Verify the health endpoint:

curl http://<node-ip>:30652/healthz
# ok

Check node plugin DaemonSet health

The node plugin runs on every node as a DaemonSet. Its health is best monitored indirectly by checking that the DaemonSet has the expected number of ready pods:

kubectl get daemonset -n kube-system csi-nfs-node
# NAME           DESIRED   CURRENT   READY   ...
# csi-nfs-node   3         3         3       ...

For direct external monitoring of the node plugin, you can expose port 29653 (node liveness probe) on a representative node:

# csi-nfs-node-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: csi-nfs-node-health
  namespace: kube-system
spec:
  type: NodePort
  selector:
    app: csi-nfs-node
  ports:
    - name: healthz
      port: 29653
      targetPort: 29653
      nodePort: 30653
      protocol: TCP

kubectl apply -f csi-nfs-node-health-svc.yaml
curl http://<node-ip>:30653/healthz
# ok

Step 2: Set up HTTP monitoring in Vigilmon

With health endpoints exposed, add CSI Driver NFS to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to http://<node-ip>:30652/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: ok
  6. Save the monitor
  7. Repeat for http://<node-ip>:30653/healthz (node plugin)

If your NFS server exposes an HTTP health or management endpoint, add that as a third HTTP monitor.

The multi-region consensus advantage

Vigilmon probes your endpoints from multiple geographic regions simultaneously. Because CSI Driver NFS health depends on both the Kubernetes control plane and external NFS servers, a single probe failure could reflect a transient network path issue rather than an actual driver failure. Vigilmon's multi-region consensus requires multiple independent probes to agree before triggering an alert — giving you accurate detection without false positives.

Step 3: Monitor your TCP layer

TCP monitoring catches port-level failures before they manifest as HTTP errors, and lets you monitor the NFS server directly without requiring an HTTP endpoint.

In Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your node IP and port 30652 (controller health)
  4. Save the monitor
  5. Add a TCP monitor for port 30653 (node plugin health)
  6. Add a TCP monitor for <nfs-server-ip> on port 2049 (NFS protocol port)

The NFS server TCP monitor is critical — it's your earliest warning that the storage backend has become unreachable, before any CSI pod reports an error.

# Confirm NFS server TCP port from cluster
nc -zv <nfs-server-ip> 2049
# Connection to <nfs-server-ip> 2049 port [tcp/nfs] succeeded!

# If using NFSv4 with 111 portmapper:
nc -zv <nfs-server-ip> 111

Step 4: Configure alert channels

A failed CSI Driver NFS node plugin blocks all new NFS volume mounts on that node. An unreachable NFS server stalls all mounted volumes cluster-wide. You need to know immediately.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your on-call or storage-team email address
  3. Assign the channel to your CSI NFS controller, node, and NFS server monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload Vigilmon sends when a monitor goes down:

{
  "monitor_name": "csi-nfs-controller /healthz",
  "status": "down",
  "url": "http://203.0.113.50:30652/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 60
}

When you receive a CSI NFS alert, correlate it with the cluster:

# Check CSI controller pod status
kubectl get pods -n kube-system -l app=csi-nfs-controller

# Check CSI node plugin DaemonSet — look for not-Ready nodes
kubectl get pods -n kube-system -l app=csi-nfs-node -o wide

# Check for pending or stuck volume attachments
kubectl get volumeattachments -A

# Check for PVCs stuck in Terminating
kubectl get pvc -A | grep Terminating

# Check CSI controller logs for mount errors
kubectl logs -n kube-system deploy/csi-nfs-controller -c nfs --tail=50

# Test NFS reachability from a node
kubectl run nfs-debug --image=busybox --rm -it --restart=Never -- \
  sh -c "nc -zv <nfs-server-ip> 2049"

Step 5: Create a public status page

NFS-backed storage is often shared across many teams. A public status page lets everyone check storage health without needing kubectl access.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Storage Infrastructure — CSI Driver NFS"
  3. Add your monitors: controller health, node plugin health, NFS server TCP
  4. Group them: CSI Controller, CSI Node Plugin, NFS Servers
  5. Publish the page

Share the status page URL in your runbook and with teams that own NFS-backed workloads.

Putting it all together

Here's a summary of the monitors to create for a CSI Driver NFS deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30652/healthz | HTTP | Controller plugin health, crash loops, OOMKill | | http://<node-ip>:30653/healthz | HTTP | Node plugin health, DaemonSet pod evictions | | <node-ip>:30652 | TCP | Controller port reachability | | <node-ip>:30653 | TCP | Node plugin port reachability | | <nfs-server-ip>:2049 | TCP | NFS server reachability — catches mount failures cluster-wide | | <nfs-server-ip>:111 | TCP | NFS portmapper (NFSv3 environments) |

Verify the full CSI NFS driver state in your cluster:

# List all CSI NFS pods
kubectl get pods -n kube-system -l 'app in (csi-nfs-controller,csi-nfs-node)' -o wide

# List PVs backed by CSI NFS
kubectl get pv -o custom-columns='NAME:.metadata.name,DRIVER:.spec.csi.driver,SERVER:.spec.csi.volumeAttributes.server'

# Check the CSI driver registration
kubectl get csidrivers nfs.csi.k8s.io

# View attached volumes and their nodes
kubectl get volumeattachments -o custom-columns='NAME:.metadata.name,ATTACHER:.spec.attacher,NODE:.spec.nodeName,ATTACHED:.status.attached'

What's next

  • Multiple NFS server coverage — if different storage classes point to different NFS servers, add a TCP monitor for each server's port 2049
  • SSL certificate expiry — if your NFS management web UI uses TLS, Vigilmon monitors the certificate and alerts before it expires
  • Heartbeat monitoring — if you run batch jobs that write to NFS-backed PVCs, Vigilmon's heartbeat monitors alert you when a job stops reporting in

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles