tutorial

How to Monitor CSI Driver HostPath with Vigilmon (HTTP + TCP + Alerts)

CSI Driver HostPath is the reference CSI driver implementation maintained by the Kubernetes project. It provisions volumes using host filesystem paths and se...

CSI Driver HostPath is the reference CSI driver implementation maintained by the Kubernetes project. It provisions volumes using host filesystem paths and serves as the canonical example for understanding CSI plugin architecture — running in CI pipelines, development clusters, and as the foundation for custom CSI driver development. Despite being a "reference" driver, it runs in real environments where failures block CI pipelines, development workflows, and driver integration tests.

In this tutorial you'll set up comprehensive uptime monitoring for CSI Driver HostPath using Vigilmon — free tier, no credit card required.

Why CSI Driver HostPath needs external monitoring

CSI Driver HostPath is typically used in automated environments where silent failures cause downstream test failures or blocked pipelines:

  • Driver pod evicted during CI run — a CI job triggers PVC creation; the hostpath driver pod is evicted mid-provision; the PVC stays Pending and the entire CI pipeline times out with an opaque error
  • Node socket directory stale — after a node restart, the kubelet's CSI socket registration directory has stale entries; the driver isn't invoked for mount requests on that node even though its pod shows Running
  • Controller sidecar crash — the external-provisioner or external-attacher sidecar container in the driver pod crashes; PV creation and deletion operations queue up and never complete
  • Host path permissions change — a system update changes permissions on the base directory the driver uses for volume subdirectories; all new volume creates fail with permission denied while the pod appears healthy
  • Volume capacity exhausted — HostPath doesn't enforce storage quotas; the underlying host disk fills up and new volume creates silently fail at the OS level

External monitoring catches these failures immediately, keeping CI pipelines and development environments unblocked.

What you'll need

  • A Kubernetes cluster with CSI Driver HostPath installed (common in Kind, Minikube, and development clusters)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose CSI Driver HostPath health endpoints

Expose the controller plugin health endpoint

The CSI Driver HostPath controller plugin exposes a liveness probe on port 9898. Expose it via NodePort:

# csi-hostpath-controller-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: csi-hostpath-controller-health
  namespace: default
spec:
  type: NodePort
  selector:
    app: csi-hostpathplugin
  ports:
    - name: healthz
      port: 9898
      targetPort: 9898
      nodePort: 30898
      protocol: TCP

Apply it:

kubectl apply -f csi-hostpath-controller-health-svc.yaml

Verify the health endpoint:

curl http://<node-ip>:30898/healthz
# ok

Expose the sidecar container health endpoints

The HostPath driver pod bundles several sidecar containers. Each exposes its own health port. Expose the external-provisioner sidecar (port 9809) for additional visibility:

# csi-hostpath-sidecar-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: csi-hostpath-sidecar-health
  namespace: default
spec:
  type: NodePort
  selector:
    app: csi-hostpathplugin
  ports:
    - name: provisioner-healthz
      port: 9809
      targetPort: 9809
      nodePort: 30809
      protocol: TCP

kubectl apply -f csi-hostpath-sidecar-health-svc.yaml
curl http://<node-ip>:30809/healthz
# ok

Confirm the full driver pod is running with all sidecar containers ready:

kubectl get pods -l app=csi-hostpathplugin -o wide
# Verify READY column shows all containers ready (e.g., 8/8)

Step 2: Set up HTTP monitoring in Vigilmon

With health endpoints exposed, add CSI Driver HostPath to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to http://<node-ip>:30898/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: ok
  6. Save the monitor
  7. Repeat for http://<node-ip>:30809/healthz (external-provisioner sidecar)

The multi-region consensus advantage

Vigilmon probes your endpoints from multiple geographic regions simultaneously. A single probe failure could reflect a transient network path issue. Vigilmon's multi-region consensus requires multiple independent probes to agree before triggering an alert — accurate detection without false positives.

Step 3: Monitor your TCP layer

TCP monitoring confirms port-level reachability before any HTTP-layer failure manifests and catches network policy or firewall changes that block CSI communication.

In Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your node IP and port 30898 (controller health)
  4. Save the monitor
  5. Add a TCP monitor for port 30809 (provisioner sidecar health)
# Confirm health ports are reachable from outside the cluster
nc -zv <node-ip> 30898
# Connection to <node-ip> 30898 port [tcp/*] succeeded!

nc -zv <node-ip> 30809
# Connection to <node-ip> 30809 port [tcp/*] succeeded!

Step 4: Configure alert channels

CSI Driver HostPath failures silently block CI pipelines and development workflows. A failed provisioner sidecar means every PVC creation request queues up indefinitely. You need to know immediately.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your on-call or platform-team email address
  3. Assign the channel to your CSI HostPath controller and sidecar monitors

Webhook alerts (Slack, PagerDuty, CI pipeline webhooks)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload Vigilmon sends when a monitor goes down:

{
  "monitor_name": "csi-hostpath-controller /healthz",
  "status": "down",
  "url": "http://203.0.113.50:30898/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 60
}

When you receive a CSI HostPath alert, correlate it with the cluster:

# Check the full driver pod status including sidecar containers
kubectl get pods -l app=csi-hostpathplugin -o wide

# Describe the pod to see container restart counts and events
kubectl describe pod -l app=csi-hostpathplugin

# Check logs for the main hostpath container
kubectl logs -l app=csi-hostpathplugin -c hostpath --tail=50

# Check logs for the external-provisioner sidecar
kubectl logs -l app=csi-hostpathplugin -c csi-provisioner --tail=50

# Check for PVCs stuck in Pending
kubectl get pvc -A | grep Pending

# Check the CSI driver registration
kubectl get csidrivers hostpath.csi.k8s.io

# Check available disk space on the host path base directory
df -h /var/lib/csi-hostpath-data/

Step 5: Create a public status page

In CI/CD environments, multiple teams share the same development cluster. A public status page lets developers check whether storage driver failures are causing their pipeline failures — without needing kubectl access.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Storage Infrastructure — CSI Driver HostPath"
  3. Add your monitors: controller health, provisioner sidecar health
  4. Group them: CSI Controller, CSI Sidecars
  5. Publish the page

Share the status page URL in CI pipeline failure runbooks so engineers check storage first.

Putting it all together

Here's a summary of the monitors to create for a CSI Driver HostPath deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30898/healthz | HTTP | Controller/driver pod health, crash loops, sidecar failures | | http://<node-ip>:30809/healthz | HTTP | External-provisioner sidecar health | | <node-ip>:30898 | TCP | Controller port reachability | | <node-ip>:30809 | TCP | Provisioner sidecar port reachability |

Verify the full CSI HostPath driver state in your cluster:

# List all pods in the hostpath driver deployment
kubectl get pods -l app=csi-hostpathplugin -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,STATUS:.status.phase,NODE:.spec.nodeName'

# List all PVs provisioned by the hostpath driver
kubectl get pv -o custom-columns='NAME:.metadata.name,DRIVER:.spec.csi.driver,STATUS:.status.phase,CAPACITY:.spec.capacity.storage'

# Check the CSI driver registration and topology keys
kubectl get csidriver hostpath.csi.k8s.io -o yaml

# View volume attachments managed by the driver
kubectl get volumeattachments -o custom-columns='NAME:.metadata.name,ATTACHER:.spec.attacher,NODE:.spec.nodeName,ATTACHED:.status.attached'

# Check for any CSI node objects (per-node driver capabilities)
kubectl get csinodes

What's next

  • CI pipeline integration — pipe Vigilmon webhook alerts into your CI system (Jenkins, GitHub Actions, GitLab CI) so pipeline runs fail fast with a clear "storage driver down" message instead of timing out waiting for PVC binding
  • Heartbeat monitoring — if you run integration tests that write and read volumes, use Vigilmon's heartbeat monitors to alert when test jobs stop reporting completion
  • Custom CSI driver development — if you're building a CSI driver based on HostPath, use Vigilmon to test that your driver's health endpoints match the spec before deploying to production

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles