tutorial

How to Monitor CSI Baremetal with Vigilmon (HTTP + TCP + Alerts)

CSI Baremetal is Intel's open-source CSI driver for bare-metal Kubernetes clusters that provisions raw disks, drives, and Logical Volume Group (LVG) volumes ...

CSI Baremetal is Intel's open-source CSI driver for bare-metal Kubernetes clusters that provisions raw disks, drives, and Logical Volume Group (LVG) volumes directly from node storage. Unlike cloud-backed CSI drivers, CSI Baremetal manages physical hardware — it discovers drives, creates partitions, and allocates storage at the block device level. When any component of this stack fails, workloads lose access to storage with no cloud provider to absorb the failure.

In this tutorial you'll set up comprehensive uptime monitoring for CSI Baremetal using Vigilmon — free tier, no credit card required.

Why CSI Baremetal needs external monitoring

CSI Baremetal's multi-component architecture — controller, node, drive manager, scheduler extender — means there are more failure points than a typical cloud CSI driver:

  • Drive manager loses a disk — a physical drive disconnects or the drive manager daemon fails to enumerate it; volumes provisioned on that drive become unavailable but the node pod may still report healthy
  • Controller crash during LVG operation — the controller pod restarts during a Logical Volume Group resize or volume group create; the operation is orphaned and the AvailableCapacity object becomes stale, blocking future provisioning on that node
  • Scheduler extender pod down — CSI Baremetal's scheduler extender filters node candidates for volume placement; if the extender is unreachable, the default Kubernetes scheduler ignores drive topology and pods may be scheduled to nodes without the required storage capacity
  • Node agent DaemonSet evicted — a node agent pod is evicted under memory pressure; that node's drives become invisible to the cluster and all new PVC binds to drives on that node fail
  • Drive health degraded — CSI Baremetal tracks drive SMART data; a degraded drive continues serving I/O while accumulating errors; without an external monitor surfacing the drive health endpoint, disk failures go unnoticed until data loss

External monitoring from multiple geographic regions catches failures spanning the physical drive layer, daemon processes, and Kubernetes API layer simultaneously.

What you'll need

  • A Kubernetes cluster with CSI Baremetal installed (Helm or manifest, typically on bare-metal nodes)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose CSI Baremetal health endpoints

Expose the controller health endpoint

The CSI Baremetal controller exposes a health probe on port 9999. Expose it via NodePort:

# csi-baremetal-controller-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: csi-baremetal-controller-health
  namespace: kube-system
spec:
  type: NodePort
  selector:
    app: csi-baremetal-controller
  ports:
    - name: healthz
      port: 9999
      targetPort: 9999
      nodePort: 30999
      protocol: TCP

Apply it:

kubectl apply -f csi-baremetal-controller-health-svc.yaml

Verify the health endpoint:

curl http://<node-ip>:30999/healthz
# ok

Expose the scheduler extender health endpoint

The CSI Baremetal scheduler extender is critical for correct drive topology-aware scheduling. It exposes a health endpoint on port 8889:

# csi-baremetal-scheduler-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: csi-baremetal-scheduler-health
  namespace: kube-system
spec:
  type: NodePort
  selector:
    app: csi-baremetal-se
  ports:
    - name: healthz
      port: 8889
      targetPort: 8889
      nodePort: 30889
      protocol: TCP

kubectl apply -f csi-baremetal-scheduler-health-svc.yaml
curl http://<node-ip>:30889/healthz
# ok

Expose the node agent health endpoint

The node agent DaemonSet manages per-node drive discovery and volume operations. Expose port 9999 on the node agent:

# csi-baremetal-node-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: csi-baremetal-node-health
  namespace: kube-system
spec:
  type: NodePort
  selector:
    app: csi-baremetal-node
  ports:
    - name: healthz
      port: 9999
      targetPort: 9999
      nodePort: 30998
      protocol: TCP

kubectl apply -f csi-baremetal-node-health-svc.yaml
curl http://<node-ip>:30998/healthz
# ok

Confirm all CSI Baremetal components are running:

kubectl get pods -n kube-system -l app.kubernetes.io/name=csi-baremetal
# Expect: controller, node DaemonSet pods, scheduler extender all Running

Step 2: Set up HTTP monitoring in Vigilmon

With health endpoints exposed, add CSI Baremetal to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to http://<node-ip>:30999/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: ok
  6. Save the monitor
  7. Repeat for http://<node-ip>:30889/healthz (scheduler extender)
  8. Repeat for http://<node-ip>:30998/healthz (node agent)

The scheduler extender monitor is particularly important — its failure causes silent scheduling failures that are almost impossible to diagnose from kubectl describe pod output alone.

The multi-region consensus advantage

Vigilmon probes your endpoints from multiple geographic regions simultaneously. A single probe failure on a bare-metal cluster could reflect a network path issue rather than a component failure. Vigilmon's multi-region consensus requires multiple independent probes to agree before triggering an alert — accurate detection without false positives.

Step 3: Monitor your TCP layer

TCP monitoring provides the fastest possible detection of port-level failures and confirms that CSI Baremetal components are accepting connections independently of their HTTP health logic.

In Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your node IP and port 30999 (controller health)
  4. Save the monitor
  5. Add a TCP monitor for port 30889 (scheduler extender)
  6. Add a TCP monitor for port 30998 (node agent)
# Confirm all health ports are reachable
nc -zv <node-ip> 30999
nc -zv <node-ip> 30889
nc -zv <node-ip> 30998

Step 4: Configure alert channels

CSI Baremetal manages physical hardware. A failed controller during an LVG operation can leave storage in an inconsistent state. A failed scheduler extender causes new pods to be placed on nodes without sufficient physical storage. You need to know immediately.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your on-call or infrastructure-team email address
  3. Assign the channel to your CSI Baremetal controller, scheduler extender, and node agent monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload Vigilmon sends when a monitor goes down:

{
  "monitor_name": "csi-baremetal-controller /healthz",
  "status": "down",
  "url": "http://203.0.113.50:30999/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 60
}

When you receive a CSI Baremetal alert, correlate it with the cluster:

# Check all CSI Baremetal pod statuses
kubectl get pods -n kube-system -l app.kubernetes.io/name=csi-baremetal -o wide

# Check controller logs for LVG or provisioning errors
kubectl logs -n kube-system deploy/csi-baremetal-controller --tail=50

# Check scheduler extender logs for filter/prioritize failures
kubectl logs -n kube-system deploy/csi-baremetal-se --tail=50

# Check node agent logs for drive discovery issues
kubectl logs -n kube-system daemonset/csi-baremetal-node --tail=50

# List all drives discovered by CSI Baremetal (custom resources)
kubectl get drives -A

# List available capacity across nodes
kubectl get availablecapacities -A

# List Logical Volume Groups
kubectl get logicalvolumegroups -A

# Check for PVCs stuck in Pending
kubectl get pvc -A | grep Pending

Step 5: Create a public status page

Bare-metal storage infrastructure is typically shared across multiple teams and workloads. A public status page lets operations, development, and DevOps teams check whether CSI Baremetal failures are the root cause of their workload issues.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Storage Infrastructure — CSI Baremetal"
  3. Add your monitors: controller health, scheduler extender health, node agent health
  4. Group them: CSI Controller, Scheduler Extender, Node Agents
  5. Publish the page

Share the status page URL in your storage runbook and with teams that own bare-metal-backed workloads.

Putting it all together

Here's a summary of the monitors to create for a CSI Baremetal deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30999/healthz | HTTP | Controller health, crash loops, LVG operation failures | | http://<node-ip>:30889/healthz | HTTP | Scheduler extender health — silent scheduling failures when down | | http://<node-ip>:30998/healthz | HTTP | Node agent health, drive discovery failures | | <node-ip>:30999 | TCP | Controller port reachability | | <node-ip>:30889 | TCP | Scheduler extender port reachability | | <node-ip>:30998 | TCP | Node agent port reachability |

Verify the full CSI Baremetal state in your cluster:

# List all CSI Baremetal components and their status
kubectl get pods -n kube-system -l app.kubernetes.io/name=csi-baremetal \
  -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,STATUS:.status.phase,NODE:.spec.nodeName'

# Check drive health across all nodes
kubectl get drives -o custom-columns='NAME:.metadata.name,NODE:.spec.NodeId,STATUS:.spec.Status,HEALTH:.spec.Health,SIZE:.spec.Size'

# Check available storage capacity per node
kubectl get availablecapacities -o custom-columns='NAME:.metadata.name,NODE:.spec.NodeId,SIZE:.spec.Size,STORAGE_CLASS:.spec.StorageClass'

# Verify CSI driver registration
kubectl get csidriver baremetal-csi

# View volumes managed by CSI Baremetal
kubectl get volumes.storage.dell.com -A 2>/dev/null || kubectl get csibaremetal -A

What's next

  • Drive health alerting — CSI Baremetal tracks SMART health data per drive; query kubectl get drives in a scheduled job and send alerts when drive health transitions to SUSPECT or BAD
  • AvailableCapacity monitoring — if available capacity across nodes drops below a threshold, new PVC binds fail; add a Vigilmon heartbeat monitor that fires when your capacity check job stops reporting
  • Node agent per-node coverage — in large bare-metal clusters, add a separate HTTP monitor for each node's agent health endpoint to pinpoint which node's drives have become invisible

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles