How to Monitor TopoLVM with Vigilmon (CSI Storage + Alerts)

TopoLVM brings capacity-aware local persistent volumes to Kubernetes by provisioning LVM logical volumes directly on cluster nodes. That local storage model is fast and cost-effective, but it introduces failure modes that Kubernetes itself cannot catch — node disk saturation, volume group exhaustion, LVM metadata corruption, and CSI controller unavailability all manifest silently inside the cluster while your workloads wait or crash.

This tutorial shows you how to add external uptime monitoring to your TopoLVM deployment using Vigilmon — free tier, no credit card required.

Why TopoLVM needs external monitoring

TopoLVM's scheduler extender and CSI controller run as in-cluster pods. Kubernetes reports them as Running, but that status says nothing about whether:

• The volume group has remaining capacity — TopoLVM's capacity-aware scheduling fails silently when a VG is full; new PVCs stay Pending indefinitely
• The CSI controller webhook is reachable — if the mutating webhook is unreachable, new pods requiring local storage are rejected
• The lvmd daemon on a node is responding — a crashed lvmd means no new volumes can be provisioned on that node even if pods report healthy
• The storage endpoint your application exposes is reachable from outside the cluster — ingress or LoadBalancer issues hide behind healthy pod status

External monitoring from Vigilmon catches the end-user impact: the moment your application's storage-backed service stops responding, an alert fires — regardless of what kubectl get pods says.

What you'll need

• A Kubernetes cluster with TopoLVM installed (via Helm or manifests)
• At least one application using TopoLVM PVCs and exposing an HTTP endpoint
• A free Vigilmon account

Step 1: Expose a health endpoint for your storage-backed application

Your monitoring target should be an application that actually uses TopoLVM storage — not just the CSI pods themselves. Here is a minimal example deployment using a TopoLVM StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: topolvm-provisioner
provisioner: topolvm.io
parameters:
  csi.storage.k8s.io/fstype: xfs
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
# app-with-storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: production
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: topolvm-provisioner
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: storage-app
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: storage-app
  template:
    metadata:
      labels:
        app: storage-app
    spec:
      containers:
        - name: storage-app
          image: your-registry/storage-app:latest
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: data
              mountPath: /data
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 30
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: app-data
---
apiVersion: v1
kind: Service
metadata:
  name: storage-app
  namespace: production
spec:
  type: LoadBalancer
  selector:
    app: storage-app
  ports:
    - port: 80
      targetPort: 8080

Apply and verify:

kubectl apply -f storageclass.yaml -f app-with-storage.yaml

# Confirm PVC is bound (TopoLVM provisioned the volume)
kubectl get pvc app-data -n production
# NAME       STATUS   VOLUME                    CAPACITY   ACCESS MODES   STORAGECLASS           AGE
# app-data   Bound    pvc-abc123...             10Gi       RWO            topolvm-provisioner    30s

# Get external IP
kubectl get svc storage-app -n production

# Test the health endpoint
curl http://203.0.113.75/health
# {"status":"ok","storage":"mounted","free_bytes":9663676416}

Step 2: Set up HTTP monitoring in Vigilmon

With your application reachable externally, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors -> New Monitor
2. Choose HTTP / HTTPS
3. Set the URL: http://your-app-ip/health
4. Set the check interval to 1 minute
5. Under Expected response, set:
   • Status code: 200
   • (Optional) Response body contains: "status":"ok"
6. Save the monitor

Multi-region consensus for storage-backed services

Storage failures often cause slow responses before hard failures. Vigilmon's multi-region consensus model probes your endpoint from multiple geographic regions simultaneously. A single slow probe (network blip) does not trigger an alert — but when multiple regions agree the service is unreachable or timing out, an incident is created immediately.

For TopoLVM workloads this matters: LVM I/O errors often cause application hangs rather than clean crashes. Multi-region consensus distinguishes a transient timeout from a genuine storage-induced outage.

Step 3: Monitor the TopoLVM CSI controller TCP port

The TopoLVM controller exposes a gRPC endpoint (default port 9808) for CSI operations. You can expose it for TCP monitoring via a NodePort:

# topolvm-controller-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: topolvm-controller-monitor
  namespace: topolvm-system
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: topolvm-controller
  ports:
    - port: 9808
      targetPort: 9808
      nodePort: 30808
      protocol: TCP

Then in Vigilmon:

1. Go to Monitors -> New Monitor
2. Choose TCP Port
3. Enter your node IP and port 30808
4. Save

Now you will know if the CSI controller becomes unreachable at the TCP level — before application errors surface.

Step 4: Configure alert channels

Email alerts

1. Go to Alert Channels -> Add Channel -> Email
2. Enter your on-call address
3. Assign to all TopoLVM monitors

Webhook alerts (Slack, PagerDuty, etc.)

1. Go to Alert Channels -> Add Channel -> Webhook
2. Enter your webhook URL
3. Assign to your monitors

Example Vigilmon alert payload:

{
  "monitor_name": "storage-app /health",
  "status": "down",
  "url": "http://203.0.113.75/health",
  "started_at": "2024-06-10T14:00:00Z",
  "duration_seconds": 90
}

When you receive an alert, correlate with TopoLVM state:

# Check LVM volume group capacity on the affected node
kubectl exec -n topolvm-system ds/topolvm-node -- lvs --reportformat json

# Check TopoLVM controller logs
kubectl logs -n topolvm-system deploy/topolvm-controller -c topolvm-controller --tail=50

# Verify PVC is still bound
kubectl get pvc app-data -n production

Step 5: Create a public status page

1. In Vigilmon, go to Status Pages -> New Status Page
2. Name it "Storage Infrastructure"
3. Add your monitors: HTTP health endpoint, TCP CSI controller
4. Publish the page

Share the URL with your team so everyone can check storage health without kubectl access during an incident.

Summary: monitors to create for TopoLVM

| Monitor | Type | What it catches | |---------|------|-----------------| | http://your-app-ip/health | HTTP | Application outages, storage mount failures, LVM I/O errors | | node-ip:30808 | TCP | CSI controller unreachability | | https://your-app-domain/health | HTTP | Ingress/DNS/TLS issues on top of storage layer |

What's next

• Heartbeat monitoring — add a Vigilmon heartbeat monitor for any CronJobs that write to TopoLVM volumes; you will know immediately if the scheduled job stops completing
• SSL monitoring — if your app uses TLS, Vigilmon tracks cert expiry and alerts before renewals fail
• Multi-environment — add separate monitors for staging and production TopoLVM clusters with grouped status pages

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.