How to Monitor Kadalu with Vigilmon (HTTP + TCP + Alerts)

Kadalu provides ReadWriteMany (RWX) persistent storage for Kubernetes by managing GlusterFS volumes transparently — you create a KadaluStorage resource and Kadalu handles provisioning, distribution, and replication across your cluster nodes. But Kadalu's distributed architecture means a failure in any brick (the GlusterFS data store on a node) or the Kadalu operator can silently degrade your storage without triggering a Kubernetes pod restart.

In this tutorial you'll set up comprehensive uptime monitoring for Kadalu and its underlying GlusterFS bricks using Vigilmon — free tier, no credit card required.

Why Kadalu needs external monitoring

Kadalu's control plane (the operator) and data plane (GlusterFS bricks + FUSE mounts) can fail independently, and neither triggers a Kubernetes alert automatically:

GlusterFS brick down — a node hosting a brick goes down; Kadalu may still serve read-write traffic from replicated bricks but is operating in a degraded state; no Kubernetes event is raised
Kadalu operator crash — the operator pod restarts silently; any in-progress PVC creation or storage pool resize operation is stalled until it recovers
CSI provisioner component failure — Kadalu's CSI provisioner is a separate pod; if it dies, new PVC requests queue indefinitely
FUSE mount failure — the Kadalu FUSE client on a node fails to remount after a node reboot; pods on that node have their RWX volumes appear empty rather than failing with an obvious error
Storage pool exhausted — a Kadalu storage pool fills up; new PVCs are silently rejected while existing ones still appear healthy

External monitoring from multiple regions gives you a view of Kadalu's health that no internal Kubernetes probe can provide.

What you'll need

A Kubernetes cluster with Kadalu installed (via Helm or kubectl apply)
Kadalu storage pools configured with at least one storage device
A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose Kadalu health endpoints

Expose the Kadalu operator health endpoint

The Kadalu operator exposes a metrics and health endpoint. Expose it via NodePort:

# kadalu-operator-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: kadalu-operator-health
  namespace: kadalu
spec:
  type: NodePort
  selector:
    app: kadalu
    app.kubernetes.io/component: operator
  ports:
    - name: metrics
      port: 8050
      targetPort: 8050
      nodePort: 30850
      protocol: TCP

Apply it:

kubectl apply -f kadalu-operator-health-svc.yaml

Verify the operator metrics endpoint:

curl http://<node-ip>:30850/metrics
# kadalu_operator_up 1

Expose the Kadalu CSI provisioner health endpoint

# kadalu-csi-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: kadalu-csi-health
  namespace: kadalu
spec:
  type: NodePort
  selector:
    app: kadalu-csi-provisioner
  ports:
    - name: healthz
      port: 9808
      targetPort: 9808
      nodePort: 30908
      protocol: TCP

kubectl apply -f kadalu-csi-health-svc.yaml
curl http://<node-ip>:30908/healthz
# ok

Expose GlusterFS brick TCP ports

Each node hosting a Kadalu storage brick runs a GlusterFS daemon on port 24007 (management) and 49152+ (brick ports). Note the IP of each storage node for TCP monitoring:

# List storage nodes
kubectl get kadalustorage -A

# Check GlusterFS daemon on a node
kubectl exec -n kadalu <server-pod> -- gluster peer status

Step 2: Set up HTTP monitoring in Vigilmon

With health endpoints exposed, add Kadalu to Vigilmon:

Log in to vigilmon.online and go to Monitors → New Monitor
Choose HTTP / HTTPS as the type
Set the URL to http://<node-ip>:30850/metrics
Set the check interval to 1 minute
Under Expected response, set:
- Status code: 200
- Response body contains: kadalu_operator_up
Save the monitor
Add a second HTTP monitor for http://<node-ip>:30908/healthz
- Status code: 200
- Response body contains: ok

The multi-region consensus advantage

Kadalu's distributed GlusterFS architecture makes false positive alerting a real risk — a transient network path issue between one Vigilmon probe and your node might look like a Kadalu failure. Vigilmon's multi-region consensus requires multiple independent probes from different geographic locations to agree before triggering an alert. This means you only get paged for genuine storage failures, not network hiccups along one probe's path.

Step 3: Monitor your TCP layer

GlusterFS has specific TCP port requirements that, if blocked or unreachable, silently break replication and client access. Monitor these ports directly.

In Vigilmon:

Go to Monitors → New Monitor
Choose TCP Port
Add a TCP monitor for each storage node: <storage-node-ip> and port 24007 (GlusterFS management daemon)
Add a TCP monitor for port 24008 (GlusterFS management daemon SSL, if enabled)
Add a TCP monitor for the CSI health port: <node-ip>:30908

# Confirm GlusterFS ports are open on storage nodes
nc -zv <storage-node-ip> 24007
# Connection to <storage-node-ip> 24007 port [tcp/glusterfs] succeeded!

# Check which brick ports are in use (varies per volume)
kubectl exec -n kadalu <server-pod> -- ss -tlnp | grep gluster

GlusterFS brick ports (49152+) are harder to monitor with fixed port numbers since they're assigned dynamically. Monitoring port 24007 (the management daemon) is sufficient as a proxy for overall GlusterFS daemon health on each node.

Step 4: Configure alert channels

Kadalu storage degradation is silent — GlusterFS can serve I/O from replicas while a brick is down, masking the problem until replication falls behind or the second replica fails. You need to know the moment any component becomes unhealthy.

Email alerts

In Vigilmon, go to Alert Channels → Add Channel → Email
Enter your on-call or platform-team email address
Assign the channel to your Kadalu operator, CSI provisioner, and GlusterFS node monitors

Webhook alerts (Slack, PagerDuty, etc.)

Go to Alert Channels → Add Channel → Webhook
Enter your webhook URL
Assign the channel to your monitors

Example payload Vigilmon sends when a monitor goes down:

{
  "monitor_name": "kadalu-operator metrics",
  "status": "down",
  "url": "http://203.0.113.50:30850/metrics",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 75
}

When you receive a Kadalu alert, correlate it with the cluster:

# Check Kadalu operator pod
kubectl get pods -n kadalu -l app=kadalu,app.kubernetes.io/component=operator

# Check CSI provisioner pod
kubectl get pods -n kadalu -l app=kadalu-csi-provisioner

# Check server pods (one per storage node)
kubectl get pods -n kadalu -l app.kubernetes.io/component=server

# Check storage pool health
kubectl get kadalustorage -A

# Check GlusterFS peer health from a server pod
kubectl exec -n kadalu <server-pod> -- gluster peer status

# Check for degraded or split-brain volumes
kubectl exec -n kadalu <server-pod> -- gluster volume heal info summary

# Check events for PVC provisioning failures
kubectl get events -n kadalu --sort-by='.lastTimestamp' | tail -30

Step 5: Create a public status page

Kadalu's RWX volumes are often shared across multiple teams' workloads. A public status page provides storage health visibility without requiring cluster access.

In Vigilmon, go to Status Pages → New Status Page
Give it a name: "Storage Infrastructure — Kadalu / GlusterFS"
Add your monitors: operator metrics, CSI health, GlusterFS TCP checks per node
Group them: Kadalu Operator, CSI Provisioner, Storage Nodes
Publish the page

Share the status page URL in your runbook so teams owning RWX workloads know where to check when shared volumes behave unexpectedly.

Putting it all together

Here's a summary of the monitors to create for a Kadalu deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30850/metrics | HTTP | Operator health, operator crash/restart | | http://<node-ip>:30908/healthz | HTTP | CSI provisioner health, PVC creation failures | | <node-ip>:30850 | TCP | Operator port reachability | | <node-ip>:30908 | TCP | CSI provisioner port reachability | | <storage-node-1-ip>:24007 | TCP | GlusterFS management daemon — node 1 | | <storage-node-2-ip>:24007 | TCP | GlusterFS management daemon — node 2 | | <storage-node-3-ip>:24007 | TCP | GlusterFS management daemon — node 3 |

Verify the full Kadalu state in your cluster:

# List all Kadalu pods and their status
kubectl get pods -n kadalu -o wide

# List all KadaluStorage objects (storage pools)
kubectl get kadalustorage -A -o wide

# List PVs backed by Kadalu
kubectl get pv | grep kadalu

# Check volume heal status (split-brain detection)
for pod in $(kubectl get pods -n kadalu -l app.kubernetes.io/component=server -o name); do
  echo "=== $pod ==="
  kubectl exec -n kadalu $pod -- gluster volume heal all info summary 2>/dev/null || true
done

# Check storage capacity
kubectl get kadalustorage -A -o jsonpath='{range .items[*]}{.metadata.name}: {.status.capacity}{"\n"}{end}'

What's next

Split-brain alerting — pair Vigilmon's uptime checks with a periodic CronJob that runs gluster volume heal info summary and pushes a heartbeat to Vigilmon; if the job detects split-brain it stops sending the heartbeat and Vigilmon alerts
Storage capacity monitoring — use Kadalu's Prometheus metrics to alert when a storage pool crosses 80% utilization before it affects workloads
Multi-pool coverage — if you run separate Kadalu storage pools for different workload tiers, add a GlusterFS TCP monitor per node for each pool

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.