How to Monitor OpenEBS with Vigilmon (Control Plane Health + Volume Availability)

OpenEBS turns your Kubernetes cluster into a software-defined storage platform — providing local PVs, replicated block storage via Mayastor, and legacy engines like cStor and Jiva. Because OpenEBS runs entirely inside Kubernetes, it inherits all of Kubernetes' operational complexity while adding its own: storage engine processes, volume provisioners, and iSCSI targets that can silently fail without any pod showing a crash loop.

In this tutorial you'll set up external uptime monitoring for OpenEBS using Vigilmon, ensuring you know about storage failures before they impact your workloads.

Why OpenEBS needs external monitoring

OpenEBS control plane components are critical path for every PersistentVolume in your cluster. When they fail, new volumes can't be provisioned and existing volumes may become unavailable — but Kubernetes often doesn't reflect this immediately.

Common failure modes that external monitoring catches early:

• NDM (Node Disk Manager) daemon dies — block device discovery stops; new OSD-style volumes fail to provision silently
• Mayastor IO engine crashes — replicated volumes lose a path; read/write latency spikes or I/O stalls before the pod is rescheduled
• cStor pool manager goes offline — existing cStor volumes switch to degraded mode; no immediate pod event is generated
• Volume provisioner webhook fails — new PVC creation silently hangs; application deployments stall indefinitely at ContainerCreating
• REST API of Mayastor control plane times out — storage orchestration stops; existing volumes aren't affected but recovery from any node failure becomes impossible

External HTTP and TCP checks from Vigilmon give you an outside-in view that catches these failures faster than alertmanager rules tied to internal Prometheus metrics.

What you'll need

• Kubernetes cluster with OpenEBS installed (v3.x or later recommended)
• OpenEBS control plane components running in openebs namespace
• For Mayastor: io-engine daemonset and loki/etcd sidecars deployed
• A free Vigilmon account — no credit card required

Step 1: Identify your OpenEBS control plane endpoints

Start by finding which services are exposed and on which ports.

kubectl get svc -n openebs

# Example output:
# openebs-apiserver          ClusterIP   10.96.55.10   <none>   5656/TCP    5d
# openebs-provisioner        ClusterIP   10.96.55.11   <none>   <none>      5d
# mayastor-api-rest          LoadBalancer 10.96.55.20  203.0.113.30  8081/TCP  2d
# mayastor-etcd              ClusterIP   10.96.55.21   <none>   2379/TCP    2d

The key services to monitor externally:

• OpenEBS API server (openebs-apiserver:5656) — if exposed; handles volume lifecycle
• Mayastor REST API (mayastor-api-rest:8081) — primary control plane endpoint for Mayastor
• NDM daemon — no HTTP endpoint, but a TCP socket check on the gRPC port confirms the process is alive

If services are ClusterIP only, expose them via NodePort for external monitoring:

apiVersion: v1
kind: Service
metadata:
  name: mayastor-api-rest-nodeport
  namespace: openebs
spec:
  type: NodePort
  selector:
    app: api-rest
  ports:
    - port: 8081
      targetPort: 8081
      nodePort: 30881

Step 2: Monitor the Mayastor REST API

The Mayastor REST API is the primary control plane interface. It exposes a /v0/nodes endpoint that returns the list of storage nodes and their health status.

curl http://203.0.113.30:8081/v0/nodes
# [{"id":"node-1","state":"Online","grpcEndpoint":"10.0.0.1:10124"},...]

In Vigilmon:

1. Log in at vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to http://203.0.113.30:8081/v0/nodes (or your NodePort equivalent)
4. Set expected status code: 200
5. (Optional) Under Response body contains, add "Online" to confirm at least one storage node is active
6. Set check interval to 1 minute
7. Save the monitor

If this endpoint goes down, the Mayastor control plane is unreachable — new volume provisioning will fail, and node failure recovery won't proceed automatically.

Step 3: Monitor the OpenEBS API server

For clusters using cStor or Jiva engines, the OpenEBS API server handles CRD reconciliation and volume provisioning. If exposed, monitor its health endpoint:

# Check if the API server has a /healthz route
curl http://node-ip:30656/healthz
# {"status":"ok"}

Expose via NodePort if needed:

apiVersion: v1
kind: Service
metadata:
  name: openebs-apiserver-nodeport
  namespace: openebs
spec:
  type: NodePort
  selector:
    app: openebs-apiserver
  ports:
    - port: 5656
      targetPort: 5656
      nodePort: 30656

In Vigilmon, add an HTTP monitor for http://node-ip:30656/healthz with expected status 200.

Step 4: TCP port checks for storage engine processes

OpenEBS storage engine processes communicate over gRPC. A TCP check on the gRPC port confirms the engine process is alive and accepting connections — faster and simpler than parsing HTTP health responses.

Mayastor IO engine gRPC (port 10124):

kubectl get pods -n openebs -l app=io-engine -o wide
# Find each node's IP, then TCP-check port 10124

In Vigilmon:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter the node IP and port 10124
4. Save the monitor

Repeat for each storage node in your cluster. If a node's IO engine goes down, the TCP check fails immediately — even if the Kubernetes pod shows Running in a transient state.

Mayastor etcd (port 2379):

etcd is critical for Mayastor's distributed consensus. A TCP check on port 2379 (exposed as NodePort) confirms quorum members are accepting connections:

kubectl get svc -n openebs mayastor-etcd -o jsonpath='{.spec.ports[0].nodePort}'

Step 5: Monitor the volume provisioner webhook

OpenEBS uses a ValidatingWebhookConfiguration or MutatingWebhookConfiguration for storage admission control. If the webhook service fails, all PVC-related operations cluster-wide may be blocked.

kubectl get svc -n openebs | grep webhook

# openebs-admission-server   ClusterIP   10.96.55.30   <none>   443/TCP   5d

Expose as NodePort and add an HTTP monitor checking /healthz:

apiVersion: v1
kind: Service
metadata:
  name: openebs-webhook-nodeport
  namespace: openebs
spec:
  type: NodePort
  selector:
    app: openebs-admission-server
  ports:
    - port: 443
      targetPort: 443
      nodePort: 30443
      protocol: TCP

Monitor https://node-ip:30443/healthz — allow self-signed certificate if needed.

Step 6: Configure storage alerts

Storage failures compound quickly. A degraded replica that isn't addressed becomes a data loss risk when a second replica fails. Fast alerting is essential.

Set up alert channels in Vigilmon

1. Go to Alert Channels → Add Channel → Email
2. Enter your storage operations email address
3. Assign to all OpenEBS monitors

For webhook integration with PagerDuty or Opsgenie:

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your webhook URL
3. Set alert thresholds — consider alerting immediately (0 confirmations) for storage monitors given the blast radius of storage failures

Correlating alerts with OpenEBS state

When a Vigilmon alert fires, run these diagnostics:

# Check control plane pod status
kubectl get pods -n openebs

# Check Mayastor volumes
kubectl get msv -n openebs

# Check node pool status
kubectl get msp -n openebs

# Check disk/block device status
kubectl get bd -n openebs

# Check provisioner logs
kubectl logs -n openebs -l app=openebs-provisioner --tail=50

If the Mayastor REST API is unreachable but pods show Running, check whether the Pod's container is actually serving traffic:

kubectl exec -n openebs -it $(kubectl get pod -n openebs -l app=api-rest -o name) -- \
  wget -qO- http://localhost:8081/v0/nodes

Recommended monitor set for OpenEBS

| Monitor | Type | What it catches | |---------|------|-----------------| | http://mayastor-api:8081/v0/nodes | HTTP | Mayastor control plane offline, no healthy nodes | | http://openebs-api:30656/healthz | HTTP | OpenEBS API server crash, cStor/Jiva provisioning broken | | node-ip:10124 | TCP | Mayastor IO engine down on a storage node | | node-ip:2379 | TCP | etcd quorum member down, consensus at risk | | https://webhook:30443/healthz | HTTP | Admission webhook failure, PVC operations blocked |

What's next

• Heartbeat monitoring — if you run snapshot or backup CronJobs using Velero or OpenEBS's own backup tools, add Vigilmon heartbeat monitors to alert when scheduled jobs stop completing
• Multi-node TCP coverage — add a TCP monitor per storage node so you know exactly which node's IO engine failed, not just that something is wrong
• Status page for platform teams — expose OpenEBS storage health on a Vigilmon status page so application teams know whether a PVC provisioning delay is a storage platform issue or a misconfiguration in their namespace

Start monitoring free at vigilmon.online — setup takes under two minutes.