How to Monitor MayaStor (OpenEBS NVMe-oF Storage) with Vigilmon

MayaStor is OpenEBS's high-performance storage engine built on NVMe-oF (NVMe over Fabrics). It delivers near bare-metal NVMe latency to Kubernetes workloads by running an I/O path in user space via SPDK. With that performance comes a tightly coupled architecture — pool health, replica placement, and NVMe target availability all affect whether your stateful pods can write at all.

This tutorial shows you how to monitor MayaStor's critical health surfaces using Vigilmon so you catch storage degradation before it becomes application downtime.

Why MayaStor needs external monitoring

MayaStor surfaces metrics through Prometheus, but Prometheus only knows what the storage plane reports internally. External monitoring adds a second opinion that catches:

I/O engine (io-engine) pod restarts — a crashed io-engine means all NVMe targets on that node are gone; Kubernetes may not immediately reschedule storage-class replicas
DiskPool degraded or faulted — a pool losing a disk transitions to Degraded and stops accepting new volumes
Replica count dropping below threshold — a replica loss makes volumes vulnerable to data unavailability on a second failure
NVMe target unreachability — the NVMe-oF target endpoint can be unreachable even when the pod shows Running
REST API unavailability — the MayaStor control-plane REST API is your single source of truth for pool and volume state; if it's down, operators are flying blind

What you'll need

A Kubernetes cluster with MayaStor / OpenEBS installed (v2.x or later)
kubectl access with permissions to read MayaStor resources
A free Vigilmon account

Step 1: Expose the MayaStor REST API health endpoint

MayaStor ships a REST API server (rest deployment in the mayastor namespace). Expose it internally so Vigilmon can reach it, or proxy it through an Ingress with an auth header.

# Confirm the REST API pod is running
kubectl get pods -n mayastor -l app=rest

# Expose via a NodePort for external health checks
kubectl expose deployment rest \
  --name=rest-external \
  --type=NodePort \
  --port=8081 \
  --target-port=8081 \
  -n mayastor

Add a simple liveness path to your Ingress so Vigilmon can probe it without exposing the full API:

# ingress-mayastor-health.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mayastor-health
  namespace: mayastor
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /v0/nodes
spec:
  ingressClassName: nginx
  rules:
    - host: storage-health.example.com
      http:
        paths:
          - path: /health
            pathType: Prefix
            backend:
              service:
                name: rest
                port:
                  number: 8081

Test the endpoint:

curl -s https://storage-health.example.com/health | head -c 200

A 200 response with a JSON node list means the control plane is alive.

Step 2: Deploy a pool-health exporter sidecar

The REST API returns pool state as JSON. Write a tiny polling sidecar that exposes a /healthz endpoint that returns 200 only when all pools are Online.

# pool_health_server.py
import os, json, urllib.request, http.server

MAYASTOR_REST = os.getenv("MAYASTOR_REST_URL", "http://rest.mayastor:8081")

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/healthz":
            self.send_response(404); self.end_headers(); return
        try:
            with urllib.request.urlopen(f"{MAYASTOR_REST}/v0/pools", timeout=5) as r:
                pools = json.loads(r.read())
            degraded = [p for p in pools if p.get("state", {}).get("status") != "Online"]
            if degraded:
                self.send_response(503)
                self.end_headers()
                self.wfile.write(json.dumps({"degraded": [p["id"] for p in degraded]}).encode())
            else:
                self.send_response(200)
                self.end_headers()
                self.wfile.write(b'{"status":"ok"}')
        except Exception as e:
            self.send_response(503)
            self.end_headers()
            self.wfile.write(str(e).encode())
    def log_message(self, *args): pass

http.server.HTTPServer(("", 9090), Handler).serve_forever()

Deploy as a sidecar next to your MayaStor REST pod or as a standalone Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mayastor-pool-healthz
  namespace: mayastor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mayastor-pool-healthz
  template:
    metadata:
      labels:
        app: mayastor-pool-healthz
    spec:
      containers:
        - name: healthz
          image: python:3.12-slim
          command: ["python", "/app/pool_health_server.py"]
          env:
            - name: MAYASTOR_REST_URL
              value: "http://rest.mayastor.svc:8081"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: script
              mountPath: /app
      volumes:
        - name: script
          configMap:
            name: mayastor-pool-healthz-script
---
apiVersion: v1
kind: Service
metadata:
  name: mayastor-pool-healthz
  namespace: mayastor
spec:
  selector:
    app: mayastor-pool-healthz
  ports:
    - port: 9090
      targetPort: 9090
  type: LoadBalancer

Step 3: Monitor io-engine pod availability

The io-engine DaemonSet runs on every storage node. If any pod is down, that node's NVMe targets are lost. Create a Kubernetes Job that polls this and exposes a health endpoint:

# Quick check — use this to validate before wiring up Vigilmon
kubectl get ds io-engine -n mayastor \
  -o jsonpath='{.status.numberReady}/{.status.desiredNumberScheduled}'

Expose this check through an HTTP endpoint your Vigilmon monitor can reach. The simplest approach is a CronJob that writes a status file, served by a tiny nginx pod — or use the Prometheus metrics endpoint directly if you have kube-prometheus-stack:

# Number of io-engine pods not ready
kube_daemonset_status_number_ready{daemonset="io-engine",namespace="mayastor"}
  < kube_daemonset_status_desired_number_scheduled{daemonset="io-engine",namespace="mayastor"}

Configure a Prometheus alerting rule and expose a /healthy endpoint via alertmanager-webhook if you want Vigilmon to consume it as an HTTP target.

Step 4: Set up Vigilmon monitors

Log into Vigilmon and create the following monitors:

Monitor 1 — MayaStor REST API

| Field | Value | |---|---| | Type | HTTP | | URL | https://storage-health.example.com/health | | Expected status | 200 | | Interval | 60 seconds | | Alert after | 2 consecutive failures |

Monitor 2 — Pool health endpoint

| Field | Value | |---|---| | Type | HTTP | | URL | http://<LoadBalancer-IP>:9090/healthz | | Expected status | 200 | | Keyword check | "status":"ok" | | Interval | 60 seconds |

Monitor 3 — NVMe target TCP reachability

MayaStor exposes NVMe-oF over TCP on port 4420. Add a TCP monitor for each storage node:

| Field | Value | |---|---| | Type | TCP | | Host | <storage-node-IP> | | Port | 4420 | | Interval | 30 seconds |

Step 5: Configure alerting

In Vigilmon, go to Alert Channels and add your on-call destination:

Slack:    #storage-alerts
PagerDuty: storage-p1 escalation policy
Email:    platform-team@example.com

Set escalation thresholds:

Pool degraded → immediate P1 (storage writes may be at risk)
REST API down → P1 (operators cannot inspect volume state)
NVMe TCP unreachable → P1 after 2 failures (node's volumes are inaccessible)
io-engine pod down → P2 → P1 after 5 minutes (replica rebuilding may be in progress)

Step 6: Baseline your SLAs

MayaStor is often used for databases and stateful workloads with strict latency SLAs. Use Vigilmon's response-time history to baseline:

REST API response time: should be under 200ms
Pool health endpoint: should respond in under 500ms
TCP connect on port 4420: should be under 5ms on the same subnet

Set Vigilmon response-time alerts at 2× the baseline to catch storage-plane slowdowns before they cascade into application timeouts.

What to monitor — quick reference

| Signal | Check | Alert threshold | |---|---|---| | Pool status | HTTP /healthz returns 200 | Any non-200 | | REST API alive | HTTP /health returns 200 | 2 consecutive failures | | NVMe TCP target | TCP port 4420 | 2 consecutive failures | | io-engine DaemonSet | Pod count == desired | Any pod missing > 2 min | | Replica count | Prometheus metric | Below desired count |

Vigilmon for high-performance storage SLAs

MayaStor's performance profile makes it ideal for latency-sensitive workloads — but that same tight coupling means failures surface fast and hard. Vigilmon's multi-region probes give you an independent signal that doesn't share fate with your storage plane. When the control API goes dark or a pool degrades at 3 AM, Vigilmon pages you before your application team files a support ticket.