tutorial

Monitoring Rook/Ceph on Kubernetes with Vigilmon

Rook orchestrates Ceph storage on Kubernetes — but storage failures are catastrophic and often silent until data loss occurs. Here's how to monitor Rook operator health, Ceph cluster status, and critical TCP ports with Vigilmon.

Rook turns Ceph — the distributed storage system used at hyperscaler scale — into a Kubernetes-native storage orchestrator. It manages OSDs, MONs, and MGR processes as Kubernetes resources, making block storage, object storage, and shared filesystems available to your cluster. But when Ceph goes unhealthy, it does so loudly and catastrophically: pods can't write to PVCs, StatefulSets freeze, and data integrity warnings stack up. Vigilmon adds an external monitoring layer that watches the Rook operator, Ceph cluster health API, and critical TCP ports, giving you alerts before a degraded cluster becomes a unavailable one.

What You'll Set Up

  • Kubernetes API check for Rook operator pod health
  • Ceph MGR dashboard HTTP health endpoint monitor
  • TCP port monitoring for Ceph MON (6789) and MGR dashboard (7000)
  • SSL certificate monitoring for the Ceph MGR dashboard
  • Heartbeat monitoring for OSD availability checks

Prerequisites

  • Rook 1.12+ deployed on Kubernetes (rook-ceph namespace)
  • Ceph MGR dashboard enabled
  • A free Vigilmon account

Step 1: Monitor Rook Operator Health via Kubernetes API

The Rook operator pod manages all Ceph components. If it crashes or enters a crash loop, no Ceph configuration changes will be reconciled and degraded components won't be replaced.

Expose a lightweight health proxy that wraps kubectl get pod output for Vigilmon to check:

# rook-health-proxy.sh — run this as a service on a node with kubectl access
#!/bin/bash
# Returns 200 if rook operator is Running, 503 otherwise
PHASE=$(kubectl -n rook-ceph get pod \
  -l app=rook-ceph-operator \
  -o jsonpath='{.items[0].status.phase}' 2>/dev/null)

if [ "$PHASE" = "Running" ]; then
  echo -e "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\r\n{\"status\":\"ok\",\"operator\":\"Running\"}"
else
  echo -e "HTTP/1.1 503 Service Unavailable\r\nContent-Type: application/json\r\n\r\n{\"status\":\"error\",\"operator\":\"$PHASE\"}"
fi

Host this behind a minimal HTTP server (e.g. ncat -lk 8080 -e rook-health-proxy.sh) on a management node reachable from Vigilmon, or expose it via a Kubernetes Service of type LoadBalancer.

Then add the monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter: http://your-management-node:8080/
  4. Set Expected HTTP status to 200.
  5. Set Check interval to 2 minutes.
  6. Click Save.

Step 2: Monitor Ceph Cluster Health via MGR Dashboard API

The Ceph MGR dashboard exposes a REST API at /api/health/full that returns the complete cluster health status including OSD map, placement group health, and monitor quorum.

  1. Enable the MGR dashboard module if not already enabled:
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- \
  ceph mgr module enable dashboard

  1. The dashboard is typically available at https://<mgr-service-ip>:7000. Add a Vigilmon monitor:

    • Click Add Monitor → HTTP / HTTPS.
    • Enter: https://rook-ceph-mgr.yourdomain.com/api/health/full
    • Set Expected HTTP status to 200.
    • Under Request headers, add the MGR dashboard API token: 
      Authorization: Bearer <your-mgr-api-token>
    • Under Response body, set Contains to "status":"HEALTH_OK".
    • Set Check interval to 2 minutes.
    • Click Save.

A healthy Ceph cluster returns:

{"status": "HEALTH_OK", "checks": {}, "mutes": []}

When Ceph enters HEALTH_WARN or HEALTH_ERR, the body no longer contains "HEALTH_OK" and Vigilmon alerts. This gives you early warning before a degraded cluster cascades into unavailability.

Step 3: TCP Port Monitoring for MON and MGR

Ceph MON processes run on TCP port 6789 (or v2 on 3300). If MONs lose quorum, the entire cluster freezes — no reads or writes succeed. Monitor MON TCP availability directly:

  1. Click Add Monitor → TCP Port.
  2. Enter the MON service address: ceph-mon.yourdomain.com
  3. Set Port to 6789.
  4. Set Check interval to 1 minute.
  5. Click Save.

Repeat for each MON node if you expose them individually, or point to the MON service address that load-balances across all MONs.

For the MGR dashboard (port 7000), add another TCP monitor:

  1. Click Add Monitor → TCP Port.
  2. Enter the MGR address: ceph-mgr.yourdomain.com
  3. Set Port to 7000.
  4. Set Check interval to 2 minutes.
  5. Click Save.

TCP-level checks catch network partition and firewall rule changes that the application-level health API won't report until it tries to serve a request.

Step 4: SSL Certificate Alerts for the MGR Dashboard

The Ceph MGR dashboard uses a self-signed certificate by default, which browsers reject. If you've configured a proper TLS certificate (via cert-manager or manual upload), monitor it:

  1. Open the MGR dashboard HTTP monitor from Step 2.
  2. Enable Monitor SSL certificate under the SSL section.
  3. Set Alert when certificate expires in less than 21 days.
  4. Click Save.

To update the MGR dashboard certificate before it expires:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- \
  ceph dashboard set-ssl-certificate -i /path/to/cert.pem
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- \
  ceph dashboard set-ssl-certificate-key -i /path/to/key.pem

Step 5: Heartbeat Monitoring for OSD Availability

OSD processes store the actual data. An OSD going down reduces replication and can cascade into HEALTH_WARN. Add a heartbeat that periodically queries OSD status and pings Vigilmon when all OSDs are in:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 30 minutes.
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123

Create a Kubernetes CronJob that checks OSD health:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: rook-osd-heartbeat
  namespace: rook-ceph
spec:
  schedule: "*/30 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: osd-check
            image: rook/ceph:v1.12.0
            command:
            - /bin/bash
            - -c
            - |
              OSD_STATUS=$(ceph osd stat --format json | jq -r '.num_up_osds')
              TOTAL=$(ceph osd stat --format json | jq -r '.num_osds')
              if [ "$OSD_STATUS" -eq "$TOTAL" ]; then
                curl -s https://vigilmon.online/heartbeat/abc123
              fi
          restartPolicy: OnFailure

Apply it with kubectl apply -f rook-osd-heartbeat.yaml. If any OSD goes down, the heartbeat stops and Vigilmon alerts after 35 minutes.

Step 6: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, PagerDuty, or a webhook.
  2. On the MON TCP monitors, set Consecutive failures to 1 — MON quorum loss is immediately critical.
  3. On the cluster health API monitor, set Consecutive failures to 2 — brief HEALTH_WARN during OSD rebalancing is normal; sustained warnings need attention.
  4. Route Rook alerts to an on-call rotation, not just a Slack channel — storage failures require immediate human intervention.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Operator health | Rook operator pod | Operator crash, reconcile failure | | Cluster health API | /api/health/full | HEALTH_WARN/HEALTH_ERR, OSD degradation | | MON TCP | :6789 | MON quorum loss, network partition | | MGR TCP | :7000 | MGR process crash, dashboard unavailable | | SSL certificate | MGR dashboard hostname | Certificate expiry | | Cron heartbeat | Heartbeat URL | OSD outage, missed health check |

Rook/Ceph failures are among the most severe in a Kubernetes cluster — a storage outage cascades to every stateful workload. Vigilmon's external monitors provide a safety net that keeps watching even when the cluster's internal alerting (Prometheus, Alertmanager) is itself running on the affected storage.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles