tutorial

How to Monitor Rook-Ceph with Vigilmon (Storage Cluster Health + Alerts)

Running Rook-Ceph in your Kubernetes cluster gives you cloud-native, self-healing block storage, object storage, and shared filesystems — without depending o...

Running Rook-Ceph in your Kubernetes cluster gives you cloud-native, self-healing block storage, object storage, and shared filesystems — without depending on a managed cloud provider. But Ceph's distributed architecture introduces failure modes that kubectl get pods simply cannot surface: degraded OSDs, split-brain monitors, placement group inconsistencies, and RADOS gateway outages that silently corrupt writes.

In this tutorial you'll set up external uptime monitoring for your Rook-Ceph storage cluster using Vigilmon — catching failures at the HTTP and TCP layer before they cascade into data unavailability.

Why Rook-Ceph needs external monitoring

Rook-Ceph exposes a rich set of internal metrics through Prometheus, and the Ceph dashboard gives you a real-time view of cluster health. But internal tooling has a fundamental blind spot: it only works when the control plane is reachable.

External monitoring catches what internal tooling misses:

  • Rook operator crashes — the operator pod exits; Kubernetes eventually restarts it, but OSD and MON reconciliation stops for minutes or longer
  • Ceph MGR dashboard goes unreachable — the active Manager process restarts mid-failover; the dashboard URL returns 503 while the cluster appears healthy to internal Prometheus
  • RADOS Gateway (RGW) outage — object storage API returns 500 for all PUT/GET requests; S3-compatible workloads silently fail
  • MON quorum loss — if two of three MON pods crash simultaneously, the cluster enters a read-only state; external checks immediately reflect this, internal tooling hangs
  • Ingress to dashboard breaks — a Traefik or Nginx Ingress update breaks routing to the Ceph dashboard; you lose visibility right when you need it

External monitoring from Vigilmon confirms reachability from outside the cluster — the view your applications and users actually have.

What you'll need

  • A Kubernetes cluster running Rook-Ceph (v1.12 or later recommended)
  • Ceph dashboard accessible via a LoadBalancer or Ingress (port 8443 by default)
  • RADOS Gateway (RGW) deployed if you use object storage
  • A free Vigilmon account — no credit card required

Step 1: Verify your Rook-Ceph endpoints

Before adding monitors, confirm which endpoints are externally reachable.

Ceph dashboard:

# Find the dashboard service
kubectl get svc -n rook-ceph | grep dashboard

# Example output:
# rook-ceph-mgr-dashboard   LoadBalancer   10.96.50.10   203.0.113.10   8443:31443/TCP   3d

Test the dashboard is responding:

curl -k https://203.0.113.10:8443/
# Should return the Ceph dashboard login page HTML

RADOS Gateway (RGW) health endpoint:

kubectl get svc -n rook-ceph | grep rgw

# rook-ceph-rgw-my-store   LoadBalancer   10.96.50.20   203.0.113.20   80:32080/TCP   3d

RGW exposes a health check at the root path — a GET / returns a simple XML response:

curl http://203.0.113.20/
# <?xml version="1.0" encoding="UTF-8"?>
# <ListAllMyBucketsResult>...

Rook operator webhook (if exposed):

kubectl get svc -n rook-ceph | grep webhook

Step 2: Monitor the Ceph dashboard with Vigilmon

The Ceph dashboard is the primary health signal for your storage cluster. If it's unreachable, the MGR process has failed or the network path is broken.

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to your dashboard endpoint, e.g. https://203.0.113.10:8443/
  4. If your dashboard uses a self-signed certificate, enable Allow invalid SSL (or install a valid cert via cert-manager)
  5. Set the check interval to 1 minute
  6. Under Expected response, set:
    • Status code: 200 or 301 (the dashboard redirects unauthenticated requests)
  7. Save the monitor

Vigilmon will probe the dashboard from multiple geographic regions. A timeout or connection refused immediately triggers an incident — even if all Ceph pods show Running in kubectl.

Step 3: Monitor the RADOS Gateway HTTP API

Object storage failures are often silent at the Kubernetes layer. A pod can be Running while the RGW process inside has crashed or is stuck in a restart loop.

  1. Go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to http://your-rgw-endpoint/ (or the S3-compatible endpoint your apps use)
  4. Set the expected status code to 200 or 403 — RGW returns 403 AccessDenied for unauthenticated requests to the root path, which confirms the process is alive
  5. Set check interval to 1 minute
  6. Save the monitor

For a more precise check, create a public bucket and monitor a known object:

# Create a test object using the AWS CLI with your RGW credentials
aws s3 cp /dev/null s3://monitoring-probe/healthcheck \
  --endpoint-url http://203.0.113.20

Then monitor http://203.0.113.20/monitoring-probe/healthcheck for a 200 response — this confirms the full read path works.

Step 4: TCP port checks for MON and OSD availability

Ceph monitors listen on TCP port 6789 (v1 protocol) or ports in the 6800–7300 range (msgr2). A TCP check confirms the MON processes are accepting connections even when the HTTP dashboard is temporarily unavailable.

# Find MON service ports
kubectl get svc -n rook-ceph | grep mon

In Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter the external IP and port of your MON service (or a NodePort if using NodePort services)
  4. Save the monitor

Repeat for each MON if you want per-MON visibility. At minimum, monitor one MON per availability zone.

Step 5: Monitor the Rook operator health endpoint

The Rook operator (v1.11+) exposes a /healthz HTTP endpoint for liveness probing. If you've exposed this externally, add it to Vigilmon:

kubectl get svc -n rook-ceph | grep operator

If the operator service is not externally exposed, set up a NodePort service:

apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-operator-health
  namespace: rook-ceph
spec:
  type: NodePort
  selector:
    app: rook-ceph-operator
  ports:
    - port: 9443
      targetPort: 9443
      nodePort: 30943

Then monitor http://node-ip:30943/healthz in Vigilmon with expected status 200.

Step 6: Configure storage-cluster alerts

Storage failures require immediate response — a degraded cluster that reaches HEALTH_ERR state can halt all writes to PersistentVolumes across your entire Kubernetes workload.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your storage on-call address
  3. Assign the channel to all Rook-Ceph monitors

Webhook alerts for incident escalation

{
  "monitor_name": "Rook-Ceph RGW",
  "status": "down",
  "url": "http://203.0.113.20/",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 60
}

Route this to your on-call rotation via PagerDuty or Opsgenie. For Slack, create a dedicated #storage-alerts channel so storage incidents don't get buried in general notifications.

Correlating Vigilmon alerts with Ceph health

When an alert fires, run these commands to triage:

# Overall cluster health
kubectl exec -n rook-ceph -it $(kubectl get pod -n rook-ceph -l app=rook-ceph-tools -o name) -- ceph status

# OSD status
kubectl exec -n rook-ceph -it $(kubectl get pod -n rook-ceph -l app=rook-ceph-tools -o name) -- ceph osd status

# MON quorum
kubectl exec -n rook-ceph -it $(kubectl get pod -n rook-ceph -l app=rook-ceph-tools -o name) -- ceph mon stat

# Check for HEALTH_WARN or HEALTH_ERR
kubectl exec -n rook-ceph -it $(kubectl get pod -n rook-ceph -l app=rook-ceph-tools -o name) -- ceph health detail

If Vigilmon shows the RGW as down but ceph status shows HEALTH_OK, the issue is likely network routing to the RGW service — not the storage cluster itself.

Recommended monitor set for Rook-Ceph

| Monitor | Type | What it catches | |---------|------|-----------------| | https://ceph-dashboard:8443/ | HTTP | MGR failure, dashboard ingress routing | | http://rgw-endpoint/ | HTTP | RGW process crash, object storage unavailability | | http://rgw-endpoint/monitoring-probe/healthcheck | HTTP | Full read path validation | | mon-node:6789 | TCP | MON process crash, quorum at risk | | http://operator:30943/healthz | HTTP | Rook operator reconciliation stopped |

What's next

  • Heartbeat monitors — if you run Rook-Ceph backup CronJobs, use Vigilmon's heartbeat monitoring to alert when scheduled backups stop completing
  • SSL certificate monitoring — if your dashboard uses a cert-manager-issued certificate, Vigilmon will alert you before the cert expires and breaks dashboard access
  • Status page — create a Vigilmon status page for your storage infrastructure so platform teams have visibility into Ceph health without needing kubectl access

Get started free at vigilmon.online — no credit card, monitors running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles