How to Monitor Ceph with Vigilmon (OSD Health, PG States + Alerts)

Ceph is the foundation of serious distributed storage — petabyte-scale object, block, and filesystem workloads running across commodity hardware. Its self-healing design is genuinely resilient, but resilience is not the same as visibility. OSD failures accumulate silently until a placement group goes inactive. Monitor quorum degrades without alerting anyone. Replication lag builds while the cluster reports HEALTH_WARN and your on-call engineer decides it can wait until morning.

This tutorial walks you through a complete Ceph monitoring stack: internal metrics from ceph status, Prometheus alerts for the critical failure modes, and Vigilmon external uptime checks that verify your Ceph dashboard and S3-compatible RADOS Gateway are actually reachable.

Why Ceph needs external monitoring

Ceph exposes rich internal health data through ceph status, the MGR dashboard, and the Prometheus plugin. But all of those depend on at least two MON daemons and an active MGR process being reachable. External monitoring covers what they miss:

MGR failover — the active Manager restarts; the dashboard returns 503 for 30–90 seconds while the cluster itself is fine
RADOS Gateway outage — the RGW process crashes or gets OOM-killed; S3 PUT/GET operations return 500 while internal Prometheus shows no alerts
Network partition — your monitoring host is cut off from the Ceph public network; ceph status hangs, external checks catch the partition first
MON quorum loss — two of three MON daemons fail; the cluster freezes writes; Alertmanager may itself be down if it runs on the same infrastructure
Dashboard TLS certificate expiry — the HTTPS endpoint returns a TLS error; no internal metric tracks this

External monitoring from Vigilmon sees your Ceph cluster from the same vantage point as your applications — outside the storage network.

What you'll need

Ceph cluster (Quincy or later recommended) with at least 3 MONs and 3 OSDs
ceph CLI access with admin keyring
MGR dashboard module enabled
RADOS Gateway (optional but common)
A free Vigilmon account — no credit card required

Step 1: Verify cluster health and expose Prometheus metrics

Check baseline cluster health:

ceph status
# cluster:
#   id:     a7f3e...
#   health: HEALTH_OK
#
# services:
#   mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3
#   mgr: ceph-mgr1(active), standbys: ceph-mgr2
#   osd: 12 OSDs; 12 up, 12 in
#
# data:
#   pools: 4 pools, 128 PGs
#   objects: 1.23M objects, 4.56 TiB

Enable the Prometheus MGR module if not already active:

ceph mgr module enable prometheus
ceph config set mgr mgr/prometheus/port 9283

Verify metrics are flowing:

curl -s http://localhost:9283/metrics | grep ceph_health_status
# ceph_health_status 0  (0=OK, 1=WARN, 2=ERR)

Key metrics to track:

ceph_health_status                    # overall cluster health (0/1/2)
ceph_osd_up                          # per-OSD up status
ceph_osd_in                          # per-OSD in-cluster status
ceph_pg_active                        # active placement groups
ceph_pg_degraded                      # PGs with missing replicas
ceph_pg_undersized                    # PGs below desired replication
ceph_mon_quorum_status               # monitor quorum (1=in quorum)
ceph_pool_stored                      # bytes stored per pool
ceph_pool_max_avail                   # available bytes per pool
ceph_osd_apply_latency_ms            # write latency per OSD

Prometheus alert rules for critical Ceph states:

- alert: CephOSDDown
  expr: ceph_osd_up == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Ceph OSD {{ $labels.ceph_daemon }} is down"

- alert: CephPGDegraded
  expr: ceph_pg_degraded > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "{{ $value }} Ceph placement groups are degraded"

- alert: CephPGUndersize
  expr: ceph_pg_undersized > 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "{{ $value }} Ceph PGs are undersized (replication lag)"

- alert: CephMonQuorumLoss
  expr: ceph_mon_quorum_status == 0
  for: 30s
  labels:
    severity: critical
  annotations:
    summary: "Ceph monitor quorum lost — cluster writes will freeze"

- alert: CephPoolCapacityHigh
  expr: |
    (ceph_pool_stored / (ceph_pool_stored + ceph_pool_max_avail)) > 0.80
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Ceph pool {{ $labels.name }} is above 80% capacity"

Step 2: Enable the MGR dashboard

The Ceph MGR dashboard provides a web UI and REST API you can monitor externally:

ceph mgr module enable dashboard
ceph dashboard create-self-signed-cert
ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
ceph config set mgr mgr/dashboard/server_port 8443
ceph mgr module disable dashboard
ceph mgr module enable dashboard

Set an admin password:

echo "YourSecurePassword" > /tmp/dashboard-password.txt
ceph dashboard ac-user-create admin -i /tmp/dashboard-password.txt administrator
rm /tmp/dashboard-password.txt

Verify the dashboard is reachable:

curl -k https://localhost:8443/
# Returns the Ceph dashboard HTML

Check the REST API health endpoint:

curl -k https://localhost:8443/api/health/minimal
# Returns: {"status": "OK"}

Step 3: Add Ceph checks in Vigilmon

MGR dashboard:

Click Add Monitor → HTTP(S)
URL: https://<CEPH_MGR_IP>:8443/api/health/minimal
Check interval: 60 seconds
Expected keyword: OK
Configure alert channels (Slack, PagerDuty, email, or webhook)

RADOS Gateway (if deployed):

The RGW S3-compatible endpoint exposes a health check:

curl http://<RGW_HOST>:7480/
# Returns 200 with empty body or "<?xml..." for healthy RGW

Add a second Vigilmon monitor:

URL: http://<RGW_HOST>:7480/
Check interval: 60 seconds
Expected HTTP status: 200

Prometheus metrics endpoint:

Add a third monitor for the MGR Prometheus endpoint to catch MGR crashes that take down both dashboard and metrics:

URL: http://<CEPH_MGR_IP>:9283/metrics
Expected keyword: ceph_health_status

Step 4: Monitor OSD and replication health via CLI

For periodic automated checks, wrap ceph status in a script and push results to a webhook:

#!/bin/bash
# /usr/local/bin/ceph-health-check.sh

HEALTH=$(ceph health --format json | jq -r '.status')
DEGRADED_PGS=$(ceph pg stat --format json | jq '.num_pg_by_state[] | select(.name | contains("degraded")) | .num' 2>/dev/null || echo 0)
OSD_DOWN=$(ceph osd stat --format json | jq '.num_osds - .num_up_osds')

if [ "$HEALTH" != "HEALTH_OK" ] || [ "$OSD_DOWN" -gt 0 ] || [ "$DEGRADED_PGS" -gt 0 ]; then
  curl -s -X POST "$VIGILMON_WEBHOOK_URL" \
    -H "Content-Type: application/json" \
    -d "{\"status\": \"$HEALTH\", \"osd_down\": $OSD_DOWN, \"degraded_pgs\": $DEGRADED_PGS}"
fi

Schedule via cron:

*/5 * * * * /usr/local/bin/ceph-health-check.sh

Recommended monitoring checklist

| Signal | Source | Threshold | |--------|--------|-----------| | Cluster health status | Ceph Prometheus | != 0 (OK) | | OSD up count | Ceph Prometheus | < total OSD count | | Degraded PG count | Ceph Prometheus | > 0 for > 5 min | | MON quorum status | Ceph Prometheus | Any quorum loss | | Pool capacity utilization | Ceph Prometheus | > 80% | | MGR dashboard reachability | Vigilmon (external) | Any downtime | | RGW API reachability | Vigilmon (external) | Any downtime | | Prometheus endpoint | Vigilmon (external) | Any downtime |

Conclusion

Ceph's distributed design gives you extraordinary storage resilience — but resilience without observability is just deferred incidents. Internal Prometheus metrics catch degradation at the RADOS layer; Vigilmon external checks confirm that the management plane and application-facing APIs are actually reachable when your team needs them.

Set up your first Ceph monitor at vigilmon.online in under five minutes.