Monitoring CubeFS with Vigilmon

CubeFS is a CNCF-hosted distributed storage system that unifies POSIX file system, S3-compatible object storage, and block device storage in a single platform. At large scale — think Kubernetes PersistentVolumes across thousands of nodes — any unchecked failure in the Master cluster, a MetaNode partition, or a DataNode disk can cascade into data unavailability. Vigilmon gives you a centralized uptime and alerting layer across all CubeFS components so you catch failures before your applications do.

What You'll Set Up

• Master server cluster health monitoring (port 17010)
• MetaNode cluster health and inode usage alerts
• DataNode storage capacity and replication health monitoring
• S3 Object Gateway availability and latency checks
• Raft consensus stability monitoring for Master and MetaNode
• Erasure Coding (EC) subsystem health checks
• Heartbeat monitors for volume mount and FUSE daemon health

Prerequisites

• CubeFS cluster deployed (Master, MetaNode, DataNode, Object Gateway nodes)
• CubeFS Master HTTP API accessible on port 17010
• A free Vigilmon account

Step 1: Monitor the Master Server Cluster

The CubeFS Master manages volume metadata, DataNode/MetaNode registration, and cluster topology. Master leader election runs over Raft — losing quorum or experiencing frequent leader failover causes all storage operations to stall.

Add a Vigilmon HTTP monitor for the Master API:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Master API URL: http://master-node-1:17010/admin/getCluster
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Enable Keyword match and enter "LeaderAddr" to verify the leader is elected.
7. Click Save.

If you run three Master nodes for quorum, add a monitor for each:

http://master-node-1:17010/admin/getCluster
http://master-node-2:17010/admin/getCluster
http://master-node-3:17010/admin/getCluster

Any Master going dark while Raft re-elects a leader will trip the monitor before quorum is lost.

Verify the Raft leader

curl -s http://master-node-1:17010/admin/getCluster | python3 -m json.tool | grep LeaderAddr

If LeaderAddr is empty, the cluster has no elected leader — immediate action required.

Step 2: Monitor MetaNode Cluster Health

MetaNodes hold in-memory metadata for POSIX file system operations. Inode exhaustion on any MetaNode causes file creation failures even when DataNode disk space is available.

Add an HTTP monitor for each MetaNode:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://metanode-1:17210/getPartitions
3. Check interval: 1 minute.
4. Expected status: 200.
5. Click Save.

Query inode usage per MetaNode and alert when any node exceeds 80%:

# Get MetaNode stats from Master API
curl -s "http://master-node-1:17010/metaNode/get?addr=metanode-1:17210" | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)
used = d['data']['UsedRatio']
print(f'MetaNode inode usage: {float(used)*100:.1f}%')
if float(used) > 0.8:
    print('ALERT: inode usage > 80%')
"

Add a Cron Heartbeat in Vigilmon and call it from a cron job that runs this check:

# /etc/cron.d/cubefs-metanode-check
* * * * * root /usr/local/bin/check_cubefs_metanodes.sh

#!/bin/bash
# check_cubefs_metanodes.sh
MASTER="http://master-node-1:17010"
THRESHOLD=0.8
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID"

NODES=$(curl -s "$MASTER/metaNode/list" | python3 -c "import sys,json; [print(n['Addr']) for n in json.load(sys.stdin)['data']['metaNodes']]")

for NODE in $NODES; do
  USAGE=$(curl -s "$MASTER/metaNode/get?addr=$NODE" | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['UsedRatio'])")
  if (( $(echo "$USAGE > $THRESHOLD" | bc -l) )); then
    echo "ALERT: MetaNode $NODE inode usage at $USAGE"
    exit 1
  fi
done

curl -s "$HEARTBEAT_URL" > /dev/null

Step 3: Monitor DataNode Cluster Health

DataNodes store actual data blocks with 3× replication. A DataNode with less than 10% free disk space will refuse new writes and create under-replicated extents.

Add HTTP monitors for each DataNode:

1. Add Monitor → HTTP / HTTPS.
2. URL: http://datanode-1:17310/getDiskStat
3. Check interval: 1 minute.
4. Expected status: 200.

Check disk usage via the Master API:

curl -s "http://master-node-1:17010/dataNode/get?addr=datanode-1:17310" | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
total = d['Total']
used = d['Used']
pct = used / total * 100
print(f'DataNode disk usage: {pct:.1f}%')
if pct > 90:
    print('ALERT: less than 10% free space')
"

Add a cron heartbeat script similar to the MetaNode check, alerting when any DataNode has less than 10% free space. Only ping the Vigilmon heartbeat when all DataNodes pass.

Step 4: Monitor Data Replication Health

CubeFS replicates data 3× across DataNodes. Under-replicated extents indicate a DataNode failure during a write or a network partition.

Query under-replicated extents from the Master:

curl -s "http://master-node-1:17010/admin/getDp?partitionID=1" | \
  python3 -c "
import sys, json
dp = json.load(sys.stdin)['data']
replicas = len(dp['Replicas'])
if replicas < 3:
    print(f'ALERT: partition has only {replicas}/3 replicas')
else:
    print('Replication OK')
"

Set up a Vigilmon cron heartbeat monitor for replication health:

1. Add Monitor → Cron Heartbeat.
2. Expected ping interval: 5 minutes.
3. Copy the heartbeat URL.
4. Add a cron job that checks all data partitions and only pings if all have 3 replicas.

Step 5: Monitor the S3 Object Gateway

The CubeFS Object Gateway provides S3-compatible API access on port 17410. Unavailability blocks all S3 API consumers.

Add an HTTP monitor for each Object Gateway node:

1. Add Monitor → HTTP / HTTPS.
2. URL: http://objectgw-1:17410/ (or your Object Gateway health endpoint).
3. Check interval: 1 minute.
4. Expected status: 200 (or 403 if authentication is required for root).
5. SSL monitoring: enable if TLS-terminated.

Test the Object Gateway with an S3 client:

aws --endpoint-url http://objectgw-1:17410 s3 ls

If this command hangs or returns a connection error, Vigilmon's monitor will already be alerting.

Step 6: Monitor Erasure Coding (EC) Nodes

CubeFS uses an EC subsystem for cold data storage. EC node failures interrupt stripe writes and repair jobs.

Add HTTP monitors for EC nodes via the Master API:

curl -s "http://master-node-1:17010/ecNode/list"

For each EC node returned, add a Vigilmon HTTP monitor targeting the EC node's status endpoint. Enable a cron heartbeat that checks the repair job health:

curl -s "http://master-node-1:17010/ecNode/get?addr=ecnode-1:17510" | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
print('EC node status:', d.get('Status', 'unknown'))
"

Step 7: Monitor Disk I/O Health on DataNodes

Disk saturation on a DataNode causes write latency spikes that propagate to application I/O. Monitor disk await time using iostat:

# Get disk I/O stats for CubeFS DataNode disks
iostat -x 1 1 | awk '/^Device/ {found=1} found && /sd[a-z]/ {
  if ($11 > 100) print "ALERT: High await on " $1 ": " $11 "ms"
}'

Add a Vigilmon cron heartbeat:

1. Add Monitor → Cron Heartbeat.
2. Expected ping interval: 2 minutes.
3. Only ping when all DataNode disks have await < 100ms.

Step 8: Monitor Raft Consensus Health

Both Master and MetaNode clusters use Raft for leader election and log replication. Frequent leader elections indicate network instability or resource pressure.

Track leader election frequency via the Master API:

# Check if the current node is the Raft leader
curl -s http://master-node-1:17010/admin/getCluster | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)['data']
print('Leader:', d.get('LeaderAddr', 'NONE'))
print('Peers:', len(d.get('PeerAddrs', [])))
"

Create a heartbeat that only pings Vigilmon when a stable leader is elected and follower count equals the expected quorum size. If leader becomes NONE, the heartbeat stops and Vigilmon alerts.

Step 9: Configure Alerts in Vigilmon

With monitors in place, configure alert routing:

1. Go to Alert Channels in Vigilmon.
2. Add your preferred channel (Slack, PagerDuty, email, webhook).
3. For each monitor, set alert thresholds:
   • Master API: alert after 1 failed check (quorum loss is immediately critical)
   • MetaNode/DataNode: alert after 2 failed checks (allow transient blips)
   • Object Gateway: alert after 2 failed checks
   • Cron heartbeats: alert when ping is 2× interval late

For the inode usage and disk space cron heartbeats, a missed ping means the threshold check failed — treat this as critical and page immediately.

Step 10: Dashboard and Status Page

Organize your CubeFS monitors into a Vigilmon status group:

1. Go to Status Pages in Vigilmon and create a new page.
2. Add monitor groups:
   • Storage Control Plane: Master API monitors
   • Metadata Tier: MetaNode monitors + inode heartbeat
   • Data Tier: DataNode monitors + replication heartbeat
   • Object Storage: Object Gateway monitors
   • EC Cold Tier: EC node monitors

Share the status page URL with your storage operations team so everyone has a single view of cluster health.

Conclusion

CubeFS spans multiple specialized components — Master Raft quorum, MetaNode inode capacity, DataNode disk space, replication health, and S3 gateway availability — and a silent failure in any layer can surface as application-level I/O errors. With Vigilmon, you get per-component HTTP monitors for live availability, cron heartbeats for threshold checks that CubeFS doesn't expose as HTTP endpoints, and centralized alerting that pages you before data unavailability propagates to production workloads.

Start with the Master API and Object Gateway monitors (they're the fastest to add), then layer in the DataNode disk and MetaNode inode cron heartbeats once your alerting channels are set up.

Sign up for Vigilmon free and add your first CubeFS monitor in under two minutes.