How to Monitor GlusterFS with Vigilmon (Brick Health, Split-Brain + Alerts)

GlusterFS gives you a distributed filesystem without a central metadata server — elastic scaling across commodity nodes with replication baked in. The tradeoff is observability: there's no single health endpoint, no built-in Prometheus exporter in older releases, and split-brain conditions can persist silently for days before a write fails in production.

This tutorial covers internal GlusterFS monitoring with gluster CLI and community exporters, plus Vigilmon external uptime checks that verify your volume health API and NFS/SMB gateways are reachable from outside the storage network.

Why GlusterFS needs external monitoring

GlusterFS's peer mesh architecture means every node is both a data plane and a control plane participant. When any brick goes down the cluster continues serving reads and writes through its replicas — silently degraded. External monitoring catches what the cluster itself can't report:

Brick process crash — glusterd remains running; the volume appears healthy; but one replica of every file on that brick is stale until the brick is healed
Split-brain accumulation — a network partition causes multiple bricks to accept conflicting writes; the split-brain count grows while the volume stays mounted
NFS-Ganesha or Samba gateway outage — the NFS or SMB export process crashes; clients get I/O errors but internal monitoring reports the cluster healthy
Peer disconnect — a storage node loses connectivity to the peer pool; new file distribution skips the disconnected node silently
Gluster management API unreachable — glusterd gets stuck; gluster volume status hangs; external checks catch the API freeze immediately

Vigilmon gives you a ground-truth check on GlusterFS reachability from the same network position as your application servers.

What you'll need

GlusterFS cluster (v10 or later recommended) with at least 2 nodes in replica set
SSH or CLI access to at least one storage node
gluster CLI with admin access
A free Vigilmon account — no credit card required

Step 1: Check baseline volume and peer health

Start with the core health commands:

# Peer connectivity
gluster peer status
# Number of Peers: 2
# Hostname: storage2.internal
# Uuid: b3e7...
# State: Peer in Cluster (Connected)

# Volume status
gluster volume status gfsvol detail
# Gluster process             TCP Port  RDMA Port  Online  Pid
# ------------------------------------------------------------------------------
# Brick storage1:/data/brick   49152     0          Y       12345
# Brick storage2:/data/brick   49152     0          Y       12346

# Check for split-brain
gluster volume heal gfsvol info split-brain
# Number of entries in split-brain: 0

# Heal status
gluster volume heal gfsvol info
# Brick storage1:/data/brick
# Status: Connected
# Number of entries: 0

Create a monitoring script that parses these outputs:

#!/bin/bash
# /usr/local/bin/gluster-health-check.sh

VOLUME="${1:-gfsvol}"

# Count offline bricks
OFFLINE_BRICKS=$(gluster volume status "$VOLUME" | grep -c "N$")

# Count split-brain files
SPLIT_BRAIN=$(gluster volume heal "$VOLUME" info split-brain 2>/dev/null \
  | grep "Number of entries in split-brain:" \
  | awk '{print $NF}')

# Count disconnected peers
DISCONNECTED_PEERS=$(gluster peer status \
  | grep -c "State: Peer Rejected\|State: Peer in Cluster (Disconnected)")

echo "volume=$VOLUME offline_bricks=$OFFLINE_BRICKS split_brain=$SPLIT_BRAIN disconnected_peers=$DISCONNECTED_PEERS"

if [ "$OFFLINE_BRICKS" -gt 0 ] || [ "${SPLIT_BRAIN:-0}" -gt 0 ] || [ "$DISCONNECTED_PEERS" -gt 0 ]; then
  exit 1
fi

Schedule the script via cron and push failures to Vigilmon or your alerting system:

*/2 * * * * /usr/local/bin/gluster-health-check.sh gfsvol || \
  curl -s "$ALERT_WEBHOOK" -d "GlusterFS health check failed on $(hostname)"

Step 2: Deploy the Prometheus GlusterFS exporter

The community gluster_exporter (from the gluster project) exposes brick, volume, and peer metrics in Prometheus format:

# Install the exporter
wget https://github.com/gluster/gluster-prometheus/releases/latest/download/gluster_exporter
chmod +x gluster_exporter
mv gluster_exporter /usr/local/bin/

# Run as a systemd service
cat > /etc/systemd/system/gluster-exporter.service << 'EOF'
[Unit]
Description=Gluster Prometheus Exporter
After=network.target glusterd.service

[Service]
ExecStart=/usr/local/bin/gluster_exporter --port 9713
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now gluster-exporter

Verify metrics are flowing:

curl -s http://localhost:9713/metrics | grep gluster_
# gluster_brick_status{brick="storage1:/data/brick",hostname="storage1",volume="gfsvol"} 1
# gluster_volume_status{volume="gfsvol"} 1
# gluster_peers_connected 2

Key Prometheus alert rules:

- alert: GlusterBrickDown
  expr: gluster_brick_status == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "GlusterFS brick {{ $labels.brick }} is offline on {{ $labels.hostname }}"

- alert: GlusterSplitBrain
  expr: gluster_heal_info_split_brain_count > 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "GlusterFS volume {{ $labels.volume }} has {{ $value }} split-brain files"

- alert: GlusterPeerDisconnected
  expr: gluster_peers_connected < 2
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "GlusterFS peer pool has {{ $value }} connected peers (expected ≥ 2)"

- alert: GlusterHealBacklogHigh
  expr: gluster_heal_info_entries > 1000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "GlusterFS heal backlog is {{ $value }} entries on {{ $labels.volume }}"

Step 3: Expose an HTTP health endpoint

GlusterFS doesn't have a built-in HTTP health API, but you can run a lightweight wrapper:

#!/usr/bin/env python3
# /usr/local/bin/gluster-health-api.py
import subprocess
from http.server import HTTPServer, BaseHTTPRequestHandler

def check_health():
    try:
        result = subprocess.run(
            ["gluster", "volume", "status", "gfsvol"],
            capture_output=True, text=True, timeout=10
        )
        if result.returncode != 0:
            return False, "volume status check failed"
        offline = result.stdout.count("N\n")
        if offline > 0:
            return False, f"{offline} bricks offline"
        return True, "OK"
    except Exception as e:
        return False, str(e)

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            ok, msg = check_health()
            status = 200 if ok else 503
            body = msg.encode()
            self.send_response(status)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(body)
        else:
            self.send_response(404)
            self.end_headers()
    def log_message(self, *args):
        pass

HTTPServer(("0.0.0.0", 9714), HealthHandler).serve_forever()

# Run as a systemd service
cat > /etc/systemd/system/gluster-health-api.service << 'EOF'
[Unit]
Description=GlusterFS Health API
After=glusterd.service

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/gluster-health-api.py
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl enable --now gluster-health-api

Test:

curl http://localhost:9714/health
# OK

Step 4: Add GlusterFS checks in Vigilmon

Health API (per node):

Click Add Monitor → HTTP(S)
URL: http://<STORAGE_NODE_IP>:9714/health
Check interval: 60 seconds
Expected keyword: OK
Configure alert channels

Prometheus exporter (per node):

URL: http://<STORAGE_NODE_IP>:9713/metrics
Expected keyword: gluster_brick_status

If you expose GlusterFS volumes via NFS-Ganesha, add an NFS health check too:

# Test NFS export reachability
showmount -e <STORAGE_NODE_IP>

Add a TCP monitor in Vigilmon targeting port 2049 on each NFS gateway node.

Recommended monitoring checklist

| Signal | Source | Threshold | |--------|--------|-----------| | Brick online status | Gluster Prometheus | Any brick offline | | Split-brain file count | Gluster Prometheus / CLI | > 0 | | Peer connectivity | Gluster Prometheus | < expected peer count | | Heal backlog | Gluster Prometheus | > 1,000 entries | | Health API reachability | Vigilmon (external) | Any downtime | | NFS gateway TCP port | Vigilmon (external) | Any downtime |

Conclusion

GlusterFS's elastic replication makes it resilient to individual brick failures — but that same resilience can mask degradation for hours. Combining gluster CLI health checks and Prometheus metrics with Vigilmon's external uptime monitoring gives you full-spectrum visibility: catch split-brain conditions before they cause data loss, detect peer disconnections before they cascade into write failures, and confirm your NFS and HTTP gateways are reachable from outside the storage network.

Set up your first GlusterFS monitor at vigilmon.online in under five minutes.