Monitoring LINSTOR and DRBD with Vigilmon

LINSTOR turns DRBD kernel replication into an orchestrated software-defined storage layer, delivering synchronous block replication with sub-millisecond recovery for databases, virtual machines, and Kubernetes PersistentVolumes. When the LINSTOR Controller goes down, all storage provisioning stops. When a DRBD resource enters DUnknown or Inconsistent state, the application using that volume loses its redundancy guarantee. Vigilmon gives you continuous visibility into every layer — Controller REST API, satellite connectivity, DRBD resource state, resync progress, and CSI driver health — so you respond before failures become data loss.

What You'll Set Up

• LINSTOR Controller REST API health monitoring (port 3370)
• DRBD satellite connectivity alerts
• DRBD resource state monitoring (UpToDate vs. degraded)
• Resync progress and estimated completion time tracking
• Storage pool utilization alerts (>80% threshold)
• CSI driver DaemonSet and controller StatefulSet health
• Controller database health heartbeat

Prerequisites

• LINSTOR cluster with Controller and at least two Satellite nodes
• linstor CLI accessible (or direct access to the REST API on port 3370)
• A free Vigilmon account

Step 1: Monitor the LINSTOR Controller REST API

The LINSTOR Controller REST API (port 3370) is the management plane for all storage operations. When it's down, new PVC provisioning fails and the CSI driver cannot fulfill volume requests.

Add an HTTP monitor:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter: http://linstor-controller:3370/v1/nodes
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Enable Keyword match and enter "name" to verify the response contains node data.
7. Click Save.

Verify manually:

curl -s http://linstor-controller:3370/v1/nodes | python3 -m json.tool | head -20

A healthy response returns a JSON array of node objects. An empty array means satellites haven't connected; a non-200 response means the Controller service is down.

Step 2: Monitor DRBD Satellite Connectivity

LINSTOR Satellites run on every storage node and configure DRBD kernel modules. A disconnected satellite means the storage node is either unreachable or the linstor-satellite service has stopped.

Query satellite connection state from the Controller:

linstor node list
# or via REST API:
curl -s http://linstor-controller:3370/v1/nodes | \
  python3 -c "
import sys, json
nodes = json.load(sys.stdin)
for n in nodes:
    state = n.get('connection_status', 'UNKNOWN')
    name = n['name']
    if state != 'ONLINE':
        print(f'ALERT: Satellite {name} is {state}')
    else:
        print(f'OK: {name} ONLINE')
"

Add a Cron Heartbeat in Vigilmon for satellite connectivity:

1. Add Monitor → Cron Heartbeat.
2. Expected ping interval: 2 minutes.
3. Copy the heartbeat URL.
4. Deploy a cron job that pings the heartbeat only when all satellites are ONLINE:

#!/bin/bash
# /usr/local/bin/check_linstor_satellites.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID"
CONTROLLER="http://linstor-controller:3370"

OFFLINE=$(curl -s "$CONTROLLER/v1/nodes" | \
  python3 -c "
import sys, json
nodes = json.load(sys.stdin)
offline = [n['name'] for n in nodes if n.get('connection_status') != 'ONLINE']
print('\n'.join(offline))
")

if [ -n "$OFFLINE" ]; then
  echo "Offline satellites: $OFFLINE"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

# /etc/cron.d/linstor-satellite-check
*/2 * * * * root /usr/local/bin/check_linstor_satellites.sh

Step 3: Monitor DRBD Resource State

LINSTOR resources backed by DRBD must be in UpToDate state on all replicas for full redundancy. Any resource in DUnknown, Inconsistent, or Outdated state indicates a degraded volume.

Check all resource states:

linstor resource list
# or via REST API:
curl -s http://linstor-controller:3370/v1/view/resources | \
  python3 -c "
import sys, json
resources = json.load(sys.stdin)
for r in resources:
    name = r['name']
    node = r['node_name']
    for vol in r.get('volumes', []):
        state = vol.get('state', {}).get('disk_state', 'UNKNOWN')
        if state != 'UpToDate':
            print(f'ALERT: {name} on {node} is {state}')
"

Add a cron heartbeat for resource health:

#!/bin/bash
# check_linstor_resources.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_RESOURCE_HEARTBEAT_ID"
CONTROLLER="http://linstor-controller:3370"

DEGRADED=$(curl -s "$CONTROLLER/v1/view/resources" | \
  python3 -c "
import sys, json
resources = json.load(sys.stdin)
bad = []
for r in resources:
    for vol in r.get('volumes', []):
        state = vol.get('state', {}).get('disk_state', '')
        if state and state != 'UpToDate':
            bad.append(f\"{r['name']}@{r['node_name']}={state}\")
print('\n'.join(bad))
")

if [ -n "$DEGRADED" ]; then
  echo "Degraded resources: $DEGRADED"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Set this cron heartbeat interval to 5 minutes in Vigilmon.

Step 4: Monitor DRBD Resync Progress

After a node recovers from failure, DRBD resyncs data to bring replicas back to UpToDate. A resync taking more than 24 hours suggests insufficient replication bandwidth or an underlying disk problem.

Check resync progress:

# Check all DRBD devices for resync activity
for dev in $(ls /dev/drbd*); do
  drbdadm status "$(drbdadm sh-resource $dev 2>/dev/null)" 2>/dev/null | \
    grep -E "replication|percent"
done

# Or via linstor:
linstor resource list | grep -v UpToDate

From /proc/drbd:

cat /proc/drbd | grep -A5 "cs:SyncTarget\|cs:SyncSource"
# Look for: [=========>..] 65.3% (12345/35678)M
# And: finish: 4:32:18 speed: 22,220 (21,752) K/sec

Add a cron heartbeat that alerts when any resync has been running for more than 24 hours:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_RESYNC_HEARTBEAT_ID"

LONG_RESYNC=$(cat /proc/drbd | grep -E "finish:" | \
  awk '{
    split($2, t, ":");
    hours = t[1];
    if (hours >= 24) print "ALERT: resync ETA > 24h: " $0
  }')

if [ -n "$LONG_RESYNC" ]; then
  echo "$LONG_RESYNC"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 5: Monitor Storage Pool Utilization

LINSTOR manages storage pools on each satellite. When a pool exceeds 80% utilization, provisioning new PVCs may fail or succeed with insufficient room for snapshots and metadata overhead.

Check storage pool usage:

linstor storage-pool list
# or via REST API:
curl -s http://linstor-controller:3370/v1/view/storage-pools | \
  python3 -c "
import sys, json
pools = json.load(sys.stdin)
for p in pools:
    total = p.get('total_capacity', 0)
    free = p.get('free_capacity', 0)
    if total > 0:
        pct = (total - free) / total * 100
        name = p['storage_pool_name']
        node = p['node_name']
        print(f'{node}/{name}: {pct:.1f}% used')
        if pct > 80:
            print(f'  ALERT: exceeds 80% threshold')
"

Add a cron heartbeat that fires every 10 minutes and only pings Vigilmon when all pools are below 80%:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_POOL_HEARTBEAT_ID"
CONTROLLER="http://linstor-controller:3370"

OVER=$(curl -s "$CONTROLLER/v1/view/storage-pools" | \
  python3 -c "
import sys, json
pools = json.load(sys.stdin)
over = []
for p in pools:
    total = p.get('total_capacity', 0)
    free = p.get('free_capacity', 0)
    if total > 0 and (total - free) / total > 0.8:
        over.append(f\"{p['node_name']}/{p['storage_pool_name']}\")
print('\n'.join(over))
")

if [ -n "$OVER" ]; then
  echo "Pools over 80%: $OVER"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 6: Monitor the CSI Driver in Kubernetes

The linstor-csi driver has two components: a DaemonSet (linstor-csi-node) on every node and a controller StatefulSet. If either is unhealthy, PVC provisioning or node attachment fails.

Add HTTP monitors using the Kubernetes API proxy:

# Check CSI node DaemonSet health
kubectl get daemonset linstor-csi-node -n linstor -o jsonpath='{.status.numberReady}/{.status.desiredNumberScheduled}'
# Expected: matching numbers, e.g. 5/5

Add a Vigilmon cron heartbeat for CSI health:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CSI_HEARTBEAT_ID"

# Check CSI controller
CTRL=$(kubectl get statefulset linstor-csi-controller -n linstor \
  -o jsonpath='{.status.readyReplicas}/{.spec.replicas}' 2>/dev/null)
if [ "$CTRL" != "1/1" ]; then
  echo "CSI controller not ready: $CTRL"
  exit 1
fi

# Check CSI node DaemonSet
DESIRED=$(kubectl get daemonset linstor-csi-node -n linstor \
  -o jsonpath='{.status.desiredNumberScheduled}' 2>/dev/null)
READY=$(kubectl get daemonset linstor-csi-node -n linstor \
  -o jsonpath='{.status.numberReady}' 2>/dev/null)
if [ "$READY" != "$DESIRED" ]; then
  echo "CSI node DaemonSet: $READY/$DESIRED ready"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 7: Monitor DRBD Network Replication Health

DRBD replication runs on a dedicated network interface. Packet loss or bandwidth saturation on the replication network causes resync queuing and increased write latency.

Check DRBD connection state and network stats:

# Check connection state for all DRBD resources
cat /proc/drbd | grep "cs:"
# Healthy output: cs:Connected
# Problem states: cs:StandAlone, cs:Disconnecting, cs:WFConnection

# Check replication network throughput
cat /proc/drbd | grep "ns:\|nr:"
# ns = network send (replication bytes sent)
# nr = network receive

Add an HTTP monitor for the DRBD peer connection via a simple status script exposed on a local port:

#!/bin/bash
# /usr/local/bin/drbd_status_server.sh — serve DRBD health over HTTP
while true; do
  STATUS=$(cat /proc/drbd | grep -c "cs:Connected")
  TOTAL=$(cat /proc/drbd | grep -c "^[0-9]")
  if [ "$STATUS" -eq "$TOTAL" ] && [ "$TOTAL" -gt 0 ]; then
    echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nOK: $STATUS/$TOTAL connected" | nc -l -p 9999 -q 1
  else
    echo -e "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\n\r\nDEGRADED: $STATUS/$TOTAL connected" | nc -l -p 9999 -q 1
  fi
done

Then add a Vigilmon HTTP monitor to http://storage-node-1:9999 with expected status 200.

Step 8: Configure Alerts in Vigilmon

1. Go to Alert Channels and add Slack, PagerDuty, or email.
2. Configure per-monitor thresholds:
   • Controller REST API: alert after 1 failed check — any downtime blocks provisioning
   • Satellite connectivity heartbeat: alert when ping is 1× interval late
   • Resource state heartbeat: alert when ping is 1× interval late
   • Storage pool heartbeat: alert when ping is 2× interval late
   • CSI driver heartbeat: alert when ping is 2× interval late
   • Resync heartbeat: alert when ping is 1× interval late (indicates >24h resync)

Step 9: Status Page for Storage Operations

Create a Vigilmon status page for your storage team:

1. Status Pages → New Page.
2. Add monitor groups:
   • Control Plane: LINSTOR Controller REST API
   • Storage Nodes: satellite connectivity heartbeats per node
   • Replication Health: resource state and DRBD network monitors
   • Capacity: storage pool utilization heartbeats
   • Kubernetes Integration: CSI driver health heartbeat

Conclusion

LINSTOR's power — synchronous DRBD replication with Kubernetes CSI integration — comes with operational complexity spread across Controller REST API, per-node satellites, per-resource DRBD states, and storage pool capacity. A silent satellite disconnection or a resource stuck in Inconsistent state won't trip any built-in alert unless you're actively watching. Vigilmon closes that gap: the Controller gets a direct HTTP monitor, and the DRBD-level state machines get cron heartbeats that only ping when everything is truly healthy.

Start with the Controller API monitor and the satellite connectivity heartbeat — those two cover the most common failure modes. Add resource state and pool utilization heartbeats once your alerting channels are configured.

Sign up for Vigilmon free and add your first LINSTOR monitor in under two minutes.