How to Monitor Corosync Cluster Messaging with Vigilmon

Corosync is the cluster communication layer underneath most Linux HA stacks — it provides the reliable messaging and membership that Pacemaker, DRBD, and other cluster software depend on. When a Corosync ring breaks or a node is fenced, the cluster can enter a partial quorum state that silently degrades cluster services without triggering an obvious error.

This tutorial shows you how to build complete observability for Corosync using Vigilmon: monitoring ring health, quorum state, node membership, and the Corosync daemon itself.

Why Corosync needs external monitoring

Corosync's reliability is its operational blind spot:

Ring faults — Corosync supports redundant rings (totem ring). A fault on ring 0 causes ring 1 to take over silently. The cluster continues running, but now has no redundancy in its communication layer — a second fault will partition the cluster.
Quorum loss — if a node leaves unexpectedly and the cluster no longer has quorum, resource agents may stop but Corosync itself keeps running, looking healthy at the process level.
Node fencing — STONITH fencing events are logged but not inherently visible to external systems. An unrecognized fencing loop can destroy availability silently.
Split-brain risk — without quorum, some cluster stacks default to stopping all resources rather than risk split-brain writes to shared storage. If quorum is never restored, resources stay stopped indefinitely.
Totem timeouts — high latency between nodes causes message timeouts in the Totem protocol, leading to node expulsion without a hard failure.

Monitoring Corosync means watching quorum, membership, ring status, and token throughput — not just whether corosync is in the process list.

What you'll need

A Corosync 2.x or 3.x cluster with at least 2 nodes
The corosync-quorumtool and corosync-cfgtool utilities (bundled with Corosync)
A free Vigilmon account

Step 1: Monitor quorum state on every cluster node

Quorum is the most critical Corosync metric. Run this probe on each node:

#!/bin/bash
# corosync-quorum-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_QUORUM_HEARTBEAT"
LOG="/var/log/corosync-probe.log"

# Check Corosync is running
if ! systemctl is-active --quiet corosync; then
    echo "$(date): FAIL — corosync service not active" >> "$LOG"
    exit 1
fi

# Check quorum status
quorum_output=$(corosync-quorumtool -p 2>/dev/null)
quorum_status=$(echo "$quorum_output" | grep -oP '(?<=Quorate:)\s*\K\w+')

if [ "$quorum_status" != "Yes" ]; then
    echo "$(date): FAIL — cluster not quorate: $quorum_status" >> "$LOG"
    echo "$quorum_output" >> "$LOG"
    exit 1
fi

# Check all expected nodes are online
node_count=$(echo "$quorum_output" | grep -c "^[[:space:]]*[0-9]")
expected_nodes=3  # set to your cluster size

if [ "$node_count" -lt "$expected_nodes" ]; then
    echo "$(date): WARN — only $node_count of $expected_nodes nodes in membership" >> "$LOG"
    # You may want to alert on partial membership — uncomment to block heartbeat:
    # exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

Add to crontab on each node:

* * * * * /opt/scripts/corosync-quorum-probe.sh

In Vigilmon:

Monitors → New Monitor → Heartbeat
Name: Corosync Quorum — node1 (repeat for each node)
Interval: 2 minutes

Run this on every node independently — if a single node loses quorum, only that node's heartbeat stops. This tells you which side of a partition lost quorum.

Step 2: Monitor ring (totem ring) health

Corosync's redundant rings provide fault tolerance for the messaging layer. A ring fault is survivable but removes redundancy:

#!/bin/bash
# corosync-ring-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_RING_HEARTBEAT"
LOG="/var/log/corosync-ring-probe.log"

# Check ring status via cfgtool
ring_output=$(corosync-cfgtool -s 2>/dev/null)

# Look for any ring that shows "FAULTY" or "not connected"
if echo "$ring_output" | grep -qiE "FAULTY|not connected|error"; then
    echo "$(date): FAIL — ring fault detected:" >> "$LOG"
    echo "$ring_output" >> "$LOG"
    exit 1
fi

echo "$(date): All rings healthy" >> "$LOG"
curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

*/2 * * * * /opt/scripts/corosync-ring-probe.sh

In Vigilmon:

Monitors → New Monitor → Heartbeat
Name: Corosync Ring Health
Interval: 5 minutes

Step 3: Monitor cluster node membership changes

Node departures are noteworthy events. Track membership over time by logging changes:

#!/bin/bash
# corosync-membership-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_MEMBERSHIP_HEARTBEAT"
LOG="/var/log/corosync-membership.log"
STATE_FILE="/var/run/corosync-member-count"

# Get current member count
current=$(corosync-quorumtool -p 2>/dev/null | grep -c "^[[:space:]]*[0-9]")
expected=3  # your expected cluster size

# Load previous count
previous=$(cat "$STATE_FILE" 2>/dev/null || echo "$expected")
echo "$current" > "$STATE_FILE"

if [ "$current" -ne "$previous" ]; then
    echo "$(date): Membership changed: $previous → $current nodes" >> "$LOG"
fi

if [ "$current" -ge "$expected" ]; then
    curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2
else
    echo "$(date): WARN — cluster at $current/$expected nodes; not pinging heartbeat" >> "$LOG"
fi

*/5 * * * * /opt/scripts/corosync-membership-probe.sh

Step 4: Expose a Corosync health endpoint for HTTP monitoring

Run a lightweight HTTP health server on each node so Vigilmon can poll it directly:

#!/usr/bin/env python3
# corosync-status-server.py

import subprocess
import json
from http.server import HTTPServer, BaseHTTPRequestHandler

def get_corosync_status():
    try:
        svc = subprocess.run(["systemctl", "is-active", "corosync"],
                             capture_output=True, text=True, timeout=5)
        if svc.stdout.strip() != "active":
            return {"status": "down", "reason": "service not active"}

        quorum = subprocess.run(["corosync-quorumtool", "-p"],
                                 capture_output=True, text=True, timeout=10)
        if "Quorate:  Yes" not in quorum.stdout:
            return {"status": "degraded", "reason": "cluster not quorate"}

        rings = subprocess.run(["corosync-cfgtool", "-s"],
                                capture_output=True, text=True, timeout=10)
        if "FAULTY" in rings.stdout or "not connected" in rings.stdout.lower():
            return {"status": "degraded", "reason": "ring fault"}

        return {"status": "ok"}
    except Exception as e:
        return {"status": "error", "reason": str(e)}

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            status = get_corosync_status()
            code = 200 if status["status"] == "ok" else 503
            body = json.dumps(status).encode()
            self.send_response(code)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(body)
        else:
            self.send_response(404)
            self.end_headers()
    def log_message(self, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("0.0.0.0", 5407), Handler).serve_forever()

Add an HTTP monitor in Vigilmon for each node:

URL: http://NODE_IP:5407/health
Expected status: 200
Name: Corosync Health — node1

Step 5: Monitor the Corosync daemon and token timeouts

Token timeouts in /var/log/cluster/corosync.log precede node expulsions. Add a log-based probe:

#!/bin/bash
# corosync-token-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_TOKEN_HEARTBEAT"
LOG="/var/log/corosync-probe.log"

# Check for token timeout messages in the last 5 minutes
recent_timeouts=$(journalctl -u corosync --since "5 minutes ago" 2>/dev/null | grep -c "totem_timeout")

if [ "$recent_timeouts" -gt 2 ]; then
    echo "$(date): WARN — $recent_timeouts Totem token timeouts in last 5 min" >> "$LOG"
    # Don't ping heartbeat — alert on missed heartbeat
    exit 1
fi

# Also check daemon is running
if systemctl is-active --quiet corosync; then
    curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2
fi

*/5 * * * * /opt/scripts/corosync-token-probe.sh

Step 6: Configure alerting

Route Corosync alerts to your cluster operations team:

Alert Channels → Add Channel → PagerDuty — for quorum loss (production-impacting immediately)
Add Channel → Slack — #cluster-alerts for ring faults and node departures
Add Channel → Email — all cluster operators

Set 0-minute escalation on quorum loss — when a cluster loses quorum, all clustered resources may stop within seconds.

Step 7: Build a cluster infrastructure status page

Status Pages → New Status Page
Name: Cluster Infrastructure
Add monitors: Quorum (all nodes), Ring Health, Membership, Daemon
Visibility: internal
Share with your on-call rotation

Complete monitoring reference

| Monitor | Type | What it catches | |---------|------|-----------------| | Corosync Quorum (per node) | Heartbeat | Quorum loss, node partition | | Ring Health | Heartbeat | Ring fault, reduced redundancy | | Node Membership | Heartbeat | Node departure below threshold | | Corosync Health HTTP | HTTP | Aggregated status per node | | Token Timeout probe | Heartbeat | Network instability, pre-expulsion events |

Troubleshooting common failures

Cluster not quorate — run corosync-quorumtool -v on each node. The node with a lower node count is the partitioned side. Restore network connectivity between nodes; the minority partition will automatically rejoin.
Ring FAULTY on corosync-cfgtool -s — check the network interface for that ring: ip link show <ring-interface>. A flapping NIC or misconfigured IP can cause ring failures. Run corosync-cfgtool -r after fixing the interface to reset the ring.
Repeated Totem timeouts in logs — packet loss or high latency between cluster nodes. Check ping -c 100 <peer-node-ip> for packet loss; if present, investigate MTU mismatches or switch issues.
Node expelled from cluster — check corosync.log for the expulsion message and timestamp. Typical causes: network glitch, overloaded node unable to process Totem tokens. Check system load and disk I/O on the expelled node around the incident time.

Next steps

Pacemaker integration — Corosync is Pacemaker's transport layer; add Pacemaker monitoring (separate tutorial) to get a full HA stack picture
Cluster-wide dashboard — combine Corosync, Pacemaker, and DRBD monitors on a single Vigilmon status page for your full HA cluster overview

Start monitoring your Corosync cluster messaging today at vigilmon.online — free, no credit card.