How to Monitor Pacemaker HA Cluster Resource Manager with Vigilmon

Pacemaker is the policy engine for Linux high-availability clusters — it decides which resources run where, handles failover, and manages STONITH fencing. Its sophistication is also its opacity: resources can be in Stopped, Failed, unmanaged, or indefinite recovery loops while crm_mon shows a deceptively calm picture.

This tutorial shows you how to build complete observability for Pacemaker using Vigilmon: monitoring cluster resource state, failover events, quorum, STONITH, and the Pacemaker daemon stack.

Why Pacemaker needs external monitoring

Pacemaker's complexity creates monitoring gaps:

• Failed resources not cleaned up — a resource in FAILED state requires manual crm resource cleanup to restart. Pacemaker will not retry indefinitely by default. Services stay down until someone notices.
• Unmanaged resources — administrators sometimes put resources into unmanaged mode for maintenance and forget to re-enable. The resource stays in whatever state it was in, silently.
• Stuck failover — a resource may be mid-migration for minutes during a slow failover. If the receiving node also fails, the resource enters a FAILED state with no automatic recovery.
• STONITH failures — if a STONITH device is unreachable, Pacemaker cannot safely fence a failed node, so it refuses to promote resources on surviving nodes. The cluster appears healthy but no resources run.
• Daemon stack — Pacemaker runs multiple daemons (pacemakerd, crmd/pacemaker-controld, pengine/pacemaker-schedulerd, attrd, stonithd). A crash in any one of them can halt cluster operations.

Monitoring Pacemaker means watching resource state, STONITH availability, and the full daemon stack — not just whether crm_mon returns without error.

What you'll need

• A Pacemaker cluster with Corosync as the messaging layer
• The crm_mon, crm_verify, and pcs or crmsh management tools
• A free Vigilmon account

Step 1: Monitor cluster resource health

Run a probe on each cluster node that checks for failed or stopped resources:

#!/bin/bash
# pacemaker-resource-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_RESOURCE_HEARTBEAT"
LOG="/var/log/pacemaker-probe.log"

# Check Pacemaker is running
if ! systemctl is-active --quiet pacemaker; then
    echo "$(date): FAIL — pacemaker service not active" >> "$LOG"
    exit 1
fi

# Parse crm_mon for failures
crm_output=$(crm_mon --one-shot --output-as=text 2>/dev/null)

# Check for failed resources
if echo "$crm_output" | grep -qiE "FAILED|stopped.*unmanaged|migration_failed"; then
    echo "$(date): FAIL — resource failures detected:" >> "$LOG"
    echo "$crm_output" | grep -iE "FAILED|stopped.*unmanaged|migration_failed" >> "$LOG"
    exit 1
fi

# Check overall cluster status
if echo "$crm_output" | grep -q "Current DC:.*NONE"; then
    echo "$(date): FAIL — no Designated Coordinator (cluster partitioned)" >> "$LOG"
    exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

Make executable and add to crontab on every node:

chmod +x /opt/scripts/pacemaker-resource-probe.sh

* * * * * /opt/scripts/pacemaker-resource-probe.sh

In Vigilmon:

1. Monitors → New Monitor → Heartbeat
2. Name: Pacemaker Resources — node1 (one per node)
3. Interval: 2 minutes

Step 2: Monitor the services managed by Pacemaker

The cleanest way to monitor HA-managed services is to check the services themselves, not just Pacemaker's view of them. Add HTTP or TCP monitors for each cluster resource's actual endpoint:

Example: clustered PostgreSQL via VIP

1. Monitors → New Monitor → TCP Port
2. Hostname: YOUR_CLUSTER_VIP
3. Port: 5432
4. Name: Cluster PostgreSQL

Example: clustered NGINX via VIP

1. Monitors → New Monitor → HTTP
2. URL: http://YOUR_CLUSTER_VIP/health
3. Expected status: 200
4. Name: Cluster NGINX

These monitors catch the case where Pacemaker thinks a resource is running but the service inside the resource agent is actually broken.

Step 3: Monitor STONITH / fencing device availability

Without STONITH, Pacemaker will not promote resources to surviving nodes after a failure. Check STONITH is operational:

#!/bin/bash
# pacemaker-stonith-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_STONITH_HEARTBEAT"
LOG="/var/log/pacemaker-stonith.log"

# Check STONITH is enabled in the cluster
stonith_enabled=$(crm configure show | grep -oP '(?<=stonith-enabled=)\w+')
if [ "$stonith_enabled" = "false" ]; then
    echo "$(date): WARN — STONITH is disabled; cluster cannot safely fence nodes" >> "$LOG"
    # This is a config issue — decide if you want to alert
fi

# Check STONITH devices are reachable using pcs
# (adjust the STONITH resource name to match yours)
STONITH_RESOURCE="fence_device1"
stonith_status=$(crm_mon --one-shot --output-as=text 2>/dev/null | grep "$STONITH_RESOURCE")

if echo "$stonith_status" | grep -qiE "FAILED|stopped|offline"; then
    echo "$(date): FAIL — STONITH device $STONITH_RESOURCE is $stonith_status" >> "$LOG"
    exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

*/5 * * * * /opt/scripts/pacemaker-stonith-probe.sh

In Vigilmon:

1. Monitors → New Monitor → Heartbeat
2. Name: Pacemaker STONITH
3. Interval: 10 minutes

Step 4: Monitor the Pacemaker daemon stack

Pacemaker runs several cooperating daemons. Monitor each key one:

#!/bin/bash
# pacemaker-daemon-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_DAEMON_HEARTBEAT"
LOG="/var/log/pacemaker-daemon-probe.log"

# Daemons that must be active for the cluster to function
REQUIRED_DAEMONS=("corosync" "pacemaker")

all_ok=true

for daemon in "${REQUIRED_DAEMONS[@]}"; do
    if ! systemctl is-active --quiet "$daemon"; then
        echo "$(date): FAIL — $daemon is not active" >> "$LOG"
        all_ok=false
    fi
done

# Also check sub-components via pacemakerd
sub_status=$(systemctl status pacemaker 2>/dev/null | grep -E "Active:|Sub-state:")
echo "$(date): $sub_status" >> "$LOG"

if $all_ok; then
    curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2
fi

* * * * * /opt/scripts/pacemaker-daemon-probe.sh

In Vigilmon:

1. Monitors → New Monitor → Heartbeat
2. Name: Pacemaker Daemon Stack
3. Interval: 2 minutes

Step 5: Monitor failover events and resource migrations

Pacemaker logs failovers to the system journal. Detect recent failover events to alert on unexpected HA activity:

#!/bin/bash
# pacemaker-failover-probe.sh
# This does NOT block heartbeat on failovers — it logs them for awareness

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_FAILOVER_AUDIT_HEARTBEAT"
LOG="/var/log/pacemaker-failovers.log"
STATE_FILE="/var/run/pacemaker-failover-count"

# Count failover-related log entries in the last 10 minutes
recent_failovers=$(journalctl -u pacemaker --since "10 minutes ago" 2>/dev/null \
    | grep -cE "ResourceAgent|stonith|fencing|migrate_to|migrate_from")

previous=$(cat "$STATE_FILE" 2>/dev/null || echo "0")
echo "$recent_failovers" > "$STATE_FILE"

if [ "$recent_failovers" -gt 0 ]; then
    echo "$(date): Cluster activity: $recent_failovers events in last 10 min (prev: $previous)" >> "$LOG"
fi

# Still ping the heartbeat — this monitor is for audit, not alerting on activity
curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

Pair this with a separate heartbeat that watches for a runaway failover loop:

#!/bin/bash
# pacemaker-failover-loop-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_FAILOVER_LOOP_HEARTBEAT"
LOG="/var/log/pacemaker-probe.log"

# If a resource has failed more than N times in the migration threshold, it's looping
loop_resources=$(crm_mon --one-shot --output-as=text 2>/dev/null \
    | grep -E "migration-threshold" | wc -l)

# A simpler check: look for resources at their fail count limit
crm_output=$(crm_mon --one-shot --output-as=text 2>/dev/null)
if echo "$crm_output" | grep -qE "fail-count=[0-9]{2,}"; then
    echo "$(date): WARN — resource with high fail count detected" >> "$LOG"
    echo "$crm_output" | grep "fail-count" >> "$LOG"
    exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

Step 6: Expose a Pacemaker status endpoint

Run a lightweight HTTP server on each node for Vigilmon HTTP polling:

#!/usr/bin/env python3
# pacemaker-status-server.py

import subprocess
import json
from http.server import HTTPServer, BaseHTTPRequestHandler

def get_pacemaker_status():
    try:
        svc = subprocess.run(["systemctl", "is-active", "pacemaker"],
                             capture_output=True, text=True, timeout=5)
        if svc.stdout.strip() != "active":
            return {"status": "down", "reason": "pacemaker not active"}

        crm = subprocess.run(["crm_mon", "--one-shot", "--output-as=text"],
                              capture_output=True, text=True, timeout=15)

        if "FAILED" in crm.stdout:
            failed = [l.strip() for l in crm.stdout.splitlines() if "FAILED" in l]
            return {"status": "degraded", "failed_resources": failed}

        if "Current DC:         NONE" in crm.stdout:
            return {"status": "degraded", "reason": "no DC — cluster partitioned"}

        return {"status": "ok"}
    except Exception as e:
        return {"status": "error", "reason": str(e)}

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            status = get_pacemaker_status()
            code = 200 if status["status"] == "ok" else 503
            body = json.dumps(status).encode()
            self.send_response(code)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(body)
        else:
            self.send_response(404)
            self.end_headers()
    def log_message(self, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("0.0.0.0", 5408), Handler).serve_forever()

Add an HTTP monitor in Vigilmon for each node:

• URL: http://NODE_IP:5408/health
• Expected status: 200
• Name: Pacemaker Health — node1

Step 7: Configure alerting

Route Pacemaker alerts to the right people at the right urgency:

1. Alert Channels → Add Channel → PagerDuty — for failed resources and STONITH failures (direct production impact)
2. Add Channel → Slack — #cluster-alerts for daemon health and quorum issues
3. Add Channel → Email — cluster administrators for failover audit events

Set 0-minute escalation on resource failures — a FAILED Pacemaker resource means a service is down and automatic recovery has already given up.

Step 8: Build a cluster status page

1. Status Pages → New Status Page
2. Name: HA Cluster
3. Add monitors: Cluster Resources, Cluster VIP services, STONITH, Daemon Stack
4. Set visibility to internal
5. Share with on-call engineers

Complete monitoring reference

| Monitor | Type | What it catches | |---------|------|-----------------| | Pacemaker Resources (per node) | Heartbeat | Failed resources, no DC, unmanaged | | Cluster VIP services | HTTP/TCP | Service down despite Pacemaker running | | Pacemaker STONITH | Heartbeat | Fencing device unreachable | | Pacemaker Daemon Stack | Heartbeat | Daemon crash | | Pacemaker Health HTTP | HTTP | Aggregated status endpoint | | Failover Loop probe | Heartbeat | Resource in unrecoverable fail loop |

Troubleshooting common failures

• Resource shows FAILED — run crm resource cleanup <resource-name> to clear failure count and allow Pacemaker to retry. If it fails again immediately, check the resource agent logs: journalctl -u pacemaker | grep <resource-name>.
• No DC (Designated Coordinator) — the cluster has lost Corosync quorum. Check network connectivity between nodes and Corosync ring status with corosync-cfgtool -s.
• STONITH device unreachable — run crm_mon -1 and look for STONITH resource status. Check fence agent connectivity manually: fence_<type> -a <ipmi-ip> -l admin -p password -o status. Without STONITH, Pacemaker will not start resources on surviving nodes after a node failure.
• Resource stuck in Stopping — the resource agent's stop operation timed out. This can happen when the service is unresponsive. Check dmesg and journalctl for kernel or OOM events. If the resource will never stop cleanly, fencing the stuck node is the standard resolution.
• Cluster maintenance mode left on — crm configure show | grep maintenance shows maintenance-mode=true. Re-enable cluster operations with crm configure property maintenance-mode=false or pcs property set maintenance-mode=false.

Next steps

• Corosync integration — pair this with the Corosync monitoring tutorial to cover the full cluster stack from messaging layer to resource management
• DRBD monitoring — if your cluster uses DRBD for replicated block storage, monitor DRBD sync state to catch split-brain before Pacemaker has to choose

Start monitoring your Pacemaker HA cluster today at vigilmon.online — free, no credit card.