tutorial

Monitoring Ganeti with Vigilmon

Ganeti clusters your KVM and Xen VMs across physical nodes — but RAPI failures, node daemon dropouts, DRBD replication splits, and watcher daemon crashes all fail silently. Here's how to monitor every Ganeti component with Vigilmon.

Ganeti manages KVM and Xen virtual machines across a cluster of physical nodes, providing automated failover, DRBD disk replication, and cluster-wide job coordination. Its distributed model is resilient — until it isn't. A silent DRBD split-brain, a node daemon that stops responding, or a watcher that misses its window can leave instances in inconsistent states without any visible alert. Vigilmon gives you continuous coverage of every Ganeti component so cluster problems surface before they become VM incidents.

What You'll Set Up

  • RAPI service availability monitor on port 5080
  • Master node daemon health heartbeat
  • Node daemon connectivity checks on every cluster node
  • Instance migration service monitoring
  • DRBD replication health probe
  • Storage backend availability checks
  • Job queue processing heartbeat
  • Cluster config distribution service monitor
  • Watcher daemon heartbeat
  • Alert channels with appropriate escalation policies

Prerequisites

  • Ganeti cluster with RAPI enabled (default port 5080)
  • SSH access to the master node and worker nodes
  • gnt-cluster, gnt-node, and gnt-instance CLI tools available on the master
  • A free Vigilmon account

Step 1: Monitor RAPI Service Availability

The Ganeti Remote API (RAPI) is the HTTP interface for cluster operations — automation scripts, monitoring tools, and dashboards all use it. If RAPI goes down, external orchestration breaks silently.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the RAPI URL: https://ganeti-master:5080/2/info
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200 (or 401 if auth is enabled — either confirms RAPI is alive).
  6. Enable Monitor SSL certificate if RAPI is configured with TLS → alert at 21 days.
  7. Click Save.

The /2/info endpoint returns cluster metadata (name, version, master node) without requiring authentication by default. A 200 here means the RAPI daemon is running and responding.

Also add a TCP monitor as a faster fallback:

  1. Click Add Monitor → TCP Port.
  2. Enter the master node IP.
  3. Set Port to 5080.
  4. Set Check interval to 1 minute.

Step 2: Monitor Master Node Daemon Health

ganeti-masterd is the central orchestration daemon on the master node. It processes all cluster mutations — instance creation, failover, DRBD configuration. If it crashes, all cluster operations queue or fail.

Add a heartbeat monitor for the masterd service:

#!/bin/bash
# /usr/local/bin/ganeti-masterd-check.sh
systemctl is-active --quiet ganeti && \
  gnt-cluster verify-disks > /dev/null 2>&1 || \
  systemctl is-active --quiet ganeti-masterd
# Simpler: check RAPI responds with cluster info
curl -sf https://localhost:5080/2/info \
  --cacert /var/lib/ganeti/server.pem \
  --cert /var/lib/ganeti/client.pem \
  --key /var/lib/ganeti/client.pem \
  > /dev/null 2>&1 && \
  curl -s https://vigilmon.online/heartbeat/MASTERD_HEARTBEAT_ID

Run via cron every 2 minutes. Set the Vigilmon heartbeat interval to 3 minutes.

Alternatively, since ganeti-masterd binds RAPI, your Step 1 RAPI monitor effectively covers master daemon health. Use the heartbeat for the process-level check.

Step 3: Monitor Node Daemon Connectivity on All Cluster Nodes

ganeti-noded runs on every cluster node and handles low-level VM operations, DRBD management, and storage. When the master loses contact with a node daemon, it marks the node as "offline" and instance migrations and failovers to that node fail.

Add TCP monitors for node daemon ports on every cluster node:

  1. Click Add Monitor → TCP Port.
  2. Enter each node's IP.
  3. Set Port to 1811 (ganeti-noded default port).
  4. Set Check interval to 2 minutes.
  5. Click Save.

Repeat for every node. Label each clearly:

Ganeti Node 01 (10.0.0.11) — noded :1811
Ganeti Node 02 (10.0.0.12) — noded :1811
Ganeti Node 03 (10.0.0.13) — noded :1811

Also add SSH (port 22) TCP monitors for each node — SSH availability confirms the node is alive at the OS level even if noded is down:

Ganeti Node 01 (10.0.0.11) — SSH :22

Step 4: Monitor Instance Migration Service

Instance live migration (between nodes) requires both noded connectivity and functioning DRBD replication. A failed migration leaves the instance in a split state.

Add a heartbeat driven by a migration capability check:

#!/bin/bash
# /usr/local/bin/ganeti-migration-check.sh
# Verify all nodes are online and migration is feasible
OFFLINE_NODES=$(gnt-node list --no-headers -o name,offline 2>/dev/null \
  | grep "True" | wc -l)
[ "$OFFLINE_NODES" -eq 0 ] && \
  curl -s https://vigilmon.online/heartbeat/MIGRATION_HEARTBEAT_ID

Run every 5 minutes. Set the heartbeat interval to 6 minutes. This catches node offline states — the primary prerequisite for successful migrations.

For inter-node migration network verification, add TCP monitors for the migration ports between node pairs:

Node 01 → Node 02: TCP :8102 (KVM live migration port range)

Step 5: Monitor DRBD Replication Health

DRBD replicates instance disk data between primary and secondary nodes. If DRBD enters StandAlone (split-brain) or Inconsistent state, the instance is running without its safety net — a secondary node failure would cause data loss.

Add a heartbeat driven by a DRBD status check:

#!/bin/bash
# /usr/local/bin/ganeti-drbd-check.sh
# Check for any DRBD resources not in UpToDate/Connected state
DRBD_BAD=$(cat /proc/drbd 2>/dev/null | \
  grep -v "UpToDate\|Connected\|^#\|^version\|^GIT\|^0" | \
  grep "ds:" | wc -l)
[ "$DRBD_BAD" -eq 0 ] && \
  curl -s https://vigilmon.online/heartbeat/DRBD_HEARTBEAT_ID

Run every 3 minutes. Set heartbeat interval to 4 minutes. Alert immediately on first miss — DRBD inconsistency is a pre-failure state.

For clusters using the gnt-instance verify approach:

#!/bin/bash
# /usr/local/bin/ganeti-disks-check.sh
gnt-cluster verify-disks 2>&1 | grep -q "verify-disks finished" && \
  curl -s https://vigilmon.online/heartbeat/DISKS_HEARTBEAT_ID

Step 6: Monitor Storage Backend Availability

Ganeti supports multiple storage backends: plain (LVM), DRBD, file, and shared file (NFS/GlusterFS). Each backend has its own failure modes.

LVM storage:

#!/bin/bash
# /usr/local/bin/ganeti-lvm-check.sh
# Check volume group is available
vgdisplay "$VG_NAME" > /dev/null 2>&1 && \
  curl -s https://vigilmon.online/heartbeat/LVM_HEARTBEAT_ID

NFS shared storage:

  1. Click Add Monitor → TCP Port.
  2. Enter the NFS server IP.
  3. Set Port to 2049.
  4. Set Check interval to 2 minutes.

iSCSI storage:

  1. Add TCP monitor for iSCSI target on port 3260.

Run LVM checks every 5 minutes on each node. For shared storage, the TCP monitors are sufficient.

Step 7: Monitor Job Queue Processing

Ganeti's job queue (/var/lib/ganeti/queue/) processes all cluster operations sequentially. If jobs pile up or stop processing, instance operations appear to hang.

Add a heartbeat that verifies the job queue isn't stalled:

#!/bin/bash
# /usr/local/bin/ganeti-jobs-check.sh
# Check for jobs stuck in "waiting" for more than 15 minutes
STUCK=$(gnt-job list --no-headers -o id,status,start_ts 2>/dev/null \
  | awk '$2=="waiting" && (systime() - $3) > 900' | wc -l)
# Also check that job processing daemon is responding
RAPI_OK=$(curl -sf https://localhost:5080/2/jobs \
  --cacert /var/lib/ganeti/server.pem \
  --cert /var/lib/ganeti/client.pem \
  --key /var/lib/ganeti/client.pem \
  > /dev/null 2>&1 && echo 1 || echo 0)
[ "$STUCK" -eq 0 ] && [ "$RAPI_OK" -eq 1 ] && \
  curl -s https://vigilmon.online/heartbeat/JOBS_HEARTBEAT_ID

Run every 10 minutes. Set heartbeat interval to 12 minutes.

Step 8: Monitor Cluster Config Distribution

Ganeti distributes cluster configuration to all nodes via ganeti-confd (config daemon, port 1814) and ganeti-luxid (query daemon). If config distribution breaks, nodes operate on stale configuration and cluster changes don't propagate.

Add TCP monitors for the config distribution daemons:

ganeti-confd (config query daemon):

  1. Click Add Monitor → TCP Port.
  2. Enter the master node IP.
  3. Set Port to 1814 (ganeti-confd UDP/TCP port).
  4. Set Check interval to 2 minutes.

ganeti-luxid (Luxi socket / internal query):

Since luxid operates on a Unix socket, monitor it via a heartbeat:

#!/bin/bash
# /usr/local/bin/ganeti-confd-check.sh
# Test that node config is in sync via RAPI
gnt-node list --no-headers 2>/dev/null | wc -l | grep -qv "^0$" && \
  curl -s https://vigilmon.online/heartbeat/CONFD_HEARTBEAT_ID

Run every 5 minutes. Set heartbeat interval to 6 minutes.

Step 9: Monitor Watcher Daemon

ganeti-watcher runs periodically (typically every 5 minutes via cron or built-in timer) to restart failed instances and perform maintenance tasks. If the watcher stops running, auto-restart of crashed instances stops.

Add a heartbeat monitor specifically for watcher execution:

#!/bin/bash
# /usr/local/bin/ganeti-watcher-check.sh
# Run the watcher and verify it completes successfully
ganeti-watcher --debug 2>&1 | tail -1 | grep -qi "Finished\|done\|watcher" && \
  curl -s https://vigilmon.online/heartbeat/WATCHER_HEARTBEAT_ID

Or check the watcher's last run time from its state file:

#!/bin/bash
# /usr/local/bin/ganeti-watcher-time-check.sh
LAST_RUN=$(stat -c %Y /var/lib/ganeti/watcher.data 2>/dev/null || echo 0)
NOW=$(date +%s)
AGE=$((NOW - LAST_RUN))
# Alert if watcher hasn't run in more than 10 minutes
[ "$AGE" -lt 600 ] && \
  curl -s https://vigilmon.online/heartbeat/WATCHER_HEARTBEAT_ID

Run every 6 minutes. Set heartbeat interval to 7 minutes.

Step 10: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add your preferred channels (email, Slack, PagerDuty, webhook).
  2. For RAPI HTTP and TCP monitors, set Consecutive failures to 2 — RAPI can have brief unavailability during masterd restarts.
  3. For DRBD heartbeats, alert on the first miss (Consecutive failures = 1) — DRBD degradation is always urgent.
  4. For node noded TCP monitors, alert on first miss — an unreachable node daemon means that node can't receive VM failovers.
  5. For watcher and job queue heartbeats, alert after 2 misses — a single slow run is acceptable.
  6. Create Maintenance windows during Ganeti version upgrades, node reboots, or DRBD resync operations.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP :5080 | /2/info | RAPI service down | | HTTP SSL | RAPI endpoint | Certificate expiry | | TCP :5080 | Master node | RAPI process crash | | TCP :1811 | Each cluster node | Node daemon unreachable | | TCP :22 | Each cluster node | Node OS unreachable | | TCP :2049 | NFS storage | Shared storage loss | | TCP :1814 | Master node | Config distribution failure | | Cron heartbeat | masterd RAPI check | Master daemon failure | | Cron heartbeat | DRBD status check | Replication inconsistency | | Cron heartbeat | LVM volume check | Storage backend failure | | Cron heartbeat | Job queue check | Queue stall | | Cron heartbeat | Watcher run check | Auto-restart disabled | | Cron heartbeat | Node online check | Migration capability loss |

Ganeti's strength is its cluster-wide resilience — but that resilience depends on every daemon, DRBD replica, and storage backend staying healthy. Vigilmon's layered monitors give you visibility from the RAPI interface down to individual DRBD disk states, so you catch the early warning signs of cluster degradation before they escalate into instance failures.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles