Monitoring Xen Orchestra with Vigilmon

Xen Orchestra is the central management plane for XCP-ng and Citrix Hypervisor environments. When it's healthy, you have a live view of every VM, storage repository, and backup job in your cluster. When it silently fails — a crashed Node.js worker, a stale XenAPI connection, a WebSocket dropout — your team loses visibility and backup jobs stop without any alert. Vigilmon watches the signals that matter before they become incidents.

What You'll Set Up

• HTTP uptime monitor for the Xen Orchestra web console
• WebSocket real-time connection health check
• XenAPI connectivity probe for each hypervisor host
• VM operation service health endpoint
• Storage repository connectivity monitor
• Cron heartbeats for backup and scheduled snapshot jobs
• Live migration service health check
• Resource pool status monitoring
• Patch management service availability
• Alert channels with escalation for critical infrastructure

Prerequisites

• Xen Orchestra installed (community or XO Appliance) on port 80/443
• Access to the Xen Orchestra config and a shell on the host
• At least one XCP-ng or Citrix Hypervisor pool connected
• A free Vigilmon account

Step 1: Monitor the Xen Orchestra Web Console

The web console is your primary signal — if Node.js crashes or nginx goes down, the entire management UI disappears.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Xen Orchestra URL: https://xo.yourdomain.com (or http://xo-host-ip).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Enable Monitor SSL certificate and set expiry alert to 21 days.
7. Click Save.

Xen Orchestra's root path returns a React SPA shell. If you want a richer health signal, check the API endpoint instead:

https://xo.yourdomain.com/api/

The REST API endpoint returns a JSON status response. A 200 here confirms both Node.js and the API router are alive.

Step 2: Check WebSocket Connection Health

Xen Orchestra relies on a persistent WebSocket connection (/api via socket.io) for real-time VM console, live migration updates, and performance graphs. A failed WebSocket means your UI will appear to load but show stale data with no live updates.

Add a TCP monitor for the WebSocket port:

1. Click Add Monitor → TCP Port.
2. Enter the Xen Orchestra host IP.
3. Set Port to 443 (or 80 for HTTP installs).
4. Set Check interval to 1 minute.
5. Click Save.

For a deeper WebSocket probe, add a custom health endpoint to your Xen Orchestra instance. In /etc/xo-server/config.toml, you can expose a health route, or add a lightweight sidecar script that connects via the JSON-RPC WebSocket API and confirms the session handshake:

// ws-healthcheck.js — run as a systemd service or cron
const WebSocket = require('ws')
const ws = new WebSocket('wss://localhost/api', { rejectUnauthorized: false })
ws.on('open', () => {
  process.exit(0)  // success
})
ws.on('error', () => {
  process.exit(1)  // failure — trigger alert
})
setTimeout(() => process.exit(1), 5000)

Wrap this in a small HTTP endpoint (e.g. using http-server or a tiny Express app) and add an HTTP monitor pointing to it.

Step 3: Monitor XenAPI Connectivity to Hypervisor Hosts

Xen Orchestra connects to each XCP-ng or Citrix Hypervisor host via the XenAPI (HTTPS on port 443). If a host's XenAPI becomes unreachable — network partition, host reboot, SSL cert issue — Xen Orchestra loses visibility into that pool without alerting you.

Add a TCP monitor for each hypervisor host:

1. Click Add Monitor → TCP Port.
2. Enter the hypervisor host IP (e.g. 192.168.1.10).
3. Set Port to 443.
4. Set Check interval to 2 minutes.
5. Click Save.

Repeat for every host in every pool. Label each monitor clearly:

Pool-A / Host-1 (192.168.1.10) — XenAPI :443
Pool-A / Host-2 (192.168.1.11) — XenAPI :443
Pool-B / Host-1 (192.168.1.20) — XenAPI :443

For XAPI health specifically, you can also probe the XenAPI management endpoint:

https://192.168.1.10/  (should return 200 — XAPI web page or redirect)

Step 4: Monitor VM Operation Service Health

The VM operation service (start, stop, migrate, snapshot) is the core of Xen Orchestra's value. Verify it's responsive by probing the Xen Orchestra JSON-RPC API with a lightweight sentinel call.

Create a small health-check script on the Xen Orchestra host:

#!/bin/bash
# /usr/local/bin/xo-vm-service-check.sh
curl -sf -X POST https://localhost/api \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"session.getVersion","params":{},"id":1}' \
  --resolve "localhost:443:127.0.0.1" \
  --cacert /etc/ssl/xo/cert.pem \
  | grep -q '"result"' && echo "ok" || exit 1

Expose it via a minimal HTTP wrapper and add an HTTP monitor in Vigilmon. Alternatively, monitor the xo-server systemd service via a heartbeat:

# /usr/local/bin/xo-heartbeat.sh — run via cron every 5 minutes
systemctl is-active --quiet xo-server && \
  curl -s https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID

Add a Cron Heartbeat monitor in Vigilmon with a 6-minute expected interval to catch missed pings.

Step 5: Monitor Storage Repository Connectivity

Storage repositories (NFS, iSCSI, NFS ISO) are attached at the hypervisor level. If an SR becomes unavailable, VM disk I/O freezes. Add TCP monitors for your storage backends:

NFS storage:

1. Click Add Monitor → TCP Port.
2. Enter the NFS server IP.
3. Set Port to 2049.
4. Set Check interval to 2 minutes.
5. Click Save.

iSCSI storage:

1. Add a TCP monitor for the iSCSI target IP on port 3260.

Ceph/RBD (if used):

1. Add a TCP monitor for the Ceph monitor node on port 6789.

Label each storage monitor with the SR name and type. When an SR TCP check fails, you'll know before VMs start reporting I/O errors.

Step 6: Heartbeat Monitors for Backup Jobs

Xen Orchestra backup jobs (full backup, delta backup, continuous replication) run on a schedule. If a job fails silently, you discover the gap only when you need to restore.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected interval to your backup job frequency plus 20% buffer (e.g. daily job → 1500 minutes).
3. Copy the heartbeat URL.
4. In Xen Orchestra, go to Backup → your job → Post-job hooks (or use the XO CLI):

# Using xo-cli — call after job completion
xo-cli backup.run jobId=YOUR_JOB_ID \
  && curl -s https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID

For automated backup jobs triggered by xo-server's built-in scheduler, add the ping to a wrapper script:

#!/bin/bash
# /usr/local/bin/run-xo-backup.sh
xo-cli backup.run jobId="$1"
EXIT=$?
if [ $EXIT -eq 0 ]; then
  curl -s https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID
fi
exit $EXIT

Step 7: Monitor Live Migration Service Health

Live migration (XenMotion) moves running VMs between hosts without downtime. The migration service relies on the inter-host management network and XAPI coordination. Monitor it by checking the management network path between hosts:

1. Add TCP monitors between host pairs on port 443 (XAPI) and port 6900–6999 (migration traffic range, if your hypervisor version exposes it).
2. Set check interval to 2 minutes.

For a functional check, add a scheduled Xen Orchestra backup job that performs a test migration between hosts on a non-production VM, running a heartbeat after success.

Step 8: Monitor Resource Pool Status

Add an HTTP monitor for each XO pool's overview endpoint to confirm pool-level coordination:

https://xo.yourdomain.com/api/  (verify pool list loads — 200 + JSON)

For a deeper probe, run a periodic script that uses the xo-cli to query pool status and emits a heartbeat only when all pools report healthy:

#!/bin/bash
# /usr/local/bin/xo-pool-check.sh
POOLS=$(xo-cli pool.getAll | python3 -c "import sys,json; pools=json.load(sys.stdin); print(len(pools))")
[ "$POOLS" -gt 0 ] && curl -s https://vigilmon.online/heartbeat/POOL_HEARTBEAT_ID

Step 9: Monitor Patch Management Service

Xen Orchestra's patch management service periodically syncs the list of available hypervisor patches from Citrix/XenServer's update catalog. If the service loses internet connectivity or the upstream catalog changes format, patching silently breaks.

Add a heartbeat monitor driven by a weekly patch-check script:

#!/bin/bash
# /usr/local/bin/xo-patch-check.sh — run weekly via cron
xo-cli pool.listMissingPatches id="$POOL_ID" 2>&1 | grep -v "error" && \
  curl -s https://vigilmon.online/heartbeat/PATCH_HEARTBEAT_ID

Set the Vigilmon heartbeat interval to 10080 minutes (7 days) with a buffer of 1440 minutes (1 day) to absorb timing variance.

Step 10: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your preferred channels (email, Slack, PagerDuty, webhook).
2. For the web console and XenAPI host monitors, set Consecutive failures before alert to 2 — a single probe timeout during host maintenance is common.
3. For backup heartbeat monitors, alert immediately on the first miss: set Consecutive failures to 1.
4. Use Maintenance windows during planned hypervisor patching or pool upgrades to suppress false positives.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP | https://xo.domain.com | Console crash, Node.js failure | | HTTP SSL | Web console domain | Certificate expiry | | TCP :443 | Each hypervisor host | XenAPI connectivity loss | | TCP :2049 | NFS storage server | SR mount failure | | TCP :3260 | iSCSI target | Block storage path failure | | Cron heartbeat | Backup job wrapper | Silent backup failure | | Cron heartbeat | VM service check | xo-server process failure | | Cron heartbeat | Pool status check | Pool coordination loss |

Xen Orchestra is the single pane of glass for your entire virtualization estate. Vigilmon ensures that pane doesn't go dark without warning — so your backup jobs, live migrations, and hypervisor health stay visible even when something silently breaks behind the scenes.