Monitoring LXD with Vigilmon

LXD lets you run system containers and full virtual machines side-by-side from one management plane, with clustering, storage pools, and OVN networking built in. But when LXD's daemon goes silent, every container and VM lifecycle operation stops cold. Vigilmon gives you continuous coverage of the LXD API, cluster member health, storage pools, and container resource metrics — so you catch problems before workloads go offline.

What You'll Set Up

LXD daemon health check via the REST API (GET /1.0)
Cluster member status monitoring
Storage pool usage alerts
Container/VM CPU and memory utilization via the LXD metrics endpoint
Network bridge health for lxdbr0
Snapshot job and migration success-rate alerts

Prerequisites

LXD 5.0+ installed (Canonical's snap package or distro package)
LXD configured with the API exposed on HTTPS port 8443 (for remote/Vigilmon probes) or via a local proxy
A free Vigilmon account

Step 1: Expose the LXD API for External Health Probes

LXD's REST API listens on a Unix socket by default. For Vigilmon to probe it remotely, enable HTTPS access:

lxc config set core.https_address :8443

Create a client certificate for Vigilmon's health check calls (or generate a trust token for passwordless certificate enrollment):

lxc config trust add --name vigilmon

Copy the generated certificate fingerprint — you'll use it in later steps to distinguish monitoring from admin traffic.

Verify the API is reachable:

curl -sk https://localhost:8443/1.0 | python3 -m json.tool | grep '"status"'

You should see "status": "Success". Vigilmon will probe this URL every minute.

Step 2: Monitor LXD Daemon Health

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
Enter the URL: https://your-lxd-host:8443/1.0
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Under Keyword, enter "status":"Success" so Vigilmon confirms the daemon returned a healthy payload — not just any 200.
Click Save.

If the LXD daemon crashes or the API socket becomes unavailable, Vigilmon fires an alert within one check cycle. Downtime on /1.0 blocks all container and VM lifecycle operations.

Step 3: Monitor Cluster Member Health

In a multi-node LXD cluster, the /1.0/cluster/members endpoint lists each member and its status. A member going offline can trigger workload migration failures.

Add a cluster members endpoint monitor:

Add Monitor → Type: HTTP / HTTPS.
URL: https://your-lxd-host:8443/1.0/cluster/members
Check interval: 1 minute.
Keyword: "status":"Online" — alert if the keyword is absent, indicating at least one member is offline or in an unknown state.
Save.

For more granular per-member alerts, write a lightweight health-check script that queries each member and exposes an HTTP endpoint Vigilmon can probe:

#!/bin/bash
# /usr/local/bin/lxd-cluster-health.sh
lxc cluster list --format json | \
  python3 -c "import sys,json; members=json.load(sys.stdin); \
    offline=[m['server_name'] for m in members if m['status']!='Online']; \
    exit(1) if offline else exit(0)"

Wrap it with a minimal HTTP server (e.g., python3 -m http.server with a health-gate wrapper) on port 9101, then monitor http://your-lxd-host:9101/cluster.

Step 4: Monitor Storage Pool Disk Usage

Full storage pools cause container starts to fail immediately. LXD reports pool usage via GET /1.0/storage-pools/{pool}/resources.

For each storage pool (ZFS, Btrfs, LVM, Ceph, directory), add a Vigilmon heartbeat monitor:

Create a script /usr/local/bin/lxd-pool-check.sh:

#!/bin/bash
POOL="${1:-default}"
RESULT=$(lxc storage info "$POOL" --format json)
USED=$(echo "$RESULT" | python3 -c "import sys,json; d=json.load(sys.stdin)['space']; print(d['used']/d['total']*100)")
python3 -c "exit(1) if float('$USED') > 80 else exit(0)"

Expose this via cron as a Vigilmon Heartbeat monitor: if the cron job stops sending pings, you'll know the check itself failed.
In Vigilmon: Add Monitor → Type: Heartbeat.
Set Expected interval to 5 minutes (match your cron schedule).
Copy the heartbeat URL and add it to cron:

*/5 * * * * /usr/local/bin/lxd-pool-check.sh default && curl -s https://vigilmon.online/heartbeat/YOUR_TOKEN

Alert threshold: >80% pool usage. Prevent failed container starts by catching capacity problems early.

Step 5: Monitor Container and VM Resource Usage

LXD exposes Prometheus-compatible metrics at GET /1.0/metrics covering CPU, memory, disk, and network per container/VM.

Add a Vigilmon HTTP monitor on the metrics endpoint to confirm it's live:

Add Monitor → Type: HTTP / HTTPS.
URL: https://your-lxd-host:8443/1.0/metrics
Check interval: 1 minute.
Expected status: 200.
Keyword: lxd_cpu_seconds_total — confirms the metrics scrape is real.
Save.

For CPU/memory alerting, scrape the /1.0/metrics endpoint with Prometheus (or VictoriaMetrics) and set alert rules. Key metrics:

| Metric | Alert threshold | |---|---| | lxd_cpu_seconds_total per instance | >90% host CPU sustained 5 min | | lxd_memory_RSS_bytes per instance | >80% host RAM | | lxd_network_receive_drop_total | Any increase |

Step 6: Monitor the LXD Network Bridge

The lxdbr0 bridge is the default network for LXD containers. If it goes down, all container network connectivity drops.

Monitor bridge availability via a heartbeat from the LXD host:

*/1 * * * * ip link show lxdbr0 | grep -q "state UP" && curl -s https://vigilmon.online/heartbeat/BRIDGE_TOKEN

Set the Vigilmon heartbeat interval to 2 minutes (2× the cron interval) so Vigilmon fires an alert if two consecutive checks miss.

Step 7: Alert on Missed Snapshots and Migration Failures

Snapshot health: Schedule snapshots with LXD policies and verify the last snapshot age via lxc info <instance>:

#!/bin/bash
INSTANCE="${1:-my-container}"
LAST=$(lxc info "$INSTANCE" --format json | python3 -c \
  "import sys,json,datetime; snaps=json.load(sys.stdin).get('snapshots',[]); \
   latest=max((s['created_at'] for s in snaps), default='1970-01-01T00:00:00Z'); \
   age=(datetime.datetime.utcnow()-datetime.datetime.fromisoformat(latest.replace('Z',''))).total_seconds(); \
   exit(1) if age > 86400 else exit(0)")
[ "$LAST" = "0" ] && curl -s https://vigilmon.online/heartbeat/SNAPSHOT_TOKEN

Add as a daily cron. If the heartbeat goes silent, Vigilmon alerts you that the snapshot schedule failed.

Migration failures: Wrap lxc move calls in CI/automation scripts and post to Vigilmon only on success. A missing heartbeat equals a failed migration.

Step 8: Configure Alerting

In Vigilmon, navigate to Alert Channels and connect your preferred delivery method:

Email — reliable for on-call escalation
Slack or Discord webhook — real-time team notification
PagerDuty — for production clusters requiring pager coverage

Set alert policies per monitor:

| Monitor | Condition | Action | |---|---|---| | LXD daemon /1.0 | Down >1 min | Page on-call | | Cluster member | Missing "Online" | Page on-call | | Storage pool | Heartbeat missed | Alert team | | Bridge lxdbr0 | Heartbeat missed | Page on-call | | Snapshot job | Daily heartbeat missed | Alert team |

Conclusion

LXD clusters combine system containers and VMs into a single management plane, but that unified control surface is also a single point of failure. With Vigilmon monitoring the daemon health endpoint, cluster member status, storage pool capacity, resource metrics, and network bridges, you catch failures the moment they appear — before workloads become unavailable or migrations fail silently. Add the heartbeat monitors for snapshot jobs and migrations, configure your alert channels, and your LXD infrastructure is observable from day one.