Monitoring Incus with Vigilmon

Incus is the community-maintained fork of LXD that lets you run Linux containers and QEMU/KVM virtual machines through a single unified REST API. It's the go-to choice for homelab operators, hosting providers, and infrastructure teams who want open-source VM and container orchestration without Canonical's commercial restrictions. But when the Incus daemon (incusd) crashes or a cluster node goes offline, every workload it manages disappears with it. Vigilmon gives you continuous API health checks, cluster member monitoring, storage pool heartbeats, and TLS certificate alerts so you know about daemon problems before your workloads do.

What You'll Set Up

• HTTP monitor for Incus API availability (port 8443)
• TCP probe confirming the Incus daemon is accepting TLS connections
• Heartbeat monitors for storage pool and network bridge health
• SSL certificate alerts for the Incus API TLS certificate

Prerequisites

• Incus installed and running (incusd active as a systemd service)
• Incus REST API accessible on port 8443 (default)
• A free Vigilmon account

Step 1: Monitor Incus API Availability

The Incus REST API exposes a health endpoint at /1.0 that returns server metadata without requiring full client certificate authentication. This is your primary uptime signal — if /1.0 stops responding, the daemon is down and all container and VM management operations will fail.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Incus API URL: https://your-incus-host:8443/1.0
4. Set Check interval to 1 minute.
5. Under Keyword monitoring, enable it and set the keyword to Running.
6. Click Save.

The /1.0 endpoint returns a JSON object that includes "status": "Running" when the daemon is healthy:

{
  "type": "server",
  "status": "Running",
  "api_version": "1.0",
  "api_extensions": [...]
}

The keyword check for "Running" confirms not just that the port is open, but that the daemon has fully initialized and is operational. If Incus is starting up or in a degraded state, the status field will differ.

Note on certificate authentication: The Incus API uses mutual TLS (mTLS) by default. The /1.0 endpoint typically returns server metadata without requiring a client certificate, but some configurations restrict even this path. If Vigilmon receives a 403 or TLS handshake error, configure the monitor to skip certificate verification (for self-signed certs) or add the Incus server's CA certificate to your Vigilmon trust configuration.

Step 2: TCP Probe for Daemon Process Health

A TCP probe to port 8443 provides a fast, authentication-independent signal that the Incus daemon process is alive and accepting connections. This catches daemon crashes before the API-level check completes its HTTP round-trip.

1. Click Add Monitor → set Type to TCP.
2. Enter your Incus host and port: your-incus-host:8443.
3. Set Check interval to 1 minute.
4. Click Save.

If incusd crashes, the TCP connection will be refused immediately. Configure this monitor to alert after 1 consecutive failure — a TCP refusal is never a transient blip, it means the process is gone.

# Verify the port is open locally
nc -zv localhost 8443
# Connection to localhost 8443 port [tcp/*] succeeded!

Step 3: Monitor Cluster Node Health

In a clustered Incus deployment, losing a cluster member means losing all the VMs and containers scheduled on that node. The /1.0/cluster/members endpoint lists every node and its current status.

Set up a heartbeat script that queries the cluster API and pings Vigilmon only if all members are online:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Create a script on your Incus host at /usr/local/bin/incus-cluster-check.sh:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Query cluster members — requires client certificate
MEMBERS=$(incus cluster list --format json 2>/dev/null)
if [ $? -ne 0 ]; then
  echo "Failed to query cluster members"
  exit 1
fi

# Check for any member not in Online status
OFFLINE=$(echo "$MEMBERS" | jq -r '.[] | select(.status != "Online") | .server_name')
if [ -z "$OFFLINE" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
else
  echo "Offline cluster members: $OFFLINE"
fi

chmod +x /usr/local/bin/incus-cluster-check.sh

Add a cron entry to run it every 5 minutes:

*/5 * * * * root /usr/local/bin/incus-cluster-check.sh

If any cluster member is not "Online", the heartbeat is withheld and Vigilmon alerts after the 5-minute window passes.

Step 4: Heartbeat Monitoring for Storage Pools and Networks

Storage pool degradation and failed network bridges cause container creation failures and connectivity loss — often silently. Use scheduled heartbeats to verify these resources stay healthy.

Storage Pool Health

1. Add a second Cron Heartbeat monitor in Vigilmon with a 10 minute interval.
2. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/def456).

Create /usr/local/bin/incus-storage-check.sh:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"

# Get all storage pools and check their status
POOLS=$(incus storage list --format json 2>/dev/null)
if [ $? -ne 0 ]; then
  exit 1
fi

DEGRADED=$(echo "$POOLS" | jq -r '.[] | select(.status != "Created") | .name')
if [ -z "$DEGRADED" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
else
  echo "Degraded storage pools: $DEGRADED"
fi

A storage pool in any state other than "Created" (such as "Errored" or "Pending") indicates underlying ZFS, Btrfs, or LVM issues that will prevent new container creation.

Network Bridge Health

Create /usr/local/bin/incus-network-check.sh:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"

NETWORKS=$(incus network list --format json 2>/dev/null)
if [ $? -ne 0 ]; then
  exit 1
fi

# Check managed bridges are up
DOWN=$(echo "$NETWORKS" | jq -r '.[] | select(.managed == true and .status != "Created") | .name')
if [ -z "$DOWN" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
else
  echo "Down networks: $DOWN"
fi

Add both scripts to cron:

*/10 * * * * root /usr/local/bin/incus-storage-check.sh
*/10 * * * * root /usr/local/bin/incus-network-check.sh

Step 5: SSL Certificate Alerts for Incus TLS

Incus generates a self-signed TLS certificate by default for its API. When this certificate expires, every API client — the incus CLI, Terraform providers, Ansible modules, and automation scripts — loses access simultaneously. The daemon keeps running, but it becomes completely unmanageable.

Add a certificate expiry monitor to your existing Incus API HTTP monitor:

1. Open the HTTP monitor created in Step 1.
2. Enable Monitor SSL certificate under the SSL section.
3. Set Alert when certificate expires in less than 30 days.
4. Click Save.

Incus self-signed certificates are typically valid for 10 years by default, so expiry is rare — but if you've replaced the certificate with a shorter-lived one from your internal CA or Let's Encrypt, a 30-day alert window is essential.

To check the current certificate expiry manually:

# Check the certificate expiry on the Incus API port
openssl s_client -connect localhost:8443 -servername localhost </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
# notBefore=Jan  1 00:00:00 2024 GMT
# notAfter=Jan  1 00:00:00 2034 GMT

If you're using a custom CA, add the CA certificate to Vigilmon's trusted certificate store so the monitor can validate the chain correctly.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your preferred notification method (Slack, email, webhook, PagerDuty).
2. For the API availability monitor, set Consecutive failures before alert to 1 — an Incus daemon crash is never a transient blip.
3. For the TCP probe, also set consecutive failures to 1.
4. For heartbeat monitors (cluster, storage, network), the alert fires automatically when the expected ping interval passes without a check-in.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP keyword | https://host:8443/1.0 → "Running" | Daemon crash, initialization failure | | TCP probe | host:8443 | Process exit, port binding failure | | Cluster heartbeat | /1.0/cluster/members | Offline cluster nodes | | Storage heartbeat | /1.0/storage-pools | Degraded ZFS/Btrfs/LVM pools | | Network heartbeat | /1.0/networks | Failed container network bridges | | SSL certificate | Port 8443 cert expiry | API certificate expiry lockout |

Incus gives you enterprise-grade container and VM management without the commercial restrictions — but that control means you own the observability too. With Vigilmon watching the daemon, cluster nodes, storage pools, network bridges, and TLS certificates, you get the monitoring coverage your infrastructure automation depends on.