Monitoring Harvester HCI with Vigilmon

Harvester HCI packs virtual machines, distributed storage, and container orchestration into a single Kubernetes-native hyperconverged platform. That power comes with layered complexity: Kubernetes control plane, KubeVirt VM operator, Longhorn distributed block storage, Multus network fabric, and Rancher integration all need to stay healthy simultaneously. When any layer silently degrades, VMs slow down, storage becomes unavailable, or new workloads fail to schedule. Vigilmon watches the externally observable signals that tell you which layer is failing before your VMs do.

What You'll Set Up

• HTTPS monitor for the Harvester management API server
• KubeVirt operator health endpoint check
• Longhorn distributed storage health probe
• Node agent connectivity monitors for each HCI node
• VM scheduling service heartbeat
• Image import and storage pipeline check
• Rancher integration status monitoring
• Backup and restore job heartbeats
• Cluster etcd health monitoring
• SSL certificate monitoring for all endpoints

Prerequisites

• Harvester HCI cluster running (port 443 for UI and API)
• kubectl configured with cluster access
• SSH access to at least one control plane node
• A free Vigilmon account

Step 1: Monitor the Harvester Management API Server

The Harvester UI and all VM operations go through the Kubernetes API server running on port 443. If the API server is unavailable, no VM operations can proceed — not even kubectl commands.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Harvester cluster VIP URL: https://harvester.yourdomain.com/api/v1/namespaces/harvester-system/services
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200 (or 401 if unauthenticated — the 401 confirms the API server is responding).
6. Enable Monitor SSL certificate → alert at 21 days.
7. Click Save.

For a public health endpoint that doesn't require auth credentials, use the Kubernetes healthz endpoint:

https://harvester.yourdomain.com/healthz

The /healthz endpoint returns ok (HTTP 200) when the API server is healthy. This is the canonical liveness check for Kubernetes control planes.

Step 2: Monitor KubeVirt Operator Health

KubeVirt is the Kubernetes operator that enables VM workloads on Harvester. If the virt-operator, virt-api, or virt-controller pods fail, VM creation, migration, and lifecycle operations stop — but the Kubernetes API itself continues to appear healthy.

Add a heartbeat driven by a KubeVirt status check:

#!/bin/bash
# /usr/local/bin/kubevirt-health-check.sh
# Check that core KubeVirt components are running
VIRT_API=$(kubectl -n kubevirt get pod -l kubevirt.io=virt-api \
  --field-selector=status.phase=Running --no-headers 2>/dev/null | wc -l)
VIRT_CTRL=$(kubectl -n kubevirt get pod -l kubevirt.io=virt-controller \
  --field-selector=status.phase=Running --no-headers 2>/dev/null | wc -l)
[ "$VIRT_API" -gt 0 ] && [ "$VIRT_CTRL" -gt 0 ] && \
  curl -s https://vigilmon.online/heartbeat/KUBEVIRT_HEARTBEAT_ID

Run via cron every 3 minutes. Set the Vigilmon heartbeat interval to 4 minutes.

Optionally expose this as an HTTP endpoint using a lightweight wrapper (e.g. a small Go or Node.js service) and add an HTTP monitor for richer alerting.

Step 3: Monitor Longhorn Distributed Storage Health

Longhorn provides distributed block storage for VM disks across all Harvester nodes. A degraded Longhorn volume means a VM's disk is running on fewer replicas than required — a single node failure could cause data loss or unavailability.

Add an HTTP monitor for the Longhorn manager health endpoint:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: https://harvester.yourdomain.com/v1/harvester/volumes (via the Harvester proxy to Longhorn).
3. Alternatively, if you access Longhorn UI directly: http://longhorn-frontend.longhorn-system.svc:80 (requires in-cluster access or port-forward).
4. Set Expected status to 200 or 401.
5. Set Check interval to 2 minutes.

For a deeper Longhorn health check, add a heartbeat:

#!/bin/bash
# /usr/local/bin/longhorn-health-check.sh
# Check Longhorn manager is running
LH_PODS=$(kubectl -n longhorn-system get pod -l app=longhorn-manager \
  --field-selector=status.phase=Running --no-headers 2>/dev/null | wc -l)
# Check no volumes are in degraded/faulted state
DEGRADED=$(kubectl -n longhorn-system get volume.longhorn.io \
  -o json 2>/dev/null | python3 -c \
  "import sys,json; vols=json.load(sys.stdin)['items']; \
   bad=[v for v in vols if v['status'].get('state') in ('degraded','faulted','error')]; \
   print(len(bad))")
[ "$LH_PODS" -gt 0 ] && [ "$DEGRADED" -eq 0 ] && \
  curl -s https://vigilmon.online/heartbeat/LONGHORN_HEARTBEAT_ID

Run every 5 minutes. Alert immediately on miss — degraded storage is a pre-failure indicator.

Step 4: Monitor Node Agent Health

Each Harvester node runs k3s (Kubernetes agent), longhorn-engine, and virt-handler (KubeVirt node agent). If a node's agent becomes unhealthy, VMs on that node stop migrating and new VMs won't schedule there.

Add TCP monitors for each node's API port:

1. Click Add Monitor → TCP Port.
2. Enter each node's management IP.
3. Set Port to 6443 (K3s API, if accessible) or 22 (SSH, confirms node is alive).
4. Set Check interval to 2 minutes.

Label each: Harvester Node 1 (192.168.1.11) — k3s agent.

Add a heartbeat for node readiness:

#!/bin/bash
# /usr/local/bin/harvester-nodes-check.sh
NOT_READY=$(kubectl get nodes --no-headers 2>/dev/null \
  | grep -v " Ready " | wc -l)
[ "$NOT_READY" -eq 0 ] && \
  curl -s https://vigilmon.online/heartbeat/NODE_HEARTBEAT_ID

Run every 2 minutes. Set heartbeat interval to 3 minutes.

Step 5: Monitor VM Scheduling Service

The Kubernetes scheduler places new VMs on healthy nodes. If the scheduler crashes (rare but possible during K3s upgrades), new VM creation hangs indefinitely without an obvious error.

#!/bin/bash
# /usr/local/bin/kube-scheduler-check.sh
SCHED=$(kubectl -n kube-system get pod -l component=kube-scheduler \
  --field-selector=status.phase=Running --no-headers 2>/dev/null | wc -l)
# For k3s, scheduler runs in-process — check the k3s service
systemctl is-active --quiet k3s && \
  curl -s https://vigilmon.online/heartbeat/SCHEDULER_HEARTBEAT_ID

Set heartbeat interval to 3 minutes.

Step 6: Monitor Image Import and Storage Pipeline

Harvester's image import service downloads VM images (QCOW2, ISO) into Longhorn-backed storage. Failed imports leave VMs stuck at provisioning — a common pain point when storage or network is degraded.

Add an HTTP monitor for the image import service:

https://harvester.yourdomain.com/v1/harvester/images

Expected: HTTP 200 or 401. This confirms the Harvester API (which fronts the image import pipeline) is responsive.

For end-to-end import health, use a heartbeat driven by a check of in-progress imports:

#!/bin/bash
# /usr/local/bin/harvester-images-check.sh
# Ensure no images are stuck in "Importing" state for >30 minutes
STUCK=$(kubectl -n harvester-system get virtualmachineimage \
  -o json 2>/dev/null | python3 -c \
  "import sys,json; imgs=json.load(sys.stdin)['items']; \
   stuck=[i for i in imgs if i['status'].get('progress', 100) < 100]; \
   print(len(stuck))")
[ "$STUCK" -eq 0 ] && \
  curl -s https://vigilmon.online/heartbeat/IMAGES_HEARTBEAT_ID

Step 7: Monitor Rancher Integration Status

Harvester integrates with Rancher for multi-cluster management. If the Rancher endpoint becomes unreachable, the Harvester-Rancher link breaks — you lose the unified management plane.

Add a TCP monitor for the Rancher server:

1. Click Add Monitor → TCP Port (or HTTP if Rancher is accessible via HTTPS).
2. Enter the Rancher server IP/hostname.
3. Set Port to 443.
4. Set Check interval to 2 minutes.
5. Click Save.

Add an HTTP monitor for the Rancher API:

https://rancher.yourdomain.com/v3/clusters

Expected: 200 or 401. This confirms Rancher is serving API requests.

Step 8: Monitor Backup and Restore Jobs

Harvester supports VM snapshots and backups to S3 or NFS. Silent backup failures leave you without recovery points.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set interval to your backup schedule plus 20% buffer.
3. Add the heartbeat ping to a backup completion hook:

#!/bin/bash
# /usr/local/bin/harvester-backup-check.sh
# Check that recent backups completed successfully
FAILED=$(kubectl -n harvester-system get virtualmachinebackup \
  --no-headers 2>/dev/null | grep -i "false\|error\|failed" | wc -l)
[ "$FAILED" -eq 0 ] && \
  curl -s https://vigilmon.online/heartbeat/BACKUP_HEARTBEAT_ID

Run daily (or after each backup window). Set heartbeat interval to match your backup schedule.

Step 9: Monitor Cluster etcd Health

K3s uses an embedded etcd for cluster state. etcd health is critical — a quorum loss means no Kubernetes API operations can complete.

#!/bin/bash
# /usr/local/bin/etcd-health-check.sh
# Check etcd endpoint health via k3s
k3s etcd-snapshot ls > /dev/null 2>&1 || \
  ETCD_STATUS=$(kubectl -n kube-system get pod \
    -l component=etcd --field-selector=status.phase=Running --no-headers 2>/dev/null | wc -l)
[ "${ETCD_STATUS:-0}" -gt 0 ] && \
  curl -s https://vigilmon.online/heartbeat/ETCD_HEARTBEAT_ID

For a direct etcd healthcheck (if using embedded etcd with K3s):

ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/client.key && \
  curl -s https://vigilmon.online/heartbeat/ETCD_HEARTBEAT_ID

Run every 2 minutes. Set heartbeat interval to 3 minutes.

Step 10: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and configure email, Slack, or PagerDuty.
2. For the API server HTTP monitor, set Consecutive failures to 2 — brief API server restarts during etcd leader elections can cause single probe timeouts.
3. For storage heartbeats (Longhorn, etcd), alert on the first miss (Consecutive failures = 1).
4. For node readiness and scheduler checks, alert on first miss.
5. Use Maintenance windows during K3s or Harvester upgrades.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP :443 | /healthz | Kubernetes API server down | | HTTP :443 | Harvester API endpoint | Management plane failure | | HTTP SSL | Harvester domain | Certificate expiry | | TCP :443 | Rancher server | Rancher integration loss | | TCP :22 | Each HCI node | Node unreachable | | Cron heartbeat | KubeVirt pod check | VM operator failure | | Cron heartbeat | Longhorn volume check | Storage degradation | | Cron heartbeat | Node readiness check | Node not-ready condition | | Cron heartbeat | etcd health check | Cluster state loss | | Cron heartbeat | Backup status check | Silent backup failure |

Harvester's layered architecture — Kubernetes, KubeVirt, Longhorn, Rancher — means failures don't always surface at the top level. Vigilmon's combination of HTTP probes, TCP checks, and heartbeat monitors gives you visibility into each layer independently, so a Longhorn volume degradation gets caught before it becomes a VM data loss incident.