tutorial

Monitoring Longhorn with Vigilmon: Manager UI Health, Storage Node Availability, Volume Health & Backup Heartbeats

How to monitor Longhorn cloud-native distributed block storage with Vigilmon — Manager UI health checks, storage node availability, volume health via API, SSL certificate monitoring, and backup completion heartbeats.

Longhorn is the cloud-native distributed block storage system for Kubernetes, providing persistent volumes with built-in replication, snapshots, and backups across your cluster nodes. As the foundation for stateful workloads — databases, message queues, and any pod requiring durable storage — Longhorn failures directly threaten data availability. A degraded storage node, a failed volume, or a missed backup can leave applications in a broken state long before an alert is raised. Vigilmon gives you external visibility into the Longhorn Manager UI, storage node health, volume availability, SSL certificates, and backup completion so you catch failures before they escalate to data loss.

What You'll Build

  • A monitor on the Longhorn Manager UI availability
  • An API-based health check at /v1/healthz
  • A node health check via the Longhorn API
  • SSL certificate monitoring for the Longhorn Manager domain
  • A heartbeat monitor confirming scheduled backups complete on time

Prerequisites

  • A running Longhorn installation in Kubernetes (v1.4+) with the Manager UI exposed at a URL
  • Longhorn Manager accessible over HTTPS (via an Ingress or LoadBalancer)
  • A free account at vigilmon.online

Step 1: Verify the Longhorn Manager Health Endpoint

Longhorn Manager exposes a /v1/healthz endpoint that confirms the Manager pod is alive and its internal API server is responding:

curl https://longhorn.example.com/v1/healthz

A healthy Longhorn Manager returns:

{"message":"ok"}

This endpoint does not aggregate node or volume health — it confirms only that the Manager service itself is operational. You will add deeper checks in subsequent steps.

Step 2: Create a Vigilmon HTTP Monitor for the Health Endpoint

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://longhorn.example.com/v1/healthz.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok (the message field value).
  7. Label: Longhorn Manager Health.
  8. Click Save.

Set alerts to trigger after 1 consecutive failure — the Longhorn Manager being down means volume operations, snapshot creation, and backup scheduling are all unavailable.

Step 3: Monitor the Longhorn Manager UI

The Longhorn Manager web UI provides operational visibility into nodes, volumes, replicas, and backup targets. Even if the /v1/healthz endpoint returns ok, the web UI can be broken if the frontend assets fail to serve. Add a separate monitor for the UI:

curl -I https://longhorn.example.com

A running UI returns 200 OK with HTML content:

  1. Add Monitor → HTTP.
  2. URL: https://longhorn.example.com (your Longhorn Manager URL, root path).
  3. Check interval: 5 minutes.
  4. Expected status: 200.
  5. Label: Longhorn Manager UI.
  6. Click Save.

Step 4: Monitor Storage Node Connectivity via the API

Longhorn stores node health in its API at /v1/nodes. A request to this endpoint returns the list of storage nodes and their readiness status. When a node goes unschedulable or disconnects from the Manager, the API reflects this before volumes start failing.

curl https://longhorn.example.com/v1/nodes \
  -H "Authorization: Bearer your-api-token"

A healthy response contains an array of node objects with "allowScheduling": true and "state": "schedulable" for each active node.

To monitor node health externally with Vigilmon, use keyword matching on the API response:

  1. Add Monitor → HTTP.
  2. URL: https://longhorn.example.com/v1/nodes.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: schedulable (present in the response when at least one node is healthy).
  6. HTTP Headers: Authorization: Bearer your-api-token (if authentication is required).
  7. Label: Longhorn Storage Nodes.
  8. Click Save.

Authentication: If your Longhorn ingress requires authentication (basic auth or an auth proxy), add the appropriate headers or credentials in the monitor's header configuration.

Keyword limitation: The schedulable keyword confirms at least one node is healthy but does not detect partial degradation (e.g., 2 of 3 nodes down). For multi-node quorum monitoring, pair this with Longhorn's Prometheus metrics or Kubernetes alerts.

Step 5: Monitor Volume Health via the Longhorn API

Beyond node availability, individual volumes can enter degraded or faulted states when replicas are lost. The Longhorn API at /v1/volumes exposes volume health:

curl https://longhorn.example.com/v1/volumes \
  -H "Authorization: Bearer your-api-token"

A healthy response contains volumes with "robustness": "healthy". When a volume is degraded, its robustness changes to degraded or faulted.

Add this monitor to alert on volume degradation:

  1. Add Monitor → HTTP.
  2. URL: https://longhorn.example.com/v1/volumes.
  3. Check interval: 5 minutes.
  4. Expected status: 200.
  5. Keyword: healthy (present when volumes are in healthy state).
  6. Label: Longhorn Volume Health.
  7. Click Save.

Note: If your cluster has no volumes yet, this endpoint returns an empty list and the healthy keyword will not match. In that case, use the schedulable node check as your primary health signal and add volume monitoring once workloads are running.

Step 6: Set Up Backup Completion Heartbeat Monitoring

Longhorn's backup feature protects your persistent volume data against cluster-level failures. However, a backup job that silently fails leaves you with stale snapshots and no recovery point. Use Vigilmon heartbeat monitoring to confirm backups complete on schedule:

Step 6a — Create a Heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat.
  2. Name: Longhorn daily backup.
  3. Expected interval: 24 hours (or match your backup schedule).
  4. Grace period: 2 hours.
  5. Copy the generated ping URL (e.g., https://vigilmon.online/ping/abc123).

Step 6b — Add a ping step to your backup CronJob:

Create a Kubernetes CronJob that runs after Longhorn's backup completes. Longhorn's backup-target reconciliation can be monitored via a custom CronJob that checks backup status and pings Vigilmon on success:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: longhorn-backup-heartbeat
  namespace: longhorn-system
spec:
  schedule: "0 3 * * *"  # match your backup window
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: ping
              image: curlimages/curl:latest
              command:
                - sh
                - -c
                - |
                  # Verify latest backup exists and ping Vigilmon
                  curl -fsS -m 10 https://vigilmon.online/ping/abc123
          restartPolicy: OnFailure

Pair this with Longhorn's backup CronJob so the heartbeat fires only after a successful backup snapshot. If the backup job fails or the cluster is unreachable, Vigilmon fires an alert after the grace period expires.

Step 7: Monitor SSL Certificates

The Longhorn Manager UI is accessed by platform engineers and SREs directly in their browsers. A certificate expiry causes TLS errors that break both browser access and any scripted API calls that use the Longhorn REST API:

  1. Add Monitor → SSL Certificate.
  2. Domain: longhorn.example.com (your Longhorn Manager domain).
  3. Alert when expiry is within: 30 days.
  4. Alert again at: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | /v1/healthz | Non-200 or ok missing | Check Longhorn Manager pod; kubectl get pods -n longhorn-system | | Manager UI | Non-200 | Ingress or UI pod issue; check ingress controller | | Storage nodes | schedulable absent | Node(s) degraded; check node taints and Longhorn node controller | | Volume health | healthy absent | Volume degraded; check replica rebuild in Longhorn UI | | Backup heartbeat | No ping within window | Backup failed or skipped; verify backup target and snapshot job | | SSL certificate | < 30 days | Renew; verify cert-manager or ingress TLS config |

Common Longhorn Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Longhorn Manager pod crash | /v1/healthz unreachable; alert within 60 s | | Storage node disconnects | Node API returns no schedulable nodes; volume replicas degrade | | Volume replica lost (1-of-3) | Volume health monitor detects degraded state | | Volume completely faulted | Volume health monitor fires; workloads may lose storage | | Backup job fails silently | Heartbeat monitor alerts after grace period | | SSL certificate expires | SSL monitor alerts at 30-day threshold | | Ingress controller misconfigured | UI monitor fires; /v1/healthz may still be reachable internally | | All storage nodes evicted | Node and volume monitors both fire; cluster-level event |

Longhorn's distributed architecture means storage failures manifest in layers — a single node going offline triggers replica rebuilds, a missed backup leaves you without a recovery point, and a faulted volume can bring down the stateful workloads depending on it. Vigilmon's layered monitoring of the Manager API, storage node health, volume state, backup heartbeats, and SSL certificates gives you external visibility into every layer of Longhorn's health, so you catch failures long before they escalate to data loss or workload downtime.

Start monitoring Longhorn in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles