Monitoring democratic-csi with Vigilmon: TrueNAS API Health, iSCSI Endpoints & Storage Provisioner Alerts

democratic-csi is a community-developed CSI (Container Storage Interface) driver that bridges Kubernetes persistent volume provisioning with TrueNAS (and FreeNAS) storage appliances over iSCSI, NFS, and SMB. When democratic-csi is healthy, Kubernetes PersistentVolumeClaims are provisioned automatically from TrueNAS datasets and zvols — stateful workloads like databases, message queues, and file servers get storage on demand. When it fails, PVC creation hangs indefinitely in Pending, existing volumes become inaccessible, and stateful pods cannot restart. Vigilmon gives you external visibility into the TrueNAS API, storage protocol endpoints, CSI controller availability, and web UI reachability so you catch storage layer failures before they cascade into application outages.

What You'll Build

• A monitor on the TrueNAS API health endpoint to detect NAS appliance failures
• TCP monitors on iSCSI and NFS endpoints for protocol-level connectivity
• A CSI controller availability check to verify the provisioner is running
• TrueNAS web UI availability monitoring
• An alerting setup that separates TrueNAS failures from CSI controller failures

Prerequisites

• democratic-csi deployed in your Kubernetes cluster (typically in the democratic-csi namespace)
• A TrueNAS Core or TrueNAS SCALE appliance accessible from your Kubernetes nodes
• The TrueNAS API and web UI accessible at a known hostname or IP
• A free account at vigilmon.online

Step 1: Understand democratic-csi's Monitoring Surface

democratic-csi has two distinct failure domains: the TrueNAS storage appliance (API, iSCSI target, NFS exports) and the Kubernetes CSI controller/node pods. Both must be healthy for PVC provisioning and volume attachment to work.

The TrueNAS API v2.0 exposes a system information endpoint that returns appliance health without authentication:

curl https://truenas.example.com/api/v2.0/system/info
# Returns JSON: {"version": "TrueNAS-SCALE-24.04", "hostname": "truenas", ...}

For systems that require authentication on all endpoints, use the API key:

curl -H "Authorization: Bearer YOUR_API_KEY" \
  https://truenas.example.com/api/v2.0/system/info
# Returns: {"version": "...", "uptime_seconds": 86400, ...}

A 200 response with a JSON body confirms the TrueNAS API is reachable and the middleware is healthy. A 502 or 503 indicates the TrueNAS middleware (the Python API layer) is restarting; a connection timeout means the appliance is unreachable from the monitoring probe.

Step 2: Create a Vigilmon HTTP Monitor for the TrueNAS API

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://truenas.example.com/api/v2.0/system/info.
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: version (present in all valid TrueNAS API responses).
7. Click Save.

This monitor catches:

• TrueNAS middleware crash or restart (the Python API layer stops responding while the OS remains up)
• Power failure or unplanned reboot of the TrueNAS appliance
• Network interface failure on the TrueNAS host that isolates it from Kubernetes nodes
• TrueNAS SCALE update that temporarily takes the API offline during migration
• Certificate issues on the TrueNAS HTTPS interface that prevent API access

Alert sensitivity: Set to trigger after 1 consecutive failure. When the TrueNAS API is unreachable, the democratic-csi controller cannot provision new PVCs, and volume resize/delete operations also fail. Existing mounted volumes continue working (iSCSI and NFS maintain their sessions independently), but new PVC requests from any workload in the cluster hang.

Step 3: Monitor iSCSI Endpoint Availability via TCP Check

democratic-csi uses iSCSI to provision block storage for databases, Redis instances, and other workloads requiring block-level access. The iSCSI target listens on TCP port 3260 on the TrueNAS appliance. TCP-level monitoring confirms that the iSCSI target is accepting connections — a necessary condition for volume attachment on new pods and nodes:

# Test iSCSI TCP connectivity
nc -zv truenas.example.com 3260
# Returns: Connection to truenas.example.com 3260 port [tcp/iscsi] succeeded!

1. Add Monitor → TCP.
2. Host: truenas.example.com.
3. Port: 3260.
4. Check interval: 2 minutes.
5. Label: TrueNAS iSCSI target.
6. Click Save.

This monitor catches:

• iSCSI target service stopped on TrueNAS (can happen after a pool import failure)
• Firewall rule change blocking port 3260 from Kubernetes nodes
• TrueNAS network interface failure that blocks iSCSI but not the web interface (LAGG failover)
• iSCSI target IP change after a TrueNAS network reconfiguration

iSCSI and the CSI controller are separate failure modes: the iSCSI TCP monitor can fail while the TrueNAS API monitor stays green (iSCSI service stopped, API running), or the API monitor can fail while iSCSI stays up (middleware crashed, existing sessions persist). This distinction helps you triage whether new provisioning will fail, existing volumes are at risk, or both.

Step 4: Monitor NFS Export Availability via TCP Check

For democratic-csi deployments using NFS (common for ReadWriteMany volumes shared across multiple pods), the NFS service listens on TCP port 2049. TCP monitoring confirms the NFS server is accepting mount requests:

# Test NFS TCP connectivity
nc -zv truenas.example.com 2049
# Returns: Connection to truenas.example.com 2049 port [tcp/nfs] succeeded!

1. Add Monitor → TCP.
2. Host: truenas.example.com.
3. Port: 2049.
4. Check interval: 2 minutes.
5. Label: TrueNAS NFS service.
6. Click Save.

Some TrueNAS deployments use a separate storage network interface for iSCSI and NFS traffic (distinct from the management interface used by the web UI and API). Ensure your Vigilmon probes target the storage network IP for iSCSI/NFS monitors, and the management IP for the API and web UI monitors — these may be different hosts.

Step 5: Monitor the TrueNAS Web UI Availability

The TrueNAS web UI is the administration surface for storage administrators — used to create datasets, configure iSCSI targets, review pool health, and approve democratic-csi provisioning requests. If the web UI is unreachable, storage administrators cannot respond to pool degradation events or reconfigure storage:

curl https://truenas.example.com
# Returns HTML containing "TrueNAS" in the page content

1. Add Monitor → HTTP.
2. URL: https://truenas.example.com.
3. Check interval: 5 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: TrueNAS.
7. Label: TrueNAS web UI.
8. Click Save.

When the web UI monitor fails but the iSCSI and NFS TCP monitors stay green, the issue is isolated to the TrueNAS middleware layer — existing volume sessions continue working, but provisioning is blocked and storage admins lose management access. This is a medium-severity alert that needs investigation but does not immediately impact running workloads.

Step 6: Monitor SSL Certificates for the TrueNAS Endpoint

TrueNAS uses HTTPS for its API and web UI. An expired certificate breaks the democratic-csi controller's HTTPS calls to the TrueNAS API — provisioning stops with TLS verification errors:

openssl s_client -connect truenas.example.com:443 2>/dev/null | openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: truenas.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

TrueNAS ships with a self-signed certificate by default. Many democratic-csi deployments configure allowInsecure: true in the driver configuration to work around this — but if you switch to a proper certificate later, ensure the democratic-csi httpConnection.verifySSL setting is updated accordingly. Vigilmon's SSL check works against both Let's Encrypt and self-signed certificates.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | TrueNAS API /api/v2.0/system/info | Non-200 or keyword missing | Check TrueNAS middleware: service middlewared status; reboot if stuck | | iSCSI TCP port 3260 | Connection refused | Check iSCSI service: TrueNAS UI → Sharing → iSCSI → check service status | | NFS TCP port 2049 | Connection refused | Check NFS service: TrueNAS UI → Sharing → NFS → verify service is running | | TrueNAS web UI | Non-200 or keyword missing | Middleware down; use console access to restart middlewared service | | SSL certificate | < 30 days to expiry | Renew TrueNAS certificate; update democratic-csi config if using cert verification |

Alert after: 1 consecutive failure for the API and iSCSI monitors. 2 consecutive failures for NFS and web UI monitors.

Common democratic-csi Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | TrueNAS middleware crash (middlewared stopped) | API and web UI monitors fire; iSCSI/NFS TCP monitors stay green | | TrueNAS appliance power loss or reboot | All monitors fire simultaneously | | iSCSI service stopped after TrueNAS update | iSCSI TCP monitor fires; API and web UI monitors stay green | | NFS export removed or service stopped | NFS TCP monitor fires; iSCSI and API monitors stay green | | TrueNAS network interface failure (management only) | API and web UI fire; iSCSI/NFS (on separate interface) may stay green | | democratic-csi controller pod crash | PVC provisioning stops; TrueNAS monitors all stay green; requires k8s-level monitoring | | Pool degraded (disk failure in RAIDZ) | API and web UI accessible; no external indicator from Vigilmon — use TrueNAS email alerts | | SSL certificate expired | SSL monitor alerts; API calls from CSI controller fail with TLS error | | Storage network IP changed after TrueNAS update | iSCSI and NFS TCP monitors fire; requires DNS or IP update in monitors | | Kubernetes node loses iSCSI session | Affects specific node; global TCP monitor stays green; requires node-level alerting |

democratic-csi transforms TrueNAS from a standalone NAS appliance into the persistent storage backend for your Kubernetes cluster — when it fails, stateful workloads cannot provision new volumes and may not be able to reschedule existing ones. Vigilmon gives you external visibility into the full storage path: the TrueNAS API health, iSCSI and NFS protocol availability, web UI reachability, and SSL certificate validity, so you know the moment the storage layer is degraded and can act before running pods are affected.

Start monitoring democratic-csi in under 5 minutes — register free at vigilmon.online.