tutorial

Monitoring Loft Virtual Clusters with Vigilmon

Loft manages Kubernetes virtual clusters and shared namespaces for multi-tenant dev environments — but vcluster provisioning failures, sleep mode edge cases, and SSO integration breaks can block your whole team silently. Here's how to monitor Loft with Vigilmon.

Loft turns a single Kubernetes cluster into a self-service platform where teams can spin up isolated virtual clusters (vclusters) on demand — with sleep mode, SSO, quota enforcement, and a management UI on top. But when Loft's own API goes down, vcluster provisioning stalls, or sleep mode fails to resume a cluster on demand, developers hit opaque errors with no clear signal. Vigilmon adds external health checks for Loft's API server, vcluster endpoints, SSO integration, and sleep/wake cycle monitoring so platform outages surface immediately.

What You'll Set Up

  • Loft API server health and availability monitors
  • Virtual cluster endpoint uptime checks
  • Sleep mode and wake cycle heartbeat monitoring
  • SSO integration health checks
  • Namespace quota utilization alerts

Prerequisites

  • Loft installed on a Kubernetes cluster (Loft 3.x or later)
  • At least one virtual cluster running (loft create vcluster)
  • A free Vigilmon account

Step 1: Monitor the Loft API Server

Loft runs its own management API server (typically at port 8080 for HTTP and 8443 for HTTPS within the cluster, or exposed via ingress). This is the entry point for all CLI operations, the UI, and vcluster provisioning requests.

Add an HTTP monitor for the Loft health endpoint:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your Loft URL: 
    https://loft.yourdomain.com/healthz
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Enable Monitor SSL certificate and set the alert threshold to 21 days.
  7. Click Save.

Add a second monitor for the API readiness endpoint:

  • URL: https://loft.yourdomain.com/readyz
  • Interval: 2 minutes
  • Expected status: 200

If you're running Loft without a public ingress, expose a health check internally and monitor it via TCP port instead:

  1. Click Add Monitor → TCP Port.
  2. Host: your Loft pod's ClusterIP or node address.
  3. Port: 8443.
  4. Interval: 1 minute.

Step 2: Monitor Virtual Cluster Endpoints

Each vcluster provisioned by Loft exposes its own Kubernetes API server through Loft's proxy layer. When the vcluster's underlying pods crash or the Loft proxy breaks, the virtual cluster's API becomes unreachable — even if the management API is healthy.

Add a monitor for each critical vcluster:

  1. Get the vcluster's accessible endpoint: 
    loft use vcluster my-cluster --print
# Outputs the kubeconfig with server URL
  2. In Vigilmon, add an HTTP / HTTPS monitor for: 
    https://loft.yourdomain.com/kubernetes/cluster/my-cluster/healthz
    Or use the direct vcluster API server URL from the kubeconfig.
  3. Set interval to 2 minutes, expected status 200.
  4. Add the Authorization header with a Loft access token if the endpoint requires authentication.

Repeat for each production or staging vcluster. Development vclusters that sleep frequently can be monitored with a longer interval (5 minutes) to avoid triggering false alerts during sleep mode.

Step 3: Heartbeat Monitoring for Sleep Mode and Wake Cycles

Loft's sleep mode hibernates idle vclusters to save resources. When a user accesses a sleeping cluster, Loft wakes it automatically — but wake failures are silent from the user's perspective: the request just hangs.

Add a cron heartbeat that wakes and probes each sleep-enabled vcluster on schedule:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 30 minutes.
  3. Copy the heartbeat URL.
  4. Create a wake-and-probe script:
#!/bin/bash
# loft-wake-check.sh

VCLUSTER_NAME="staging"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Wake the cluster if sleeping (loft wakes on kubectl access)
READY=$(kubectl --context "loft_${VCLUSTER_NAME}" get nodes \
  --timeout=60s --no-headers 2>/dev/null | grep -c Ready)

if [ "$READY" -gt 0 ]; then
  curl -s "$HEARTBEAT_URL"
else
  echo "vcluster $VCLUSTER_NAME failed to wake or has no ready nodes"
fi

Run this script every 20 minutes via cron. If a cluster gets stuck in a wake cycle (which can happen when the underlying host node has scheduling pressure), the heartbeat stops firing and Vigilmon alerts.

Step 4: Monitor SSO Integration Health

Loft integrates with identity providers (Dex, OIDC, GitHub, GitLab, LDAP) for team authentication. SSO breaks silently when tokens expire, provider configurations drift, or network routes to the IdP change.

Add an HTTP monitor for the Loft authentication endpoint:

  1. In Vigilmon, add an HTTP / HTTPS monitor for: 
    https://loft.yourdomain.com/auth/login
  2. Set Expected HTTP status to 200 (the login page renders, even if the IdP isn't fully reachable).
  3. Set interval to 5 minutes.

For deeper SSO validation, monitor the OIDC discovery endpoint of your identity provider directly:

https://your-idp.yourdomain.com/.well-known/openid-configuration

Add a Vigilmon monitor for this URL with an expected 200 status. If the IdP goes down or its certificate expires, Loft's SSO breaks for all users — catching this upstream is faster than waiting for login failure reports.

If you're using Dex (Loft's bundled identity provider):

# Check Dex health endpoint
curl https://loft.yourdomain.com/dex/healthz

Add this as a third HTTP monitor with a 5 minute interval.

Step 5: Namespace Quota Monitoring

Loft enforces namespace and vcluster quotas to prevent runaway resource consumption. When a team's namespace approaches its quota, new deployments fail with cryptic Forbidden errors. Monitor quota utilization to catch this before it blocks teams.

Add a cron heartbeat that checks quota usage:

#!/bin/bash
# loft-quota-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
NAMESPACE="team-alpha"
THRESHOLD=80  # Alert if usage exceeds 80%

# Check CPU quota
CPU_USED=$(kubectl get resourcequota -n "$NAMESPACE" -o jsonpath='{.items[0].status.used.cpu}' | sed 's/m//')
CPU_HARD=$(kubectl get resourcequota -n "$NAMESPACE" -o jsonpath='{.items[0].status.hard.cpu}' | sed 's/m//')

# Check memory quota
MEM_USED=$(kubectl get resourcequota -n "$NAMESPACE" -o jsonpath='{.items[0].status.used.memory}' | sed 's/Mi//')
MEM_HARD=$(kubectl get resourcequota -n "$NAMESPACE" -o jsonpath='{.items[0].status.hard.memory}' | sed 's/Mi//')

CPU_PCT=$(( CPU_USED * 100 / CPU_HARD ))
MEM_PCT=$(( MEM_USED * 100 / MEM_HARD ))

if [ "$CPU_PCT" -lt "$THRESHOLD" ] && [ "$MEM_PCT" -lt "$THRESHOLD" ]; then
  curl -s "$HEARTBEAT_URL"
else
  echo "Quota warning: CPU=${CPU_PCT}% MEM=${MEM_PCT}% in namespace ${NAMESPACE}"
fi

Set a Vigilmon cron heartbeat with a 15 minute interval. Create one quota monitor per critical team namespace.

Step 6: Configure Alert Channels for Platform Teams

Loft outages affect multiple developer teams simultaneously, so platform team alert routing matters:

  1. Go to Alert Channels in Vigilmon and configure a Slack webhook pointing to #platform-alerts.
  2. Add an escalation email to the platform team lead for API server monitors.
  3. Set Consecutive failures before alert to 2 for vcluster monitors (Loft's proxy layer can return transient errors during reconciliation).

Add maintenance window automation to your Loft upgrade pipeline:

#!/bin/bash
# Before running a Loft upgrade

MONITOR_ID="your-loft-api-monitor-id"
VIGILMON_API_KEY="your-key"

# Open maintenance window
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer $VIGILMON_API_KEY" \
  -d "{\"monitor_id\": \"$MONITOR_ID\", \"duration_minutes\": 15}"

# Perform Loft upgrade
helm upgrade loft loft --namespace loft --version 3.x.x

# Verify Loft is healthy after upgrade
kubectl wait --for=condition=ready pod -l app=loft -n loft --timeout=120s

The maintenance window prevents alert spam during planned upgrades while still catching unexpected outages if the upgrade fails.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health | /healthz on Loft API | Loft API server crash | | HTTP readiness | /readyz on Loft API | Loft component failure | | HTTP endpoint | vcluster API proxy URL | vcluster unavailable, proxy break | | Cron heartbeat | Sleep/wake probe script | Wake cycle failure, cluster stuck | | HTTP health | IdP OIDC discovery URL | SSO provider outage | | Cron heartbeat | Namespace quota check | Quota exhaustion before teams notice |

Loft's virtual cluster model gives platform teams fine-grained control over shared Kubernetes resources — but the management plane, SSO integrations, and sleep cycle automation add components that can fail independently of the underlying cluster. Vigilmon gives you the external watchdog layer that catches those failures before they turn into platform-wide outages or silent developer productivity blockers.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles