tutorial

Monitoring vCluster with Vigilmon

vCluster lets you run virtual Kubernetes clusters inside a host cluster — but each vCluster has its own API server, sync controller, and network layer. Here's how to monitor every critical component with Vigilmon.

vCluster by Loft Labs lets you spin up fully isolated Kubernetes clusters inside a namespace of a host cluster — no separate VMs, no separate etcd, dramatically lower cost. Each virtual cluster has its own API server, scheduler, and controller manager, but shares the host's nodes and networking. That layered architecture means you have twice as many things to monitor. Vigilmon gives you the uptime and alerting layer to keep tabs on every vCluster API server, sync controller, and tenant workload endpoint simultaneously.

What You'll Set Up

  • Vigilmon HTTP checks on each vCluster's API server health endpoint
  • Sync controller status monitoring via the vCluster status API
  • Resource quota and host cluster connectivity checks
  • Tenant application endpoint monitoring per vCluster
  • Heartbeat monitoring for vCluster lifecycle automation pipelines

Prerequisites

  • Kubernetes host cluster (1.24+) with vcluster CLI installed
  • At least one running vCluster instance
  • A free Vigilmon account

Step 1: Install vCluster and Verify Health

Install the vCluster CLI and create your first virtual cluster:

# Install vCluster CLI
curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64"
chmod +x vcluster && sudo mv vcluster /usr/local/bin

# Create a vCluster in the "team-alpha" namespace
vcluster create team-alpha --namespace team-alpha

# List running vClusters
vcluster list
# NAME        NAMESPACE    STATUS    VERSION    CONNECTED    CREATED
# team-alpha  team-alpha   Running   0.19.0     true         2m ago

Check the vCluster pods in the host cluster:

kubectl get pods -n team-alpha
# NAME                                         READY   STATUS    RESTARTS   AGE
# coredns-68b9d4b8b6-jwzlt-x-kube-system-...  1/1     Running   0          2m
# team-alpha-0                                 2/2     Running   0          2m

Step 2: Expose the vCluster API Server

Each vCluster runs its own API server. To monitor it externally, expose it via a LoadBalancer or Ingress:

# Port-forward for local testing
vcluster connect team-alpha --namespace team-alpha -- kubectl cluster-info

For production, expose via a service:

# vcluster-api-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: team-alpha-api
  namespace: team-alpha
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: team-alpha.vclusters.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: team-alpha
            port:
              number: 443

Apply and verify:

kubectl apply -f vcluster-api-ingress.yaml
curl -k https://team-alpha.vclusters.yourdomain.com/healthz
# ok

Step 3: Add Vigilmon API Server Health Checks

With the vCluster API server accessible, configure Vigilmon:

3a. vCluster API Server Health

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the URL: https://team-alpha.vclusters.yourdomain.com/healthz
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Set Expected response body contains to ok.
  7. Click Save.

Repeat for each vCluster — give each monitor a meaningful name like vCluster: team-alpha API.

3b. Liveness and Readiness Probes

The vCluster API server also exposes /livez and /readyz:

curl -k https://team-alpha.vclusters.yourdomain.com/livez
# ok

curl -k https://team-alpha.vclusters.yourdomain.com/readyz
# ok

Add a separate Vigilmon monitor for /readyz — it checks internal controllers and will fail before the cluster becomes fully unusable.

Step 4: Monitor the Sync Controller

The vCluster sync controller is the most critical component — it synchronizes resources between the virtual and host cluster. Monitor it via the vCluster metrics endpoint:

# Connect to the vCluster and check sync controller status
vcluster connect team-alpha --namespace team-alpha -- \
  kubectl get pods -n kube-system -l app=vcluster

The vCluster StatefulSet also exposes metrics. Add a scrape job to Prometheus:

# prometheus.yml snippet
scrape_configs:
  - job_name: 'vcluster-team-alpha'
    static_configs:
      - targets: ['team-alpha.team-alpha.svc.cluster.local:8443']
    scheme: https
    tls_config:
      insecure_skip_verify: true
    metrics_path: /metrics

Key sync controller metrics:

| Metric | Description | |--------|-------------| | vcluster_sync_controller_queue_length | Pending sync operations (should be near 0) | | vcluster_sync_errors_total | Cumulative sync failures | | vcluster_synced_objects_total | Total objects successfully synced | | vcluster_fake_nodes_total | Virtual nodes registered in vCluster |

Alert when vcluster_sync_errors_total rate exceeds 5 errors/minute.

Step 5: Host Cluster Connectivity Check

Each vCluster must maintain connectivity to the host cluster's API server to sync resources. Write a connectivity check script:

#!/bin/bash
# vcluster-connectivity-check.sh
VCLUSTER_NAME="team-alpha"
NAMESPACE="team-alpha"
VIGILMON_HEARTBEAT="https://vigilmon.online/api/v1/heartbeat/<YOUR_TOKEN>"

# Connect to vCluster and verify it can reach host cluster
vcluster connect "$VCLUSTER_NAME" --namespace "$NAMESPACE" -- \
  kubectl get nodes --request-timeout=5s > /dev/null 2>&1

if [ $? -eq 0 ]; then
    echo "vCluster $VCLUSTER_NAME: host connectivity OK"
    curl -s "$VIGILMON_HEARTBEAT" > /dev/null
else
    echo "ERROR: vCluster $VCLUSTER_NAME cannot reach host cluster"
    exit 1
fi

Deploy as a Kubernetes CronJob on the host cluster:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vcluster-connectivity-check
  namespace: default
spec:
  schedule: "*/5 * * * *"  # every 5 minutes
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: vcluster-monitor
          containers:
          - name: checker
            image: loftsh/vcluster:0.19.0
            command: ["/bin/bash", "/scripts/vcluster-connectivity-check.sh"]
            volumeMounts:
            - name: scripts
              mountPath: /scripts
          volumes:
          - name: scripts
            configMap:
              name: vcluster-monitor-scripts
          restartPolicy: OnFailure

Step 6: Resource Quota Monitoring

vCluster can enforce resource quotas that limit tenant workloads. Monitor quota utilization to prevent tenants from hitting hard limits:

# Connect to the vCluster and check resource quotas
vcluster connect team-alpha --namespace team-alpha -- \
  kubectl describe resourcequota -A

Write a quota utilization check:

#!/usr/bin/env python3
# vcluster_quota_check.py
import subprocess
import json
import sys

QUOTA_THRESHOLD = 0.85  # alert at 85% utilization

def get_quota_usage(vcluster_name, namespace):
    result = subprocess.run(
        ["vcluster", "connect", vcluster_name, "--namespace", namespace,
         "--", "kubectl", "get", "resourcequota", "-A", "-o", "json"],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

def main():
    quotas = get_quota_usage("team-alpha", "team-alpha")
    warnings = []

    for item in quotas.get("items", []):
        name = item["metadata"]["name"]
        ns = item["metadata"]["namespace"]
        hard = item["status"].get("hard", {})
        used = item["status"].get("used", {})

        for resource, limit in hard.items():
            if resource in used:
                util = _parse_quantity(used[resource]) / _parse_quantity(limit)
                if util > QUOTA_THRESHOLD:
                    warnings.append(f"{ns}/{name}: {resource} at {util:.0%}")

    if warnings:
        print("QUOTA WARNING: " + ", ".join(warnings))
        sys.exit(1)

    print("OK: all quotas within limits")

def _parse_quantity(q):
    q = str(q)
    if q.endswith("m"):
        return int(q[:-1])
    if q.endswith("Ki"):
        return int(q[:-2]) * 1024
    if q.endswith("Mi"):
        return int(q[:-2]) * 1024 ** 2
    if q.endswith("Gi"):
        return int(q[:-2]) * 1024 ** 3
    return int(q)

if __name__ == "__main__":
    main()

Step 7: Tenant Application Monitoring

Each vCluster hosts tenant workloads. Monitor tenant application endpoints directly from Vigilmon as though they were independent clusters:

  1. Get the external IP or hostname of a tenant service:
vcluster connect team-alpha --namespace team-alpha -- \
  kubectl get svc -A
  1. Add a Vigilmon monitor for each critical tenant service:
    • Type: HTTP / HTTPS
    • URL: https://app.team-alpha.yourdomain.com/health
    • Check interval: 1 minute
    • Tags: tenant:team-alpha, env:production

Using tags lets you group all monitors for a single tenant and see their aggregate status at a glance.

Alerting Configuration

Set up alert channels in Vigilmon Settings → Notifications:

  • Slack: #platform-alerts for API server and sync controller failures
  • Email: tenant team leads for their own quota alerts
  • PagerDuty: on-call engineers for API server readyz failures lasting more than 2 minutes
  • Webhooks: trigger auto-remediation scripts (e.g., restart vCluster pod) on persistent failures

Configure per-monitor alert policies so that a transient tenant app failure doesn't page the platform team, but a vCluster API server failure always does.

Troubleshooting

vCluster API server unreachable

kubectl logs -n team-alpha statefulset/team-alpha -c syncer
kubectl describe statefulset -n team-alpha team-alpha

Common causes: host cluster node pressure, PVC not bound, network policy blocking ingress.

Sync controller queue growing High vcluster_sync_controller_queue_length indicates the syncer is falling behind. Check:

kubectl top pods -n team-alpha

Increase CPU/memory limits in the vCluster Helm values:

syncer:
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 1000m
      memory: 1Gi

Tenant pods not syncing to host

vcluster connect team-alpha --namespace team-alpha -- \
  kubectl get events -A --sort-by='.lastTimestamp' | tail -20

Summary

You now have complete vCluster observability with Vigilmon:

  • API server health checks via /healthz and /readyz for each vCluster
  • Sync controller metrics tracked via Prometheus for lag and error detection
  • Host connectivity heartbeats every 5 minutes via CronJob
  • Resource quota alerts at 85% utilization before tenants hit hard limits
  • Tenant application monitors with per-tenant tagging for grouped status views

vCluster gives you isolated multi-tenant Kubernetes — Vigilmon ensures every layer of that isolation stays healthy.

Ready to monitor your vCluster deployments? Create a free Vigilmon account and add your first vCluster health check in minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles