Monitoring Capsule Kubernetes Multi-Tenancy with Vigilmon

Capsule is a Kubernetes operator that implements multi-tenancy through the concept of Tenants—groups of namespaces with shared resource quotas, RBAC policies, network isolation, and admission controls. A single Capsule Tenant can govern dozens of namespaces for a single team or customer. When Capsule's admission webhook goes down or the operator crashes, tenants can create namespaces that bypass policy enforcement, exceed quota limits, and reach resources they shouldn't touch.

This tutorial shows how to monitor Capsule's operator health, tenant resource quota utilization, policy enforcement status, and admission webhook availability using Vigilmon—so policy enforcement never silently fails.

Why Capsule needs external monitoring

Capsule enforces multi-tenancy through Kubernetes admission webhooks and a controller. It can fail in ways that have serious security and reliability consequences:

• Admission webhook becomes unavailable — when the Capsule webhook service goes down, Kubernetes falls back to allowing or denying requests based on failurePolicy; if set to Ignore, tenants can create resources that bypass all Capsule policies
• Operator crash leaves Tenant objects unreconciled — new Tenants are created but their RBAC, quotas, and namespace bindings are never applied; tenants operate in uncontrolled namespaces
• Tenant quota violations go undetected — a namespace within a Tenant approaches or hits its ResourceQuota; without monitoring, teams hit hard stops unexpectedly
• Network policy enforcement gaps — Capsule enforces network isolation between tenants; if policies fail to apply, cross-tenant traffic becomes possible without any alert
• TenantUser binding breaks — Capsule maps users to tenants via TenantUser; a bug or webhook failure causes a user to lose access or gain unexpected access to other namespaces

What you'll need

• A Kubernetes cluster with Capsule installed (Helm)
• Capsule operator running in capsule-system namespace
• A free Vigilmon account

Step 1: Install Capsule and verify operator health

Install Capsule via Helm:

helm repo add projectcapsule https://projectcapsule.github.io/charts
helm repo update

helm install capsule projectcapsule/capsule \
  --namespace capsule-system \
  --create-namespace \
  --set manager.resources.requests.cpu=100m \
  --set manager.resources.requests.memory=128Mi

Verify the operator and webhook are running:

kubectl get pods -n capsule-system
# NAME                               READY   STATUS    RESTARTS   AGE
# capsule-controller-manager-xxx     1/1     Running   0          5m

# Check the Capsule webhook
kubectl get validatingwebhookconfigurations | grep capsule
kubectl get mutatingwebhookconfigurations | grep capsule

Check that Tenants are healthy:

kubectl get tenants -o wide
# NAME      STATE     NAMESPACES   AGE
# team-a    Active    3            2d
# team-b    Active    5            2d

Step 2: Expose Capsule metrics

Capsule exposes Prometheus metrics through its controller-manager. Create a metrics service:

# capsule-metrics-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: capsule-metrics
  namespace: capsule-system
  labels:
    app.kubernetes.io/component: controller-manager
spec:
  selector:
    app.kubernetes.io/component: controller-manager
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080

Verify:

kubectl port-forward -n capsule-system svc/capsule-metrics 8080:8080
curl http://localhost:8080/metrics | grep capsule

Key metrics to watch:

# HELP capsule_tenants Total number of Tenants
capsule_tenants 12

# HELP capsule_namespaces_per_tenant Namespaces assigned per Tenant
capsule_namespaces_per_tenant{tenant="team-a"} 3

# HELP controller_runtime_reconcile_errors_total Number of reconcile errors
controller_runtime_reconcile_errors_total{controller="tenant"} 0

Step 3: Build a Capsule tenant health bridge

Create a health service that monitors tenant status and quota utilization:

# capsule-health-bridge.py
import os, subprocess, json
from flask import Flask, jsonify

app = Flask(__name__)
QUOTA_WARNING_THRESHOLD = float(os.environ.get("QUOTA_WARNING_PCT", "85"))

def get_tenant_status():
    result = subprocess.run(
        ["kubectl", "get", "tenants", "-o", "json"],
        capture_output=True, text=True, timeout=15
    )
    if result.returncode != 0:
        return None, result.stderr

    data = json.loads(result.stdout)
    inactive = []
    for item in data.get("items", []):
        name = item["metadata"]["name"]
        state = item.get("status", {}).get("state", "Unknown")
        if state != "Active":
            inactive.append(f"{name}: {state}")

    return {"total": len(data["items"]), "inactive": inactive}, None

def get_quota_utilization():
    result = subprocess.run(
        ["kubectl", "get", "resourcequota", "--all-namespaces", "-o", "json"],
        capture_output=True, text=True, timeout=15
    )
    if result.returncode != 0:
        return None, result.stderr

    data = json.loads(result.stdout)
    warnings = []

    for item in data.get("items", []):
        ns = item["metadata"]["namespace"]
        hard = item.get("status", {}).get("hard", {})
        used = item.get("status", {}).get("used", {})

        for resource in ["requests.cpu", "requests.memory", "pods", "persistentvolumeclaims"]:
            if resource in hard and resource in used:
                hard_val = hard[resource]
                used_val = used[resource]
                try:
                    hard_num = float(hard_val.rstrip("m").rstrip("Ki").rstrip("Mi").rstrip("Gi"))
                    used_num = float(used_val.rstrip("m").rstrip("Ki").rstrip("Mi").rstrip("Gi"))
                    if hard_num > 0:
                        pct = (used_num / hard_num) * 100
                        if pct >= QUOTA_WARNING_THRESHOLD:
                            warnings.append(f"{ns}/{resource}: {pct:.0f}% used ({used_val}/{hard_val})")
                except (ValueError, AttributeError):
                    pass

    return warnings, None

@app.route("/health")
def health():
    tenants, err = get_tenant_status()
    if err:
        return jsonify({"status": "error", "message": f"Cannot list tenants: {err}"}), 503

    if tenants["inactive"]:
        return jsonify({
            "status": "error",
            "message": f"{len(tenants['inactive'])} inactive tenant(s)",
            "inactive_tenants": tenants["inactive"]
        }), 503

    quota_warnings, err = get_quota_utilization()
    if err:
        return jsonify({"status": "error", "message": f"Cannot check quotas: {err}"}), 503

    if quota_warnings:
        return jsonify({
            "status": "warning",
            "message": f"{len(quota_warnings)} quota(s) above {QUOTA_WARNING_THRESHOLD}%",
            "quota_warnings": quota_warnings[:10]
        }), 200

    return jsonify({
        "status": "ok",
        "tenants": tenants["total"],
        "quota_warnings": 0
    }), 200

@app.route("/tenants")
def tenant_list():
    tenants, err = get_tenant_status()
    if err:
        return jsonify({"status": "error", "message": err}), 503
    return jsonify(tenants), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Deploy:

# capsule-bridge.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: capsule-health-bridge
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: capsule-health-bridge
  template:
    metadata:
      labels:
        app: capsule-health-bridge
    spec:
      serviceAccountName: capsule-health-reader
      containers:
        - name: bridge
          image: python:3.12-slim
          command: ["python", "/app/capsule-health-bridge.py"]
          env:
            - name: QUOTA_WARNING_PCT
              value: "85"
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: capsule-health-reader
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: capsule-health-reader
rules:
  - apiGroups: ["capsule.clastix.io"]
    resources: ["tenants"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["resourcequotas", "namespaces"]
    verbs: ["list", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: capsule-health-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: capsule-health-reader
subjects:
  - kind: ServiceAccount
    name: capsule-health-reader
    namespace: monitoring

Step 4: Monitor the Capsule admission webhook

The most critical Capsule component to monitor is its admission webhook. If it's unreachable, Capsule's multi-tenancy policies stop being enforced.

Check webhook configuration:

# Get the webhook service
kubectl get validatingwebhookconfiguration capsule-validating-webhook-configuration \
  -o jsonpath='{.webhooks[*].failurePolicy}'
# Should output: Fail Fail Fail
# If any shows "Ignore", policy enforcement bypasses on webhook failure

Add a webhook health check to Vigilmon:

Create a simple endpoint that verifies the webhook is reachable (Capsule's webhook returns 400 for malformed requests, which proves it's alive):

# Test webhook directly from within the cluster
kubectl run webhook-test --rm -it --image=curlimages/curl -- \
  curl -sk https://capsule-controller-manager-metrics-service.capsule-system:9443/healthz

Step 5: Set up Vigilmon monitors

Monitor 1 — Capsule operator metrics liveness:

1. Log in to vigilmon.online → Add Monitor
2. Type: HTTP
3. URL: your Capsule metrics endpoint (via ingress)
4. Keyword: capsule_tenants
5. Interval: 1 minute

Monitor 2 — Tenant and quota health bridge:

1. Add Monitor → HTTP
2. URL: https://your-cluster/capsule-health
3. Expected status: 200
4. Interval: 2 minutes

Monitor 3 — Capsule operator /healthz:

1. Add Monitor → HTTP
2. URL: https://capsule-controller-manager.capsule-system.svc:9443/healthz
3. Expected status: 200
4. Interval: 1 minute

Step 6: Prometheus alerting for Capsule

# capsule-alerts.yaml
groups:
  - name: capsule-multitenancy
    rules:
      - alert: CapsuleReconcileErrors
        expr: rate(controller_runtime_reconcile_errors_total{controller="tenant"}[5m]) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Capsule Tenant reconciler errors"
          description: "Capsule is failing to reconcile Tenant objects — RBAC and quotas may not be applied"

      - alert: CapsuleTenantInactive
        expr: capsule_tenant_state{state!="Active"} > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "One or more Capsule Tenants are not Active"
          description: "Tenant {{ $labels.tenant }} is in state {{ $labels.state }}"

      - alert: CapsuleQuotaNearLimit
        expr: |
          (kube_resourcequota{type="used"} / kube_resourcequota{type="hard"}) > 0.90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tenant namespace approaching resource quota limit"
          description: "{{ $labels.namespace }}/{{ $labels.resource }} is at {{ $value | humanizePercentage }}"

Step 7: Configure Vigilmon alert routing

1. In Vigilmon → Alerts → Notification Channels
2. Add channels per severity:
   • Critical (webhook down, operator crash): page on-call immediately
   • Warning (quota at 85%): notify team lead via email
3. For the tenant health bridge monitor:
   • Alert after: 1 failure
   • Recovery: send notification (important — teams need to know when policy enforcement resumes)

What you're now monitoring

With this setup in place:

• Operator health — reconciler errors are caught within 2 minutes
• Tenant status — inactive or misconfigured tenants surface immediately
• Quota utilization — teams get early warning before hitting hard limits
• Admission webhook liveness — the policy enforcement layer is always verified
• Network policy gaps — cross-tenant access monitoring is backed by Capsule's namespace isolation

Capsule makes Kubernetes multi-tenancy manageable at scale—but its policy enforcement is only as reliable as the webhook and operator behind it. Vigilmon ensures you always know when that enforcement layer is healthy.