Monitoring Capsule Operator with Vigilmon: Tenant Namespace Availability, Admission Webhook Health, Policy Enforcement Status & SSL Certificate Alerts

Capsule is the Kubernetes operator that implements multi-tenancy at the namespace level — grouping namespaces into isolated tenant boundaries with enforced resource quotas, network policies, and RBAC rules without requiring separate clusters for each team. When the Capsule webhook becomes unavailable, every resource creation in the cluster fails validation, causing a complete service disruption across all tenants. When the Capsule controller stops reconciling, tenant policies drift silently from their intended state. Vigilmon gives you external visibility into the Capsule admission webhook, the controller reconciliation health, the Kubernetes API it depends on, and the TLS certificates protecting the webhook endpoint so you know immediately when your multi-tenancy enforcement layer is compromised.

What You'll Build

• An HTTP monitor on the Capsule admission webhook health endpoint
• A monitor for the Capsule controller managing tenant lifecycle and policy enforcement
• Kubernetes API server health monitoring that Capsule depends on
• SSL certificate monitoring for the Capsule webhook TLS endpoint
• An alert runbook mapping Vigilmon findings to Capsule and kubectl remediation commands

Prerequisites

• Capsule v0.7+ installed via Helm or manifests in your Kubernetes cluster
• At least one Tenant resource created
• Network access to your Capsule webhook endpoint from an external monitor
• A free account at vigilmon.online

Step 1: Understand Capsule's Observability Surface

Capsule operates as a Kubernetes admission webhook and controller. The webhook is the most critical component — it intercepts every resource creation and modification to enforce tenant policies:

# Check Capsule controller pod status
kubectl get pods -n capsule-system
kubectl logs -n capsule-system deployment/capsule-controller-manager --tail=30

# Verify webhook configuration is registered
kubectl get validatingwebhookconfigurations | grep capsule
kubectl get mutatingwebhookconfigurations | grep capsule

# Describe the webhook configuration to see the endpoint and failure policy
kubectl describe validatingwebhookconfiguration capsule-validating-webhook-configuration
# Critical field: "Failure Policy: Fail" (default) means webhook unavailability blocks ALL cluster operations

# List all Capsule tenants and their status
kubectl get tenants -A
kubectl describe tenant my-tenant

# Check tenant namespace assignments
kubectl get namespaces --show-labels | grep capsule.clastix.io/tenant

# Capsule controller health endpoint
kubectl port-forward -n capsule-system svc/capsule-controller-manager-metrics-service 8080:8080
curl http://localhost:8080/healthz
# Expected: "ok"

The key observability signals are:

• Admission webhook availability: Capsule's webhook uses failurePolicy: Fail by default, meaning if the webhook is unreachable, Kubernetes rejects ALL CREATE/UPDATE operations in the cluster — not just tenant-related ones. This is the highest-severity failure mode.
• Capsule controller health: The controller reconciles Tenant resources, enforcing resource quotas, network policies, and RBAC bindings across tenant namespaces. Controller failure allows tenants to drift from policy.
• Kubernetes API health: Capsule's controller watches the API server for resource events. API server degradation stalls all Capsule reconciliation.
• TLS certificates: The Capsule webhook requires a valid TLS certificate. An expired webhook certificate causes the webhook endpoint to fail TLS handshake, triggering Fail policy and blocking cluster operations.

Step 2: Monitor the Capsule Admission Webhook

The admission webhook is Capsule's most critical operational dependency. Expose its health endpoint externally:

# Check Capsule webhook service
kubectl get svc -n capsule-system
# Output includes: capsule-controller-manager-metrics-service (port 8080)

# Test the webhook health endpoint via port-forward
kubectl port-forward -n capsule-system svc/capsule-controller-manager-metrics-service 8080:8080
curl http://localhost:8080/healthz
# Expected: "ok"

# Verify the webhook is responding to Kubernetes internally
kubectl auth can-i create namespace --as=system:serviceaccount:default:my-sa

Expose the health endpoint externally via an Ingress:

# capsule-webhook-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: capsule-health
  namespace: capsule-system
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: capsule-health.example.com
    http:
      paths:
      - path: /healthz
        pathType: Exact
        backend:
          service:
            name: capsule-controller-manager-metrics-service
            port:
              number: 8080
  tls:
  - hosts:
    - capsule-health.example.com
    secretName: capsule-health-tls

kubectl apply -f capsule-webhook-health-ingress.yaml

In Vigilmon:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://capsule-health.example.com/healthz.
3. Check interval: 1 minute.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: capsule webhook health.
8. Click Save.

Critical context: Capsule uses failurePolicy: Fail for its admission webhook by default. A failed check here means your entire cluster may be rejecting resource creation operations. Set your alert escalation to P0/immediate for this monitor.

Step 3: Monitor Capsule Controller Reconciliation

The Capsule controller runs the reconciliation loop that enforces tenant policies. While less immediately critical than the webhook, controller failure allows tenant resource quotas and network policies to drift:

# Check controller pod status and restart count
kubectl get pods -n capsule-system -o wide
kubectl get pods -n capsule-system -o jsonpath='{range .items[*]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount}{"\n"}{end}'

# Check controller logs for reconciliation errors
kubectl logs -n capsule-system deployment/capsule-controller-manager --tail=50 | grep -E "error|ERROR|reconcile"

# Check leader election (ensures only one controller is active)
kubectl get leases -n capsule-system
kubectl describe lease -n capsule-system capsule-controller-manager

# Verify tenants are being reconciled
kubectl get tenants -A -o wide
# Check "Age" vs "Last Updated" — stale tenants indicate reconciliation failure

In Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://capsule-health.example.com/readyz.
3. Check interval: 2 minutes.
4. Expected status: 200.
5. Label: capsule controller readiness.
6. Click Save.

The /readyz endpoint is stricter than /healthz — it reflects whether the controller has completed initial cache sync and is ready to serve reconciliation requests.

Step 4: Monitor Kubernetes API Server Availability

Capsule watches the Kubernetes API for resource events. API server degradation stalls all tenant reconciliation and may delay admission webhook processing:

# Check Kubernetes API server health
kubectl get --raw /healthz
# Expected: "ok"

kubectl get --raw /readyz
# Expected: "ok"

# Find API server address
kubectl cluster-info

# Check that Capsule's service account can reach the API
kubectl auth can-i list tenants --as=system:serviceaccount:capsule-system:capsule-controller-manager

In Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://api.example.com:6443/healthz.
3. Check interval: 2 minutes.
4. Expected status: 200.
5. Keyword: ok.
6. Label: kubernetes api server.
7. Click Save.

Add a TCP monitor for a connectivity check independent of HTTP routing:

1. Add Monitor → TCP.
2. Host: api.example.com.
3. Port: 6443.
4. Check interval: 1 minute.
5. Label: kubernetes api tcp.
6. Click Save.

Step 5: Monitor Capsule Webhook TLS Endpoint Directly

The Capsule webhook server itself listens on port 9443 inside the cluster. Its TLS certificate is injected by cert-manager. Monitor the certificate expiry and the underlying TCP connectivity:

# Check the Capsule webhook service
kubectl get svc -n capsule-system capsule-webhook-service

# Capsule webhook listens on port 9443 internally
# The MutatingWebhookConfiguration points to this service

# Check the TLS certificate used by the webhook
kubectl get secret -n capsule-system capsule-webhook-server-cert -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -dates

# List cert-manager Certificate resources for Capsule
kubectl get certificates -n capsule-system

Expose the Capsule webhook port for SSL monitoring:

# capsule-webhook-external-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: capsule-webhook-external
  namespace: capsule-system
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: capsule
  ports:
  - name: webhook
    port: 9443
    targetPort: 9443

In Vigilmon:

1. Add Monitor → SSL Certificate.
2. Domain: capsule-webhook-external.example.com.
3. Port: 9443.
4. Alert when expiry is within: 30 days.
5. Alert again: 14 days, 7 days, 3 days, 1 day.
6. Click Save.

Also add an SSL monitor for the health endpoint Ingress:

1. Add Monitor → SSL Certificate.
2. Domain: capsule-health.example.com.
3. Alert when expiry is within: 30 days.
4. Click Save.

Step 6: Monitor Tenant Namespace Availability (Application-Level)

Beyond infrastructure monitoring, verify that tenant namespaces are accessible to their users:

# Verify tenant namespace exists and has correct labels
kubectl get namespace tenant1-prod -o yaml | grep capsule
# Should show: capsule.clastix.io/tenant: my-tenant

# Check namespace resource quotas are applied
kubectl get resourcequota -n tenant1-prod

# Verify network policies are enforced
kubectl get networkpolicies -n tenant1-prod

# Test that cross-tenant access is blocked (policy enforcement check)
kubectl auth can-i list pods --namespace=tenant2-prod \
  --as=system:serviceaccount:tenant1-prod:default
# Expected: "no"

If your tenants host web services, add application-level HTTP monitors:

1. Add Monitor → HTTP.
2. URL: https://app.tenant1.example.com/health.
3. Check interval: 5 minutes.
4. Expected status: 200.
5. Label: tenant1 application health.
6. Click Save.

Step 7: Configure Alerting and Runbook

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Immediate Action | |---|---|---| | Capsule webhook health | Non-200 | P0 — check kubectl get pods -n capsule-system; kubectl logs deployment/capsule-controller-manager; webhook failurePolicy:Fail may be blocking ALL cluster operations | | Capsule controller readiness | Non-200 | kubectl get pods -n capsule-system; check leader election; verify RBAC allows controller to watch resources | | Kubernetes API HTTP | Non-200 | Check API server health; Capsule reconciliation stalled; check node health | | Kubernetes API TCP | Connection refused | P0 — Kubernetes cluster API unavailable; escalate immediately | | SSL certificate (webhook) | < 30 days | kubectl get certificates -n capsule-system; trigger cert-manager renewal; expired cert will block ALL cluster admission operations | | Tenant application health | Non-200 | Check tenant namespace; verify Capsule policies allow the application's RBAC and network access |

Escalation policy: The Capsule webhook health alert is P0 — set immediate paging for your platform team. A Fail policy webhook failure blocks resource creation cluster-wide, affecting every team using the cluster, not just Capsule tenants. SSL certificate expiry on the webhook is also P0 for the same reason.

Common Capsule Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Capsule controller pod OOM killed during large tenant reconciliation | Webhook health monitor catches pod restart; controller readiness goes down temporarily | | cert-manager fails to renew webhook TLS certificate | SSL monitor catches 30 days before expired cert causes all admission to fail | | Kubernetes API server upgrade causes brief webhook connectivity interruption | API TCP monitor catches connection refused; Capsule webhook retries buffer brief unavailability | | Capsule namespace quota CRD validation webhook returns error for all namespaces | Webhook health check stays green but namespace creates return 500; surface via application monitors | | Network policy misconfiguration blocks Capsule controller from watching resources | Controller readiness goes down; tenant policy drift begins; Vigilmon alerts within 2 min | | Capsule upgraded with breaking admission webhook changes during business hours | Webhook health goes down during rollout; surfaces before cluster-wide resource creation failures are reported | | Tenant abuses resource quota causing controller reconciliation backlog | Controller readiness still green but tenant config shows stale Last Updated | | Capsule running as single replica and node drains for maintenance | Webhook health catches pod eviction → failurePolicy:Fail takes down cluster admission |

Capsule provides namespace-level multi-tenancy without separate clusters, but its admission webhook creates a single point of failure for the entire cluster: when the webhook is down with failurePolicy: Fail, no resource can be created or updated anywhere in the cluster. Vigilmon provides external monitoring across all critical layers — the admission webhook health, controller reconciliation readiness, Kubernetes API connectivity, and the TLS certificates that protect every webhook connection. When any layer degrades, you know within minutes before tenant teams start reporting that their deployments are failing.

Start monitoring Capsule in under 5 minutes — register free at vigilmon.online.