Monitoring Kamaji with Vigilmon: Tenant Control Plane API Availability, etcd Datastore Health, Kamaji Controller Sync & SSL Certificate Alerts

Kamaji is the Kubernetes operator that runs tenant Kubernetes control planes as pods inside a management cluster, allowing you to host dozens or hundreds of isolated control planes on shared infrastructure without provisioning dedicated virtual machines for each one. When a tenant control plane pod crashes, all users of that tenant cluster lose access to their Kubernetes API. When the shared etcd datastore that Kamaji uses for tenant control plane state becomes degraded, all hosted control planes are simultaneously affected. Vigilmon gives you external visibility into each tenant control plane API, the shared etcd health, the Kamaji controller reconciliation loop, and the TLS certificates that protect every hosted API endpoint so you know immediately when your hosted control plane service is degraded.

What You'll Build

• HTTP monitors on tenant control plane API endpoints to detect per-tenant failures
• A monitor for the shared etcd datastore that backs all hosted control planes
• A monitor for the Kamaji controller managing tenant control plane lifecycle
• SSL certificate monitoring for all tenant API server TLS endpoints
• An alert runbook mapping Vigilmon findings to Kamaji remediation commands

Prerequisites

• Kamaji v1.0+ installed in a management cluster
• At least one TenantControlPlane resource created and running
• The etcd datastore configured (multi-tenant etcd or dedicated per-tenant datastores)
• Network access to tenant control plane API endpoints from an external monitor
• A free account at vigilmon.online

Step 1: Understand Kamaji's Observability Surface

Kamaji's architecture has three layers to monitor: the Kamaji controller, the shared datastore, and each individual tenant control plane:

# Check Kamaji controller health
kubectl get pods -n kamaji-system
kubectl logs -n kamaji-system deployment/kamaji --tail=30

# List all tenant control planes and their status
kubectl get tenantcontrolplanes -A
kubectl describe tenantcontrolplane my-tenant -n kamaji-tenants
# Look for: "Status.Conditions: Ready=True" and "Status.APIServer.URL"

# Get the tenant control plane API URL
kubectl get tenantcontrolplane my-tenant -n kamaji-tenants \
  -o jsonpath='{.status.controlPlaneEndpoint.host}'

# Test the tenant API server health endpoint
TENANT_API=$(kubectl get tcp my-tenant -n kamaji-tenants \
  -o jsonpath='https://{.status.controlPlaneEndpoint.host}:{.status.controlPlaneEndpoint.port}')
curl -k $TENANT_API/healthz
# Expected: "ok"

# Check etcd datastore status
kubectl get etcdclusters -n kamaji-system

The key observability signals are:

• Tenant control plane API: Each TenantControlPlane runs kube-apiserver, kube-controller-manager, and kube-scheduler as pods. A pod crash or OOM kill takes the entire tenant API offline.
• Shared etcd datastore: Kamaji typically uses a multi-tenant etcd ring shared across all hosted control planes. etcd degradation cascades to every tenant simultaneously.
• Kamaji controller: The controller manages the full lifecycle of tenant control planes — creating, updating, and restarting them. Controller failure means tenant control planes can no longer be created, upgraded, or recovered automatically.
• TLS certificates: Each tenant control plane generates its own PKI. Certificate expiry breaks kubectl access for all users of that tenant cluster.

Step 2: Monitor Tenant Control Plane APIs

Each tenant control plane exposes a standard Kubernetes API at a unique endpoint. This is the primary availability signal for each tenant cluster:

# Get all tenant control plane endpoints
kubectl get tenantcontrolplanes -A \
  -o jsonpath='{range .items[*]}{.metadata.name}={.status.controlPlaneEndpoint.host}:{.status.controlPlaneEndpoint.port}{"\n"}{end}'

# Test a specific tenant API health
curl -k https://tenant1.kamaji.example.com:6443/healthz
# Expected: "ok"

# Test tenant API readiness (stricter check)
curl -k https://tenant1.kamaji.example.com:6443/readyz
# Expected: "ok"

# Verify tenant API is serving resources
curl -k https://tenant1.kamaji.example.com:6443/api/v1/namespaces \
  --header "Authorization: Bearer $TENANT_TOKEN"

In Vigilmon for each critical tenant control plane:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://tenant1.kamaji.example.com:6443/healthz.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: kamaji tenant-1 api.
8. Click Save.

Add a TCP monitor for an independent low-level connectivity check:

1. Add Monitor → TCP.
2. Host: tenant1.kamaji.example.com.
3. Port: 6443.
4. Check interval: 1 minute.
5. Label: kamaji tenant-1 tcp.
6. Click Save.

Step 3: Monitor the Shared etcd Datastore

Kamaji's multi-tenant etcd is the most critical shared dependency — its degradation affects all hosted control planes simultaneously:

# Check etcd cluster members (if running as etcd operator)
kubectl get etcdclusters -n kamaji-system
kubectl exec -n kamaji-system etcd-0 -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  endpoint health --cluster

# Check etcd health endpoint
curl https://etcd.kamaji-system.svc.cluster.local:2381/health
# Expected: {"health":"true","reason":""}

# Check etcd pod status
kubectl get pods -n kamaji-system -l app=etcd

Expose the etcd health endpoint via an Ingress for external monitoring:

# etcd-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kamaji-etcd-health
  namespace: kamaji-system
spec:
  rules:
  - host: etcd-health.kamaji.example.com
    http:
      paths:
      - path: /health
        pathType: Exact
        backend:
          service:
            name: etcd
            port:
              number: 2381
  tls:
  - hosts:
    - etcd-health.kamaji.example.com
    secretName: etcd-health-tls

kubectl apply -f etcd-health-ingress.yaml

In Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://etcd-health.kamaji.example.com/health.
3. Check interval: 1 minute.
4. Expected status: 200.
5. Keyword: true.
6. Label: kamaji etcd health.
7. Click Save.

Step 4: Monitor the Kamaji Controller

The Kamaji controller pod runs in the management cluster and manages all tenant control plane lifecycles. If it crashes, tenant control planes can no longer be automatically recovered or upgraded:

# Check Kamaji controller pod
kubectl get pods -n kamaji-system -l app.kubernetes.io/name=kamaji
kubectl logs -n kamaji-system deployment/kamaji --tail=30

# Kamaji controller metrics (if Prometheus scraping is enabled)
kubectl port-forward -n kamaji-system svc/kamaji-metrics 8080:8080
curl http://localhost:8080/metrics | grep kamaji_tenant_control_planes

# Check controller leader election status
kubectl get leases -n kamaji-system

Expose the Kamaji controller health endpoint externally:

# kamaji-controller-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kamaji-controller-health
  namespace: kamaji-system
spec:
  rules:
  - host: kamaji-health.example.com
    http:
      paths:
      - path: /healthz
        pathType: Exact
        backend:
          service:
            name: kamaji-controller-manager-metrics-service
            port:
              number: 8080
  tls:
  - hosts:
    - kamaji-health.example.com
    secretName: kamaji-health-tls

kubectl apply -f kamaji-controller-health-ingress.yaml

In Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://kamaji-health.example.com/healthz.
3. Check interval: 2 minutes.
4. Expected status: 200.
5. Label: kamaji controller health.
6. Click Save.

Step 5: Monitor the Management Cluster API

The Kamaji management cluster hosts all tenant control plane pods. If the management cluster API server degrades, all hosted control planes crash simultaneously:

# Check management cluster API health
kubectl get --raw /healthz
# Expected: "ok"

# Find management cluster API endpoint
kubectl config view --minify | grep server

# Check management cluster node health
kubectl get nodes
kubectl top nodes

In Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://mgmt-api.example.com:6443/healthz.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: kamaji management cluster api.
8. Click Save.

Step 6: Monitor SSL Certificates

Each Kamaji tenant control plane generates its own Kubernetes PKI. Certificate expiry for a tenant control plane breaks kubectl access for all users of that tenant:

# Check tenant control plane API certificate
openssl s_client -connect tenant1.kamaji.example.com:6443 2>/dev/null | \
  openssl x509 -noout -dates

# Check etcd health endpoint certificate
openssl s_client -connect etcd-health.kamaji.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# List cert-manager certificates for Kamaji-managed endpoints
kubectl get certificates -n kamaji-system
kubectl get certificates -n kamaji-tenants

# Check for soon-to-expire certificates across all namespaces
kubectl get certificates -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} {.status.notAfter}{"\n"}{end}'

In Vigilmon:

1. Add Monitor → SSL Certificate.
2. Domain: tenant1.kamaji.example.com.
3. Port: 6443.
4. Alert when expiry is within: 30 days.
5. Alert again: 14 days, 7 days, 3 days, 1 day.
6. Click Save.

Repeat for each tenant control plane domain and for etcd-health.kamaji.example.com.

Step 7: Configure Alerting and Runbook

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Immediate Action | |---|---|---| | Tenant control plane API HTTP | Non-200 | kubectl get tenantcontrolplanes -n kamaji-tenants; check tenant pods; kubectl describe tcp <name> | | Tenant control plane TCP | Connection refused | Check management cluster node where tenant pods are scheduled; verify Ingress routing | | etcd health | Non-200 / false keyword | CRITICAL — all tenants affected; kubectl get pods -n kamaji-system -l app=etcd; check etcd logs immediately | | Kamaji controller health | Non-200 | kubectl get pods -n kamaji-system; check controller logs; verify leader election | | Management cluster API | Non-200 | P0 incident — all hosted control planes offline; escalate to platform SRE immediately | | SSL certificate | < 30 days | kubectl get certificates; trigger cert-manager renewal; verify auto-rotation is enabled |

Escalation policy: etcd health failures are the highest-severity alert — they take down every tenant control plane simultaneously. Route them to your on-call platform SRE as a P0 incident. Individual tenant control plane failures are P1 affecting only that tenant's users.

Common Kamaji Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Tenant control plane kube-apiserver pod OOM killed | Tenant API TCP check catches connection refused within 1 min | | Shared etcd loses quorum during node failure | etcd health monitor catches "health":"false" immediately; all tenants affected | | cert-manager fails to renew tenant API TLS certificate | SSL monitor catches 30 days before tenant kubeconfig connections start failing | | Kamaji controller crashes during tenant upgrade rollout | Controller health goes down; in-progress upgrade stalls; unfinished tenants remain degraded | | Management cluster API upgrade causes brief unavailability | Management cluster HTTP monitor catches; surfaces before tenant users report issues | | etcd compaction causes latency spike above 300ms | Tenant API monitors catch elevated response times; can be correlated with etcd logs | | Tenant control plane node selector targets a drained node | Tenant pod stays in Pending; tenant API health goes down; Vigilmon alerts before tenant notices | | Multiple tenant control planes flood etcd with watch connections | etcd health stays green but performance degrades; TCP monitor catches connection latency |

Kamaji solves the economics of hosted Kubernetes control planes by running tenant API servers as regular pods on shared infrastructure — but that architecture concentrates risk in the shared etcd datastore and the management cluster itself. A single etcd outage or management cluster failure takes down every tenant simultaneously. Vigilmon provides external monitoring across all critical layers: individual tenant control plane APIs, the shared etcd datastore, the Kamaji controller, and the management cluster API. When any layer degrades, you know within minutes with the diagnostic context to distinguish a single-tenant failure from a platform-wide incident.

Start monitoring Kamaji in under 5 minutes — register free at vigilmon.online.