Monitoring KCP with Vigilmon: Control Plane API Availability, Workspace Sync Health, Virtual Cluster Connectivity & SSL Certificate Alerts

KCP is the Kubernetes Control Plane project that decouples the Kubernetes API machinery from cluster compute, enabling you to run thousands of lightweight, isolated virtual control planes — called workspaces — as a shared, multi-tenant platform. When a workspace API server becomes unavailable, all tenants accessing that virtual cluster lose connectivity to their Kubernetes resources. When the KCP syncer that bridges workspaces to physical clusters fails, workloads drift silently between the desired and actual state. Vigilmon gives you external visibility into the KCP workspace APIs, syncer health endpoints, the underlying physical cluster connectivity, and the TLS certificates protecting every control plane endpoint so you know immediately when your multi-tenant platform is degraded.

What You'll Build

• HTTP monitors on KCP workspace API endpoints to detect virtual control plane failures
• A monitor for the KCP syncer health status bridging workspaces to physical compute
• TCP monitors for independent low-level KCP API connectivity checks
• SSL certificate monitoring for all KCP TLS endpoints
• An alert runbook mapping Vigilmon findings to specific kubectl kcp and syncer remediation commands

Prerequisites

• KCP v0.24+ installed with at least one workspace and one syncer configured
• A running KCP root shard or single-shard KCP installation
• The kubectl kcp plugin for workspace navigation
• Network access to your KCP API endpoints from an external monitor
• A free account at vigilmon.online

Step 1: Understand KCP's Observability Surface

KCP exposes multiple API endpoints depending on your deployment architecture — a root shard, optional additional shards, and per-workspace virtual API endpoints:

# Check KCP root shard health
curl https://kcp.example.com/healthz
# Expected: "ok"

# List workspaces (requires kcp plugin)
kubectl kcp workspace list

# Switch to a specific workspace
kubectl kcp workspace use my-workspace

# Check the virtual workspace API endpoint
kubectl get --raw /healthz
# Expected: "ok"

# Check syncer status in the physical cluster
kubectl get synctargets -A
kubectl describe synctarget my-syncer

# Find the KCP API server address
kubectl config view --minify | grep server
# Output: server: https://kcp.example.com:6443

The key observability signals are:

• Root shard API health: All workspace creation and management goes through the root shard. If it fails, tenants cannot create or access workspaces.
• Virtual workspace API endpoints: Each workspace exposes its own Kubernetes API path. Workspace-level failures are isolated but affect all users of that workspace.
• Syncer health: The KCP syncer process bridges workspace resource specs to physical cluster workloads. A syncer crash causes drift between the desired workspace state and what runs on compute.
• TLS certificates: KCP uses distinct certificates per shard and per virtual workspace endpoint; expired certificates break client connectivity silently with TLS handshake failures.

Step 2: Monitor the KCP Root Shard API

The KCP root shard is the central control plane. All workspace management operations and API discovery flow through it:

# Verify root shard health endpoint
curl -k https://kcp.example.com:6443/healthz
# Expected: "ok"

# Check readiness (more granular than liveness)
curl -k https://kcp.example.com:6443/readyz
# Expected: "ok"

# List root shard API groups
curl -k https://kcp.example.com:6443/apis | jq '.groups[].name' | head -10

In Vigilmon:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://kcp.example.com:6443/healthz.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: kcp root shard health.
8. Click Save.

Also add a TCP monitor for a connectivity check independent of HTTP and TLS routing:

1. Add Monitor → TCP.
2. Host: kcp.example.com.
3. Port: 6443.
4. Check interval: 1 minute.
5. Label: kcp root shard tcp.
6. Click Save.

Step 3: Monitor Virtual Workspace API Endpoints

KCP virtual workspaces expose their own Kubernetes API paths under the root shard URL. Monitor your most critical tenant workspaces independently:

# Get the virtual workspace API URL for a specific workspace
kubectl kcp workspace use my-workspace
kubectl config view --minify | grep server
# Output example: https://kcp.example.com:6443/clusters/root:my-org:my-workspace

# Check the virtual workspace health
curl -k https://kcp.example.com:6443/clusters/root:my-org:my-workspace/healthz
# Expected: "ok"

# List resources available in the workspace
kubectl --server=https://kcp.example.com:6443/clusters/root:my-org:my-workspace api-resources

For each critical workspace in Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://kcp.example.com:6443/clusters/root:my-org:my-workspace/healthz.
3. Check interval: 5 minutes.
4. Expected status: 200.
5. Keyword: ok.
6. Label: kcp workspace my-org/my-workspace.
7. Click Save.

Repeat for each production workspace. For large-scale deployments with hundreds of workspaces, prioritize monitoring critical shared infrastructure workspaces and use workspace health aggregation rather than per-workspace monitors.

Step 4: Monitor the KCP Syncer

The KCP syncer runs inside the physical cluster and pulls resource specs from workspaces, creating and updating actual workloads on compute nodes. If the syncer loses connectivity to the KCP API or to the physical cluster's API, it stops applying changes silently:

# Check syncer pod health in the physical cluster
kubectl get pods -n kcp-system -l app=syncer
kubectl logs -n kcp-system deployment/syncer --tail=30

# Syncer exports a health endpoint on port 8080 by default
kubectl port-forward -n kcp-system svc/syncer 8080:8080
curl http://localhost:8080/healthz
# Expected: "ok"

# Check SyncTarget status from the workspace side
kubectl kcp workspace use my-workspace
kubectl get synctargets -o wide
kubectl describe synctarget my-physical-cluster
# Look for: "Last Sync Time" and "Conditions: Ready"

Expose the syncer health endpoint externally:

# syncer-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kcp-syncer-health
  namespace: kcp-system
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: syncer-health-auth
spec:
  rules:
  - host: syncer-health.example.com
    http:
      paths:
      - path: /healthz
        pathType: Exact
        backend:
          service:
            name: syncer
            port:
              number: 8080
  tls:
  - hosts:
    - syncer-health.example.com
    secretName: syncer-health-tls

kubectl apply -f syncer-health-ingress.yaml

In Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://syncer-health.example.com/healthz.
3. Check interval: 2 minutes.
4. Expected status: 200.
5. Keyword: ok.
6. Label: kcp syncer health.
7. Click Save.

Step 5: Monitor KCP API Shards (Multi-Shard Deployments)

Production KCP deployments use multiple shards for horizontal scaling. Each shard independently hosts a subset of workspaces. Monitor all shards:

# List available shards
kubectl get shards -A

# Check individual shard health
curl -k https://shard1.kcp.example.com:6443/healthz
curl -k https://shard2.kcp.example.com:6443/healthz

# Check shard-specific workspace routing
kubectl kcp workspace use --server=https://shard1.kcp.example.com:6443 my-workspace

In Vigilmon, add HTTP and TCP monitors for each shard following the pattern from Step 2. Use labels like kcp shard-1 health and kcp shard-2 health to distinguish them in your alert dashboard.

For a high-level shard aggregation endpoint (if you front KCP shards with an L7 load balancer):

1. Add Monitor → HTTP.
2. URL: https://kcp-lb.example.com:6443/healthz.
3. Check interval: 1 minute.
4. Label: kcp load balancer.
5. Click Save.

Step 6: Monitor SSL Certificates

KCP generates and uses TLS certificates for every shard API and virtual workspace endpoint. A certificate expiry silently breaks kubectl connectivity for all tenants on that shard:

# Check root shard certificate
openssl s_client -connect kcp.example.com:6443 2>/dev/null | \
  openssl x509 -noout -dates

# Check additional shard certificates
openssl s_client -connect shard1.kcp.example.com:6443 2>/dev/null | \
  openssl x509 -noout -dates

# Check syncer health endpoint certificate
openssl s_client -connect syncer-health.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# List cert-manager certificates if used for KCP TLS
kubectl get certificates -n kcp-system

In Vigilmon:

1. Add Monitor → SSL Certificate.
2. Domain: kcp.example.com.
3. Port: 6443.
4. Alert when expiry is within: 30 days.
5. Alert again: 14 days, 7 days, 3 days, 1 day.
6. Click Save.

Repeat for shard1.kcp.example.com, shard2.kcp.example.com, and syncer-health.example.com.

Step 7: Configure Alerting and Runbook

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Immediate Action | |---|---|---| | KCP root shard HTTP | Non-200 | kubectl get pods -n kcp-system; check root shard logs; verify etcd health | | KCP root shard TCP | Connection refused | Check node health; verify kube-apiserver process on KCP nodes | | KCP workspace API | Non-200 | kubectl kcp workspace use <name> → check workspace status; look for shard placement issues | | KCP syncer health | Non-200 | kubectl get pods -n kcp-system -l app=syncer; check syncer logs for connectivity errors | | SSL certificate | < 30 days | kubectl get certificates -n kcp-system; trigger cert-manager renewal |

Escalation policy: Route root shard failures to your platform team as P1 — a root shard outage blocks all tenant workspace operations. Workspace-level failures affect only that workspace's tenants and can be triaged as P2.

Common KCP Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | KCP root shard node OOM killed during peak workspace creation | Root shard TCP check catches connection refused; alert within 1 min | | cert-manager fails to renew shard API TLS certificate | SSL monitor catches 30 days before kubectl connections start failing | | Syncer loses network connectivity to KCP API due to network policy change | Syncer health check goes down; workspace workloads stop being reconciled; alert within 2 min | | etcd compaction causes temporary KCP API latency spikes | HTTP monitor catches timeouts above response threshold; alert on degraded performance | | Multi-shard imbalance causes workspace migration to overloaded shard | Shard-specific HTTP monitors identify which shard is degraded while others remain healthy | | Syncer pod evicted from physical cluster node | Syncer health endpoint goes down; SyncTarget shows stale last-sync timestamp | | KCP API upgrade causes workspace virtual endpoint path changes | HTTP monitor returns 404; surfaces broken routing before tenant support tickets arrive |

KCP separates the Kubernetes API plane from compute, enabling platform teams to offer thousands of isolated virtual control planes on shared infrastructure — but that separation creates new failure modes at the shard, workspace, and syncer layers that traditional cluster-level monitoring does not cover. Vigilmon provides external monitoring across every layer: the root shard API, individual virtual workspace endpoints, syncer health, and the TLS certificates that protect every connection. When any layer degrades, you know within minutes with enough diagnostic context to pinpoint whether the problem is in the shard, a specific workspace, or the syncer bridging to physical compute.

Start monitoring KCP in under 5 minutes — register free at vigilmon.online.