Monitoring KWOK Kubernetes Clusters with Vigilmon

KWOK (Kubernetes WithOut Kubelet) lets you simulate thousands of nodes and pods without spinning up real workloads — ideal for testing autoscalers, schedulers, and control plane behavior at scale. But "fake" nodes and pods don't mean zero risk: the API server still handles real traffic, control plane components still compete for resources, and simulation drift can quietly corrupt your test results. Vigilmon gives you the external health checks and alerting that KWOK clusters need to catch API server saturation, node status divergence, and scheduler stalls before they break your CI pipelines or capacity planning models.

What You'll Set Up

• API server health and availability monitors
• Fake node and pod status consistency checks via webhook heartbeats
• Control plane responsiveness tracking with latency alerts
• Cron heartbeats for simulation accuracy jobs
• Alert channels for CI/CD pipeline integration

Prerequisites

• KWOK installed and a simulated cluster running (kwokctl create cluster)
• kubectl configured to reach the cluster API server
• A free Vigilmon account

Step 1: Monitor the API Server Health Endpoint

KWOK clusters expose the standard Kubernetes API server, which includes a built-in /healthz endpoint. This is your primary uptime signal — if the API server is down, your entire simulation is unreachable.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the API server health URL:

   https://127.0.0.1:6443/healthz
   

   If your KWOK cluster is on a remote machine, replace the address with the actual host. Add ?verbose to get per-component health breakdowns.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Enable Monitor SSL certificate if you're using TLS, and set the alert threshold to 21 days.
7. Click Save.

For clusters managed with kwokctl, the API server port is configurable. Check the actual port with:

kwokctl get cluster --name my-cluster -o json | jq '.status.apiServerAddress'

Step 2: Check Control Plane Component Health

Beyond the API server itself, Kubernetes exposes individual component health via /readyz and /livez. KWOK clusters include these endpoints on the API server even though they're simulating the full cluster.

Add a second monitor for the readiness probe:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://127.0.0.1:6443/readyz
3. Set interval to 2 minutes.
4. Expected status: 200.
5. Click Save.

You can also probe specific component health checks:

# Check etcd health through API server
curl -k https://127.0.0.1:6443/readyz/etcd

# Check scheduler health
curl -k https://127.0.0.1:6443/readyz/poststarthook/start-kube-scheduler

For deeper visibility, expose the scheduler and controller-manager metrics endpoints and add them as separate monitors.

Step 3: Heartbeat Monitoring for Simulation Accuracy Jobs

KWOK simulations often include periodic jobs that reconcile fake node states, advance pod phases, or feed node metrics to the API server. If these jobs silently fail, your simulation drifts and produces meaningless test data.

Use Vigilmon's cron heartbeat to catch silent failures:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to match your reconciliation job frequency (e.g., 5 minutes).
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
4. Add the ping to the end of your simulation reconciliation script:

#!/bin/bash
set -e

# Advance all pending pods to Running state
kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | select(.status.phase=="Pending") | .metadata.name + " " + .metadata.namespace' | \
  while read name ns; do
    kubectl patch pod "$name" -n "$ns" --subresource=status \
      --type=merge -p '{"status":{"phase":"Running"}}'
  done

# Signal success to Vigilmon
curl -s https://vigilmon.online/heartbeat/abc123

If the script fails mid-run due to API server overload or a malformed patch, the heartbeat ping never fires, and Vigilmon alerts after the expected interval.

Step 4: Monitor API Server Load with TCP Checks

When simulating thousands of nodes, the KWOK API server can saturate under watch pressure or list operations from poorly tuned test clients. Add a TCP-level monitor as a lightweight canary that fires before HTTP-level health endpoints start returning errors:

1. Click Add Monitor → TCP Port.
2. Enter the API server host and port (e.g., 127.0.0.1:6443).
3. Set interval to 30 seconds.
4. Click Save.

A TCP monitor that succeeds while the HTTP monitor is failing usually means the API server is up but overloaded — responding to connections but not serving requests. This distinction helps you triage faster.

Pair this with a watch on API server latency metrics using the /metrics endpoint:

# Check API server request latency
curl -sk https://127.0.0.1:6443/metrics | \
  grep 'apiserver_request_duration_seconds_bucket'

Step 5: Track Fake Node Health Consistency

KWOK's fake nodes report status via the kwok-controller, which patches node status objects to simulate heartbeats. If the controller stops running, nodes will eventually appear NotReady — which may be intentional (testing node failure scenarios) or accidental.

Add a Vigilmon webhook monitor that fires from your cluster validation script:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 10 minutes.
3. Copy the heartbeat URL.
4. Create a validation script:

#!/bin/bash
# Check all KWOK nodes are in expected state
READY_NODES=$(kubectl get nodes --selector='kwok.x-k8s.io/node=fake' \
  --field-selector='status.conditions[?(@.type=="Ready")].status=True' \
  --no-headers | wc -l)

TOTAL_NODES=$(kubectl get nodes --selector='kwok.x-k8s.io/node=fake' \
  --no-headers | wc -l)

if [ "$READY_NODES" -eq "$TOTAL_NODES" ]; then
  curl -s https://vigilmon.online/heartbeat/abc123
else
  echo "Node mismatch: $READY_NODES/$TOTAL_NODES ready — skipping heartbeat"
fi

Set this script on a cron schedule matching your heartbeat interval. If nodes diverge from expected state, the heartbeat stops firing.

Step 6: CI/CD Pipeline Integration

KWOK is typically used in CI for testing schedulers, autoscalers, and admission controllers. Add Vigilmon API calls to your pipeline to suppress alerts during intentional cluster resets:

# .github/workflows/kwok-test.yaml
jobs:
  test:
    steps:
      - name: Start maintenance window
        run: |
          curl -X POST https://vigilmon.online/api/maintenance \
            -H "Authorization: Bearer ${{ secrets.VIGILMON_API_KEY }}" \
            -d '{"monitor_id": "${{ secrets.KWOK_MONITOR_ID }}", "duration_minutes": 10}'

      - name: Reset KWOK cluster
        run: kwokctl delete cluster && kwokctl create cluster

      - name: Run tests
        run: ./run-scheduler-tests.sh

This prevents false-positive alerts when your CI intentionally tears down and recreates the cluster between test runs.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health | /healthz on API server | API server crash or restart | | HTTP readiness | /readyz on API server | Control plane component failure | | TCP port | API server port | Connectivity loss, port binding failure | | Cron heartbeat | Simulation reconciliation job | Silent job failure, API overload stall | | Cron heartbeat | Node consistency validator | KWOK controller crash, node state drift |

KWOK removes the infrastructure overhead of testing Kubernetes at scale, but the API server and simulation controller running that fake cluster are real processes with real failure modes. Vigilmon gives you the external watchdog that catches those failures before your test results become unreliable or your CI pipelines start reporting false confidence.