Monitoring Krustlet with Vigilmon: Keeping Your WebAssembly Kubernetes Node Agent Healthy

Krustlet is a Kubernetes kubelet implementation written in Rust that runs WebAssembly workloads as native Kubernetes pods using WASI (WebAssembly System Interface). Unlike runwasi (a containerd shim), Krustlet implements the full kubelet API — it registers as a Kubernetes node and accepts pod specs directly from the API server, executing Wasm modules in place of OCI containers. Platform teams use Krustlet to dedicate nodes to Wasm workloads, enabling strong isolation between Wasm and container workloads while using the standard Kubernetes scheduler, RBAC, and namespace model. When Krustlet fails, its node disappears from the cluster or stops accepting pod assignments, leaving Wasm workloads unschedulable cluster-wide. Vigilmon gives you external monitoring that watches Krustlet's kubelet API and validates that Wasm pod lifecycle operations are completing on schedule.

What You'll Build

Vigilmon HTTP monitors for Krustlet's kubelet API health endpoints
A heartbeat monitor validating that Wasm pods schedule and run end-to-end
A node registration heartbeat to catch Krustlet nodes going NotReady
Alert channels routed to your platform engineering team

Prerequisites

Krustlet deployed and registered as a Kubernetes node
kubectl configured to reach the cluster API server
A free account at vigilmon.online

Why Monitoring Krustlet Matters

Krustlet acts as a Kubernetes node — when it fails, its failure mode is a node going NotReady, which Kubernetes handles by evicting pods and marking them Unschedulable. But Krustlet failures have characteristics that make them harder to detect than standard node failures:

Node registration loss is the most common Krustlet failure. Krustlet must maintain a heartbeat lease with the Kubernetes API server. If the Krustlet process crashes or loses network connectivity to the API server, the node transitions to NotReady after the node eviction timeout (typically 5 minutes). During that window, the scheduler continues assigning new Wasm pods to the NotReady node — they queue and are never executed. By the time eviction completes, there may be a backlog of Wasm pods all attempting to reschedule simultaneously.

WASI runtime engine failures crash pod execution silently. Krustlet delegates actual Wasm execution to a backend WASI engine (Wasmtime or WAGI). If that engine is missing, wrong version, or has a broken system library dependency, Krustlet logs the error but continues reporting the node as Ready to Kubernetes. New Wasm pods are scheduled to the node and immediately fail — appearing as CrashLoopBackOff at the pod level, with no indication at the node level.

TLS certificate expiry breaks Krustlet's kubelet serving certificate. Krustlet generates a serving certificate during bootstrap that the Kubernetes API server uses to communicate back to the Krustlet node (for kubectl exec, log streaming, and port-forward). When this certificate expires, the node continues reporting Ready but interactive operations against its pods fail with TLS handshake errors. Most teams don't discover expired Krustlet certs until an engineer tries to debug a failing Wasm pod.

Pod spec incompatibility occurs as Kubernetes API versions evolve. Krustlet may not implement newer pod spec fields that a newer Kubernetes control plane sets by default (e.g., new security context fields, pod OS selector). Pods that include these fields may be rejected silently by Krustlet's pod handler, appearing to schedule but never executing.

Step 1: Monitor Krustlet's Kubelet API Health

Krustlet exposes a kubelet-compatible HTTP API on port 3000 (default). Monitor its health endpoint:

# Verify Krustlet's kubelet API is responding
curl -sk https://krustlet-node-ip:3000/healthz
# Expected: ok

# Check node status from Kubernetes
kubectl get node krustlet-node -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Expected: True

In the Vigilmon dashboard:

Go to Add Monitor → HTTP
Configure:

| Field | Value | |---|---| | Name | Krustlet kubelet /healthz | | URL | https://krustlet-node-ip:3000/healthz | | Method | GET | | Expected status | 200 | | Response body contains | ok | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

If Krustlet's TLS certificate is self-signed (which it is by default), configure Vigilmon to skip certificate verification or add Krustlet's CA cert to your Vigilmon monitoring configuration.

Step 2: Monitor Krustlet Node Ready Status

Watch for Krustlet node transitions to NotReady using a Kubernetes probe CronJob:

# krustlet-node-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: krustlet-node-probe
  namespace: monitoring
spec:
  schedule: "*/2 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: krustlet-probe-sa
          containers:
            - name: probe
              image: bitnami/kubectl:latest
              env:
                - name: KRUSTLET_NODE_NAME
                  value: "krustlet-node"
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: krustlet-node-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e

                  STATUS=$(kubectl get node "$KRUSTLET_NODE_NAME" \
                    -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' \
                    2>/dev/null)

                  if [ "$STATUS" = "True" ]; then
                    echo "Krustlet node $KRUSTLET_NODE_NAME is Ready"
                    wget -qO- "$HEARTBEAT_URL"
                    echo "Node heartbeat sent"
                  else
                    echo "ERROR: Krustlet node status: '$STATUS'" >&2
                    exit 1
                  fi

# RBAC for the probe
apiVersion: v1
kind: ServiceAccount
metadata:
  name: krustlet-probe-sa
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: krustlet-probe-node-reader
subjects:
  - kind: ServiceAccount
    name: krustlet-probe-sa
    namespace: monitoring
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

Store secrets:

kubectl create secret generic vigilmon-secrets \
  --from-literal=krustlet-node-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_NODE_TOKEN' \
  --from-literal=krustlet-wasm-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_WASM_TOKEN' \
  -n monitoring

Create the heartbeat monitor in Vigilmon:

Add Monitor → Heartbeat
Name: Krustlet node Ready
Expected interval: 2 minutes
Grace period: 3 minutes

Step 3: Heartbeat Monitor for Wasm Pod End-to-End Execution

Validate the full scheduling path — from pod creation to Wasm execution completion — with a CronJob that schedules a Wasm workload on the Krustlet node:

# krustlet-wasm-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: krustlet-wasm-probe
  namespace: monitoring
spec:
  schedule: "*/10 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: krustlet-probe-sa
          containers:
            - name: wasm-scheduler
              image: bitnami/kubectl:latest
              env:
                - name: KRUSTLET_NODE_NAME
                  value: "krustlet-node"
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: krustlet-wasm-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  PROBE_NAME="wasm-probe-$(date +%s)"

                  # Create a minimal Wasm probe pod on the Krustlet node
                  cat <<EOF | kubectl apply -f -
                  apiVersion: v1
                  kind: Pod
                  metadata:
                    name: $PROBE_NAME
                    namespace: monitoring
                    labels:
                      app: wasm-probe
                  spec:
                    nodeName: $KRUSTLET_NODE_NAME
                    restartPolicy: Never
                    tolerations:
                      - key: "kubernetes.io/arch"
                        operator: "Equal"
                        value: "wasm32-wasi"
                        effect: "NoSchedule"
                    containers:
                      - name: wasm
                        image: webassembly.azurecr.io/hello-wasm:v1
                  EOF

                  # Wait for the probe pod to complete
                  for i in $(seq 1 24); do
                    PHASE=$(kubectl get pod "$PROBE_NAME" -n monitoring \
                      -o jsonpath='{.status.phase}' 2>/dev/null || echo "Unknown")
                    echo "Pod $PROBE_NAME phase: $PHASE (attempt $i/24)"
                    if [ "$PHASE" = "Succeeded" ]; then
                      kubectl delete pod "$PROBE_NAME" -n monitoring --ignore-not-found
                      wget -qO- "$HEARTBEAT_URL"
                      echo "Wasm lifecycle heartbeat sent"
                      exit 0
                    elif [ "$PHASE" = "Failed" ]; then
                      kubectl delete pod "$PROBE_NAME" -n monitoring --ignore-not-found
                      echo "ERROR: Wasm probe pod failed" >&2
                      exit 1
                    fi
                    sleep 5
                  done

                  kubectl delete pod "$PROBE_NAME" -n monitoring --ignore-not-found
                  echo "ERROR: Wasm probe pod did not complete within timeout" >&2
                  exit 1

Create the heartbeat monitor in Vigilmon:

Add Monitor → Heartbeat
Name: Krustlet Wasm pod execution
Expected interval: 10 minutes
Grace period: 5 minutes

Step 4: Configure Alert Channels

Route Krustlet alerts to your platform engineering team:

In Vigilmon: Alert Channels → Add Channel → Slack Webhook
Add your #platform-engineering webhook URL
Assign to all Krustlet monitors

For node Ready status failures (a NotReady Krustlet node causes Wasm workload eviction):

Alert Channels → Add Channel → PagerDuty
Route the Krustlet node Ready heartbeat to on-call with high urgency
Node NotReady within the eviction window means active Wasm workloads are being killed

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | Krustlet kubelet /healthz | HTTP GET | Krustlet process down, kubelet API unreachable | | Krustlet node Ready | Heartbeat (2 min) | Node transitions to NotReady, lease heartbeat lost | | Wasm pod execution | Heartbeat (10 min) | Scheduling failures, WASI runtime errors, pod never completes |

Conclusion

Krustlet's native Kubernetes integration is its greatest strength and the source of its most critical failure mode: when Krustlet is unhealthy, it looks like a node going NotReady — a familiar signal that Kubernetes tries to handle automatically through eviction. But eviction of Wasm pods means re-queuing them, and if the only Krustlet node is down, they queue forever without clear operator notification. External monitoring with Vigilmon closes that gap by watching the kubelet API from outside the cluster, probing node Ready status independently, and validating the full Wasm pod execution path end-to-end.

Get started free at vigilmon.online. The kubelet health endpoint and node Ready heartbeat together give you the two most critical Krustlet signals in under ten minutes: is the Krustlet process alive, and is the node available to accept Wasm pod assignments?