Monitoring OpenKruise with Vigilmon

OpenKruise is a CNCF project that extends Kubernetes with powerful workload management primitives — CloneSet, Advanced StatefulSet, SidecarSet, and more. When the OpenKruise controller manager goes down, your custom workloads stop reconciling and your webhook server stops admitting pods. Unlike kube-apiserver, OpenKruise has no built-in alerting. Vigilmon fills that gap with health endpoint checks, TCP port monitoring, and controller heartbeat verification.

What You'll Set Up

• HTTP monitor for the OpenKruise controller manager /healthz endpoint
• HTTP monitor for the /readyz endpoint
• TCP port monitor for the webhook server (port 9443)
• Heartbeat monitor for OpenKruise controller pod availability
• Kruise-daemon health check monitoring

Prerequisites

• OpenKruise installed in a Kubernetes cluster (v1.4+)
• The controller manager and webhook server reachable from the Vigilmon probe network (or via a tunneled endpoint)
• A free Vigilmon account

Step 1: Expose OpenKruise Health Endpoints

The OpenKruise controller manager exposes /healthz and /readyz on a metrics/health port (default 8080). These endpoints are available inside the cluster by default. To monitor them from Vigilmon, expose them via a Kubernetes Service or an Ingress.

Create a ClusterIP service that Vigilmon can reach through your existing ingress:

apiVersion: v1
kind: Service
metadata:
  name: kruise-controller-health
  namespace: kruise-system
spec:
  selector:
    control-plane: controller-manager
  ports:
    - name: health
      port: 8080
      targetPort: 8080

Then expose it via an Ingress or use kubectl port-forward for a quick test:

kubectl port-forward -n kruise-system svc/kruise-controller-manager 8080:8080
curl http://localhost:8080/healthz
# ok
curl http://localhost:8080/readyz
# ok

If you expose the endpoint externally, protect it with network policies or basic auth — these endpoints don't require authentication by default.

Step 2: Monitor the /healthz and /readyz Endpoints

Once the health endpoints are reachable, add HTTP monitors in Vigilmon:

Liveness monitor (/healthz):

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: https://kruise-health.example.com/healthz
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Response body must contain, enter ok.
7. Click Save.

Readiness monitor (/readyz):

1. Click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: https://kruise-health.example.com/readyz
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Response body must contain, enter ok.
7. Click Save.

The /healthz endpoint reflects controller liveness — it fails if the process is deadlocked or crashed. The /readyz endpoint reflects readiness — it fails during leader election or when the controller hasn't finished syncing its caches.

Step 3: TCP Port Monitor for the Webhook Server

OpenKruise's webhook server runs on port 9443 and handles admission webhooks for all OpenKruise custom resources. If the webhook server is down and your webhooks have failurePolicy: Fail, pod creation for affected workloads will be blocked cluster-wide.

Add a TCP monitor:

1. Click Add Monitor → TCP Port.
2. Set Host to your cluster's webhook ingress or the node IP where the webhook pod runs.
3. Set Port to 9443.
4. Set Check interval to 1 minute.
5. Click Save.

To verify connectivity from outside the cluster:

# Test TCP handshake to the webhook service
nc -zv kruise-webhook.example.com 9443
# Connection to kruise-webhook.example.com 9443 port [tcp/*] succeeded!

If your cluster doesn't expose port 9443 externally, use the cluster's API server proxy:

kubectl get --raw /api/v1/namespaces/kruise-system/services/kruise-webhook-service:9443/proxy/healthz

Step 4: Monitor Kruise-Daemon Health

kruise-daemon runs as a DaemonSet on every node and handles node-level operations like image pre-pulling and container restart policies. Each daemon exposes a health endpoint on port 10221.

To monitor it, create an aggregated health check endpoint using a small service that queries the DaemonSet status:

# Check kruise-daemon pod count vs node count
kubectl get daemonset -n kruise-system kruise-daemon
# NAME           DESIRED   CURRENT   READY   ...
# kruise-daemon  5         5         5       ...

For direct monitoring, expose the daemon's health port on a sample node and add a Vigilmon monitor:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter the URL: http://node1.example.com:10221/healthz
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.
5. Click Save.

Alternatively, monitor the DaemonSet rollout health via the Kubernetes API:

kubectl get ds kruise-daemon -n kruise-system -o jsonpath='{.status.numberReady}'

Wrap this in a cron job that pings a Vigilmon heartbeat when all daemons are ready.

Step 5: Heartbeat Monitoring for Controller Pod Availability

The OpenKruise controller manager runs as a Deployment with leader election. If all replicas crash, reconciliation stops and no workloads are updated. Use a Vigilmon cron heartbeat driven by a Kubernetes CronJob to verify controller availability.

Create a CronJob that checks the controller and pings Vigilmon:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kruise-heartbeat
  namespace: kruise-system
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kruise-heartbeat-sa
          containers:
            - name: heartbeat
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  READY=$(kubectl get deployment kruise-controller-manager \
                    -n kruise-system \
                    -o jsonpath='{.status.readyReplicas}')
                  if [ "$READY" -ge "1" ]; then
                    wget -qO- https://vigilmon.online/heartbeat/abc123
                  fi
          restartPolicy: OnFailure

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to 7 minutes (5-minute cron + 2-minute buffer).
3. Name it openkruise-controller-availability.
4. Paste the heartbeat URL into the CronJob command above.
5. Click Save.

If the controller deployment has zero ready replicas, the heartbeat is never sent — Vigilmon alerts after the interval expires.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, PagerDuty, or email.
2. Set Consecutive failures before alert to 2 on health endpoint monitors to tolerate brief leader election cycles during controller restarts.
3. Keep the TCP webhook monitor at 1 failure before alerting — a down webhook server causes immediate cluster-wide pod admission failures.
4. Add a Maintenance window in Vigilmon when upgrading OpenKruise to suppress false alerts during the rolling restart.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP /healthz | Controller manager health port | Process crash, deadlock | | HTTP /readyz | Controller manager readiness | Leader election failure, cache sync issue | | TCP port 9443 | Webhook server | Admission webhook unavailability | | HTTP health | Kruise-daemon per node | Node-level daemon crash | | Cron heartbeat | CronJob → controller ready check | Zero-replica controller outage |

OpenKruise is a critical extension of your Kubernetes control plane — its health directly affects your workload reliability. With Vigilmon watching health endpoints, the webhook port, daemon nodes, and controller availability via heartbeat, you'll detect OpenKruise failures before they cascade into workload reconciliation outages.