How to Monitor Kubebuilder Controllers with Vigilmon

Kubebuilder is the official Kubernetes SDK for building APIs and controllers using Custom Resource Definitions. It ships the controller-runtime library, generates scaffolding for reconcilers and webhooks, and produces operators that extend the Kubernetes API surface. Kubebuilder controllers run as Deployments, own reconcile loops for custom resources, and often serve admission webhooks — making them load-bearing infrastructure for everything that depends on the custom resources they manage.

When a Kubebuilder controller goes down, custom resource reconciliation halts, admission webhooks block or reject API calls, and status conditions on managed resources stop updating. The failure is often invisible until a human notices a deployment stuck or a configuration that never applied. In this tutorial you'll set up uptime and response-time monitoring for your Kubebuilder controllers using Vigilmon — free tier, no credit card required.

Why Kubebuilder controllers need external monitoring

Kubernetes pod-level probes only check whether the controller process is alive. They miss the operational failure modes that matter most:

• Reconcile queue backlogged — the controller is running but processing so slowly that resources are stuck in a pending state; no pod restart occurs and no alarm fires
• Webhook backend down — a ValidatingWebhookConfiguration or MutatingWebhookConfiguration points to the controller's webhook server; if it's unreachable, kubectl apply operations fail cluster-wide with timeout errors
• Cache sync not completed — after a restart, controller-runtime's informer cache may not have synced before the controller begins serving; stale reads produce incorrect reconciliations
• External API dependency unavailable — the reconciler calls a cloud API, database, or external service; that dependency going down stalls all reconciliations silently
• Metrics server port closed — Kubebuilder exposes Prometheus metrics on port 8443 by default; if that port closes, your observability pipeline loses operator metrics while the controller appears healthy

External monitoring from Vigilmon catches these at the endpoint level before your users notice.

What you'll need

• A controller built with Kubebuilder (v3 or v4 project layout)
• The controller Deployment running in a Kubernetes cluster
• A free Vigilmon account

Step 1: Expose controller health and metrics endpoints

Kubebuilder scaffolds a manager that exposes /healthz and /readyz on port 8081 by default. Expose these externally:

# controller-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-controller-health
  namespace: my-controller-system
  labels:
    control-plane: controller-manager
spec:
  type: LoadBalancer
  selector:
    control-plane: controller-manager
  ports:
    - name: health
      port: 8081
      targetPort: 8081
      protocol: TCP
    - name: webhook
      port: 9443
      targetPort: 9443
      protocol: TCP

kubectl apply -f controller-health-service.yaml
kubectl get svc my-controller-health -n my-controller-system
# NAME                      TYPE           EXTERNAL-IP     PORT(S)
# my-controller-health      LoadBalancer   203.0.113.52    8081:31900/TCP,9443:31901/TCP

# Confirm the health endpoints respond
curl http://203.0.113.52:8081/healthz
# ok
curl http://203.0.113.52:8081/readyz
# ok

If you use Ingress instead of LoadBalancer:

# controller-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: controller-health-ingress
  namespace: my-controller-system
spec:
  ingressClassName: nginx
  rules:
    - host: controllers.yourdomain.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: my-controller-health
                port:
                  number: 8081
          - path: /readyz
            pathType: Exact
            backend:
              service:
                name: my-controller-health
                port:
                  number: 8081

Step 2: Add a reconciler health check

Extend the built-in health check to verify your reconcile loop is actively processing:

// internal/controller/health.go
package controller

import (
    "fmt"
    "net/http"
    "sync/atomic"
    "time"
)

// ReconcilerHealthChecker tracks the last successful reconcile time.
type ReconcilerHealthChecker struct {
    lastSuccessNano atomic.Int64
}

func (r *ReconcilerHealthChecker) RecordSuccess() {
    r.lastSuccessNano.Store(time.Now().UnixNano())
}

func (r *ReconcilerHealthChecker) Check(_ *http.Request) error {
    stored := r.lastSuccessNano.Load()
    if stored == 0 {
        return nil // no reconcile attempted yet — controller just started
    }
    last := time.Unix(0, stored)
    if time.Since(last) > 10*time.Minute {
        return fmt.Errorf("reconciler stalled: no successful reconcile in %v", time.Since(last))
    }
    return nil
}

Register it in main.go:

checker := &controller.ReconcilerHealthChecker{}
if err := mgr.AddHealthzCheck("reconciler", checker); err != nil {
    setupLog.Error(err, "unable to register reconciler health check")
    os.Exit(1)
}

// Pass checker to your reconciler so it can call RecordSuccess()
if err = (&controller.MyResourceReconciler{
    Client:          mgr.GetClient(),
    Scheme:          mgr.GetScheme(),
    HealthChecker:   checker,
}).SetupWithManager(mgr); err != nil {
    setupLog.Error(err, "unable to create controller")
    os.Exit(1)
}

Step 3: Set up HTTP monitoring in Vigilmon

With endpoints exposed and a meaningful health check in place, add monitors to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Controller healthz | http://203.0.113.52:8081/healthz | 200 | | Controller readyz | http://203.0.113.52:8081/readyz | 200 | | Webhook server | https://controllers.yourdomain.com:9443/healthz | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200
6. Save each monitor

Vigilmon checks from multiple geographic regions. A controller that's reachable from inside the cluster but not from external networks — a common pattern with misconfigured LoadBalancer or Ingress rules — is immediately surfaced.

Step 4: Monitor the webhook TCP layer

Kubebuilder admission webhooks listen on port 9443 with TLS. Add a TCP monitor to catch port-level failures:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your webhook hostname and port: controllers.yourdomain.com / 9443
4. Save the monitor

When the webhook port is unreachable, every kubectl apply against resources with a webhook produces a context deadline exceeded error — one of the most disruptive operator failure modes to diagnose under pressure.

Step 5: Configure alert channels

Controller failures are silent at first and then suddenly catastrophic when everyone tries to apply changes.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your platform team's on-call email
3. Assign the channel to all controller monitors

Webhook alerts for incident routing

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your PagerDuty, Opsgenie, or Slack webhook URL
3. Vigilmon sends this payload on failure:

{
  "monitor_name": "Controller healthz",
  "status": "down",
  "url": "http://203.0.113.52:8081/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Use this to trigger a runbook that checks whether any admission webhooks are set to Fail mode — these will start blocking all API operations if the webhook server stays down.

Step 6: Correlate Vigilmon alerts with controller diagnostics

When you get a controller downtime alert from Vigilmon, run these checks in order:

# 1. Check the controller manager pod
kubectl get pods -n my-controller-system -l control-plane=controller-manager

# 2. Read recent controller logs
kubectl logs -n my-controller-system \
  -l control-plane=controller-manager --tail=100

# 3. Check custom resource status conditions
kubectl get myresource -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{range .status.conditions[*]}{.type}={.status}{" "}{end}{"\n"}{end}'

# 4. Check if webhooks are in Fail mode and potentially blocking operations
kubectl get validatingwebhookconfigurations -o \
  jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .webhooks[*]}{.failurePolicy}{" "}{end}{"\n"}{end}'

# 5. Check leader election lease
kubectl get lease -n my-controller-system

# 6. Verify the controller's service account permissions
kubectl auth can-i list myresource \
  --as=system:serviceaccount:my-controller-system:my-controller-controller-manager

If /healthz fails but the pod is Running, the custom reconciler health check has fired — look for a 10+ minute gap in reconcile activity in the logs.

Step 7: Create a status page for controller-managed APIs

Teams that use your controller's custom resources should be able to self-diagnose controller health:

1. Go to Status Pages → New Status Page
2. Name it: "Platform Controllers"
3. Add your monitors: Controller healthz, Webhook server
4. Share the URL with development teams using the managed CRDs

When someone files a bug that kubectl apply is timing out, they can check the status page before filing an incident ticket.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on /healthz | Controller crashes, reconciler stalls | | HTTP monitor on /readyz | Cache not synced after restart | | HTTP monitor on webhook server | Admission webhook failures | | TCP monitor on webhook port 9443 | TLS handshake failures, port outages | | Email + webhook alert channels | Immediate notification before API operations block | | Status page | Developer self-service for kubectl apply failures |

Kubebuilder controllers are the authoritative source of truth for your custom resources. External monitoring is the only way to know when that authority goes offline before users experience it.

Get started at vigilmon.online — free tier, no credit card required.