How to Monitor Operator SDK Applications with Vigilmon

Operator SDK is Red Hat's framework for building Kubernetes operators in Go, Ansible, or Helm. Operators built with the SDK extend the Kubernetes API through custom controllers that watch custom resources and reconcile real-world state toward a desired configuration. They run as Deployments inside the cluster and typically manage databases, message queues, certificates, and other stateful services on behalf of application teams.

But operators are infrastructure. A crashed operator controller means your custom resources stop being reconciled — databases don't get provisioned, certificates don't get renewed, and configuration drift accumulates silently. In this tutorial you'll set up uptime and response-time monitoring for your Operator SDK controllers using Vigilmon — free tier, no credit card required.

Why Operator SDK controllers need external monitoring

Internal Kubernetes probes on the operator Pod catch container crashes but miss the broader class of operational failures:

• Controller reconcile loop stalled — the operator is running but a dependency (cloud API, external service, webhook backend) is unreachable; CRs pile up in error state while the pod reports Healthy
• Leader election lost — in multi-replica operator deployments, a leader election failure leaves no active reconciler without any pod restarting
• Webhook server unavailable — admission or conversion webhooks served by the operator return errors; every attempt to create or update the managed CRs fails cluster-wide
• Manager HTTP server down — Operator SDK operators expose /metrics and /healthz on a built-in manager server; if that server stops responding, the controller is effectively blind
• RBAC permission revoked — a cluster role change removes the operator's access to a resource type; reconciliations silently fail without crashing the pod

External monitoring from Vigilmon watches the endpoints your operator exposes and alerts you when any of these failure modes appear.

What you'll need

• An operator built with Operator SDK (Go, Ansible, or Helm project type)
• The operator deployed to a Kubernetes cluster with its manager HTTP server reachable externally or via Ingress
• A free Vigilmon account

Step 1: Expose the operator manager endpoints

Operator SDK operators expose /healthz and /readyz on the manager HTTP server (default port 8081) and metrics on port 8443. Make these reachable for external monitoring:

# operator-metrics-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-operator-metrics
  namespace: my-operator-system
  labels:
    app: my-operator
spec:
  type: ClusterIP
  selector:
    control-plane: controller-manager
  ports:
    - name: health
      port: 8081
      targetPort: 8081
    - name: metrics
      port: 8443
      targetPort: 8443
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-operator-health-ingress
  namespace: my-operator-system
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
    - host: ops.yourdomain.com
      http:
        paths:
          - path: /operator(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: my-operator-metrics
                port:
                  number: 8081

kubectl apply -f operator-metrics-service.yaml

# Verify the health endpoint is reachable
kubectl port-forward svc/my-operator-metrics 8081:8081 -n my-operator-system &
curl http://localhost:8081/healthz
# ok
curl http://localhost:8081/readyz
# ok

If you prefer to expose via LoadBalancer for direct external access:

kubectl patch svc my-operator-metrics -n my-operator-system \
  -p '{"spec":{"type":"LoadBalancer"}}'
kubectl get svc my-operator-metrics -n my-operator-system
# NAME                   TYPE           EXTERNAL-IP     PORT(S)
# my-operator-metrics    LoadBalancer   203.0.113.51    8081:31081/TCP

Step 2: Add a custom health handler to your operator

Operator SDK's built-in /healthz only checks that the HTTP server is running. Add a custom check that verifies your reconcile loop is functioning:

// main.go — add to your manager setup
import (
    "net/http"
    "sigs.k8s.io/controller-runtime/pkg/healthz"
)

// Custom reconciler health check
type reconcilerCheck struct {
    lastReconcileTime *time.Time
}

func (r *reconcilerCheck) Check(req *http.Request) error {
    if r.lastReconcileTime == nil {
        return nil // not yet started
    }
    if time.Since(*r.lastReconcileTime) > 5*time.Minute {
        return fmt.Errorf("reconciler stalled: last reconcile was %v ago",
            time.Since(*r.lastReconcileTime))
    }
    return nil
}

// Register in main():
if err := mgr.AddHealthzCheck("reconciler", &reconcilerCheck{&lastReconcileTime}); err != nil {
    setupLog.Error(err, "unable to set up reconciler health check")
    os.Exit(1)
}

Now /healthz returns a non-200 status when your reconcile loop stalls — making it a meaningful signal for Vigilmon to monitor.

Step 3: Set up HTTP monitoring in Vigilmon

With your endpoints accessible, add them to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Operator healthz | https://ops.yourdomain.com/operator/healthz | 200 | | Operator readyz | https://ops.yourdomain.com/operator/readyz | 200 | | Webhook server | https://webhook.yourdomain.com/healthz | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match ok in the response body
6. Save each monitor

Vigilmon probes from multiple geographic regions simultaneously. If your webhook server is responding from inside the cluster but unreachable from external clients, Vigilmon will catch the discrepancy that your internal probes miss.

Step 4: Monitor the webhook TCP layer

Operator SDK operators commonly serve admission webhooks on port 9443. Add a TCP monitor to detect TLS or port-level failures independently of the HTTP layer:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your webhook host and port: webhook.yourdomain.com / 9443
4. Save the monitor

A TCP failure here means every kubectl apply of your CRDs will return a timeout until the webhook server recovers — often a worse user experience than a reconcile stall.

Step 5: Configure alert channels

When an operator controller goes down, managed resources stop being reconciled. Catch this before it cascades.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your platform team's on-call address
3. Assign the channel to all operator monitors

Webhook alerts for incident routing

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. The payload Vigilmon sends:

{
  "monitor_name": "Operator healthz",
  "status": "down",
  "url": "https://ops.yourdomain.com/operator/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Wire this alert into a runbook that immediately checks whether any CRs have been stuck in a reconciling state longer than expected.

Step 6: Correlate Vigilmon alerts with operator diagnostics

When you receive an operator downtime alert, run these checks:

# 1. Check the operator controller pod
kubectl get pods -n my-operator-system -l control-plane=controller-manager

# 2. Check controller manager logs for reconcile errors
kubectl logs -n my-operator-system \
  -l control-plane=controller-manager --tail=100 \
  | grep -i "error\|fail\|reconcile"

# 3. Check your custom resource status conditions
kubectl get myresource -A -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.conditions[*].type}{" "}{.status.conditions[*].status}{"\n"}{end}'

# 4. Inspect leader election
kubectl get lease -n my-operator-system

# 5. Check webhook configuration
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations

# 6. Verify RBAC is intact
kubectl auth can-i list myresource --as=system:serviceaccount:my-operator-system:my-operator-controller-manager

If Vigilmon shows /healthz returning non-200 but the pod is Running, the custom reconciler health check is firing — look for stalled reconciliations in the logs.

Step 7: Create a status page for operator-managed services

Operator controllers underpin databases, message queues, and configuration management that many teams depend on. A status page makes failures visible without requiring kubectl access:

1. Go to Status Pages → New Status Page
2. Name it: "Platform Operators"
3. Add your monitors: Operator healthz, Operator readyz, Webhook server
4. Share the URL with application teams who depend on the operator's managed resources

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on /healthz | Controller crashes, reconcile loop stalls | | HTTP monitor on /readyz | Operator not yet ready after restart | | HTTP monitor on webhook server | Admission webhook failures blocking all CRD operations | | TCP monitor on webhook port | TLS failures, port-level outages | | Email + webhook alert channels | Immediate notification before resource drift accumulates | | Status page | Team self-service during operator incidents |

Operator SDK controllers are the machinery that keeps your platform resources in sync. External monitoring is the only reliable way to know when that machinery stops without waiting for a user-visible failure.

Get started at vigilmon.online — free tier, no credit card required.