tutorial

How to Monitor kubectl-doctor with Vigilmon

kubectl-doctor is a kubectl plugin that scans Kubernetes workloads for common misconfigurations and anti-patterns, checking for missing resource requests and...

kubectl-doctor is a kubectl plugin that scans Kubernetes workloads for common misconfigurations and anti-patterns, checking for missing resource requests and limits, absent liveness and readiness probes, containers running as root, missing pod disruption budgets, and other reliability and security issues that accumulate silently in production clusters. Platform and SRE teams run kubectl-doctor as part of cluster health audits, pre-deployment gates, and weekly configuration reviews to surface technical debt before it causes availability incidents. When the Kubernetes API server that kubectl-doctor queries for workload configuration data is degraded, the plugin returns incomplete scan results or times out, leaving configuration debt invisible until it manifests as an outage.

In this tutorial you'll set up uptime monitoring for the Kubernetes infrastructure that kubectl-doctor depends on using Vigilmon — free tier, no credit card required.

Why kubectl-doctor's dependencies need external monitoring

kubectl-doctor scans all workload resources across namespaces by making broad list calls to the Kubernetes API for Deployments, StatefulSets, DaemonSets, Pods, and related resources. Its failure modes are tied to API availability, RBAC scope, and cluster scale:

  • API server unavailability returns empty scan results — kubectl-doctor exits cleanly with no findings when the Kubernetes API is unreachable rather than reporting a connection error; teams may interpret an empty result as "no issues found" when they should interpret it as "scan failed — results are meaningless"
  • RBAC gaps cause silently incomplete scans — if the user or service account running kubectl-doctor lacks list permission on one or more resource types (e.g., StatefulSets or DaemonSets), those resource classes are skipped without warning; the resulting report shows findings only for accessible resources and appears comprehensive to engineers who do not know which resources were excluded
  • Large clusters trigger API server rate limiting — kubectl-doctor performs broad list operations across all namespaces; in clusters with hundreds of namespaces and thousands of workloads, these list operations can hit API server rate limits, causing some resource pages to be skipped and producing partial findings without a clear indication that the scan was truncated
  • CRD workload types not supported by default — kubectl-doctor scans standard Kubernetes workload types; if your platform runs workloads via custom CRDs (e.g., Argo Rollouts, Knative Services, KEDA ScaledJobs), kubectl-doctor does not scan those resources and the configuration debt in CRD-managed workloads is never surfaced
  • Cluster version drift causes false positives — kubectl-doctor applies rule sets calibrated for specific Kubernetes API versions; if the cluster has been upgraded to a version where some API fields changed meaning or default values shifted, kubectl-doctor may flag compliant configurations as non-compliant, filling reports with noise that causes engineers to ignore genuine findings

External monitoring from Vigilmon watches the Kubernetes infrastructure that kubectl-doctor relies on and alerts you when workload configuration scans cannot be trusted to be complete.

What you'll need

  • kubectl-doctor installed (kubectl doctor --version returns without error)
  • kubectl access to your monitored cluster with broad read permissions
  • A node or LoadBalancer endpoint to host the health check
  • A free Vigilmon account

Step 1: Expose a health endpoint for kubectl-doctor's scan dependencies

kubectl-doctor has no built-in HTTP interface. Deploy a health exporter that validates the Kubernetes API is reachable and that workload resource listing — the core operation kubectl-doctor performs — works correctly:

# kubectl-doctor-health.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubectl-doctor-health
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubectl-doctor-health-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces", "nodes", "services", "endpoints"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
    verbs: ["get", "list"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubectl-doctor-health-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubectl-doctor-health-reader
subjects:
  - kind: ServiceAccount
    name: kubectl-doctor-health
    namespace: observability
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubectl-doctor-health
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubectl-doctor-health
  template:
    metadata:
      labels:
        app: kubectl-doctor-health
    spec:
      serviceAccountName: kubectl-doctor-health
      containers:
        - name: checker
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Validate API server is reachable
                if ! kubectl get nodes --request-timeout=10s > /dev/null 2>&1; then
                  echo "unhealthy api=unreachable nodes=inaccessible" > /tmp/status
                  sleep 30
                  continue
                fi

                NODE_COUNT=$(kubectl get nodes --no-headers 2>/dev/null | wc -l | tr -d ' ')

                # Validate workload resource listing (core kubectl-doctor operations)
                DEPLOY_COUNT=$(kubectl get deployments --all-namespaces --no-headers --request-timeout=15s 2>/dev/null | wc -l | tr -d ' ')
                if [ -z "$DEPLOY_COUNT" ] || [ "$DEPLOY_COUNT" = "0" ]; then
                  # May be a genuinely empty cluster — check if API returned
                  if ! kubectl get deployments --all-namespaces --request-timeout=10s > /dev/null 2>&1; then
                    echo "degraded api=reachable deployments=inaccessible rbac=check_required" > /tmp/status
                    sleep 30
                    continue
                  fi
                fi

                SS_COUNT=$(kubectl get statefulsets --all-namespaces --no-headers --request-timeout=15s 2>/dev/null | wc -l | tr -d ' ')
                DS_COUNT=$(kubectl get daemonsets --all-namespaces --no-headers --request-timeout=15s 2>/dev/null | wc -l | tr -d ' ')
                POD_COUNT=$(kubectl get pods --all-namespaces --no-headers --request-timeout=15s 2>/dev/null | wc -l | tr -d ' ')
                PDB_COUNT=$(kubectl get pdb --all-namespaces --no-headers --request-timeout=15s 2>/dev/null | wc -l | tr -d ' ')
                NS_COUNT=$(kubectl get namespaces --no-headers --request-timeout=10s 2>/dev/null | wc -l | tr -d ' ')

                echo "healthy api=reachable nodes=$NODE_COUNT namespaces=$NS_COUNT deployments=$DEPLOY_COUNT statefulsets=$SS_COUNT daemonsets=$DS_COUNT pods=$POD_COUNT pdbs=$PDB_COUNT" > /tmp/status
                sleep 60
              done
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if echo "$STATUS" | grep -q "^healthy"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: kubectl-doctor-health
  namespace: observability
spec:
  type: NodePort
  selector:
    app: kubectl-doctor-health
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30092

kubectl apply -f kubectl-doctor-health.yaml

# Verify
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:30092/
# healthy api=reachable nodes=3 namespaces=12 deployments=47 statefulsets=8 daemonsets=4 pods=186 pdbs=12

Step 2: Validate kubectl-doctor and workload scan completeness

Before setting up monitoring, confirm kubectl-doctor can perform complete scans with the expected RBAC coverage:

#!/bin/bash
# check-kubectl-doctor.sh

echo "=== kubectl-doctor installation ==="
if kubectl doctor --version > /dev/null 2>&1 || kubectl doctor --help > /dev/null 2>&1; then
  echo "OK: kubectl-doctor installed"
else
  echo "FAILED: kubectl-doctor not found — install via: kubectl krew install doctor"
  exit 1
fi

echo ""
echo "=== RBAC coverage for kubectl-doctor ==="
RESOURCES=(
  "pods"
  "deployments"
  "statefulsets"
  "daemonsets"
  "replicasets"
  "services"
  "endpoints"
  "poddisruptionbudgets"
  "horizontalpodautoscalers"
)

RBAC_OK=true
for resource in "${RESOURCES[@]}"; do
  if kubectl auth can-i list "$resource" --all-namespaces > /dev/null 2>&1; then
    echo "OK: can list $resource across all namespaces"
  else
    echo "WARNING: cannot list $resource — kubectl-doctor will skip this resource type"
    RBAC_OK=false
  fi
done

if [ "$RBAC_OK" = "false" ]; then
  echo ""
  echo "WARNING: incomplete RBAC — kubectl-doctor scan results will be partial"
fi

echo ""
echo "=== Workload resource counts (kubectl-doctor scan scope) ==="
echo "Deployments:   $(kubectl get deployments --all-namespaces --no-headers 2>/dev/null | wc -l)"
echo "StatefulSets:  $(kubectl get statefulsets --all-namespaces --no-headers 2>/dev/null | wc -l)"
echo "DaemonSets:    $(kubectl get daemonsets --all-namespaces --no-headers 2>/dev/null | wc -l)"
echo "Pods:          $(kubectl get pods --all-namespaces --no-headers 2>/dev/null | wc -l)"
echo "PDBs:          $(kubectl get pdb --all-namespaces --no-headers 2>/dev/null | wc -l)"
echo "HPAs:          $(kubectl get hpa --all-namespaces --no-headers 2>/dev/null | wc -l)"

echo ""
echo "=== kubectl-doctor scan test ==="
echo "Running kubectl-doctor (may take time on large clusters)..."
SCAN_RESULT=$(kubectl doctor 2>&1 | head -20)
if echo "$SCAN_RESULT" | grep -q "error\|failed\|cannot"; then
  echo "WARNING: kubectl-doctor reported errors during scan:"
  echo "$SCAN_RESULT"
else
  echo "OK: kubectl-doctor scan completed without errors"
  FINDING_COUNT=$(echo "$SCAN_RESULT" | grep -c "WARNING\|ERROR\|CRITICAL" || true)
  echo "Findings in first 20 lines: $FINDING_COUNT"
fi

chmod +x check-kubectl-doctor.sh
./check-kubectl-doctor.sh

Step 3: Set up HTTP monitoring in Vigilmon

With the health endpoint running, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add monitors for kubectl-doctor's workload scan infrastructure:

| Monitor name | URL | Expected status | |---|---|---| | kubectl-doctor scan health | http://your-node-ip:30092/ | 200 | | Kubernetes API livez | https://your-api-server:6443/livez | 200 | | Kubernetes API readyz | https://your-api-server:6443/readyz | 200 | | Kubernetes workload API | https://your-api-server:6443/apis/apps/v1 | 200 |

  1. Set the check interval to 1 minute
  2. Under Expected response, match healthy in the response body for the health endpoint
  3. For the workload API monitor, match deployments in the response body to confirm the apps/v1 API group is serving correctly
  4. Save each monitor

Step 4: Add a TCP monitor for the Kubernetes API server

kubectl-doctor makes extensive API calls during each scan. A TCP monitor catches network failures before the scan begins:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your Kubernetes API server hostname and port 6443
  4. Set the check interval to 1 minute
  5. Save the monitor

A TCP failure on port 6443 means kubectl-doctor cannot perform any part of the workload configuration scan, returning empty results that appear identical to a clean scan in environments that do not distinguish between "no findings" and "scan failed."

Step 5: Configure alert channels

kubectl-doctor scans are often run on a schedule — nightly or weekly — as part of cluster hygiene processes. When the API infrastructure is degraded, these scheduled scans fail silently and configuration debt accumulates undetected across the gap between successful scans.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your platform engineering and SRE on-call addresses
  3. Assign the channel to all kubectl-doctor monitors

Webhook alerts

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
  3. Sample Vigilmon webhook payload:
{
  "monitor_name": "kubectl-doctor scan health",
  "status": "down",
  "url": "http://your-node-ip:30092/",
  "started_at": "2024-01-15T22:00:00Z",
  "duration_seconds": 120
}

Route this to a runbook that instructs platform engineers to check the API server status and reschedule kubectl-doctor scans once the infrastructure is restored, rather than trusting the last successful scan results during the outage window.

Step 6: Correlate Vigilmon alerts with kubectl-doctor diagnostics

When you receive a kubectl-doctor scan health downtime alert:

# 1. Check Kubernetes API server status
kubectl cluster-info
kubectl get nodes

# 2. Test workload resource listing (core kubectl-doctor operations)
kubectl get deployments --all-namespaces --no-headers | wc -l
kubectl get statefulsets --all-namespaces --no-headers | wc -l
kubectl get daemonsets --all-namespaces --no-headers | wc -l

# 3. Check RBAC completeness for kubectl-doctor
kubectl auth can-i list deployments --all-namespaces
kubectl auth can-i list statefulsets --all-namespaces
kubectl auth can-i list pods --all-namespaces
kubectl auth can-i list poddisruptionbudgets --all-namespaces

# 4. Test the apps/v1 API group specifically
kubectl get --raw /apis/apps/v1 | head -20

# 5. Check for API rate limiting
kubectl get events --all-namespaces --field-selector reason=FailedCreate | grep rate

# 6. Run kubectl-doctor manually to check for errors
kubectl doctor 2>&1 | head -40

# 7. Inspect the health pod logs
kubectl logs -n observability -l app=kubectl-doctor-health -c checker --tail=20

# 8. Check for API server errors that would affect list operations
kubectl logs -n kube-system -l component=kube-apiserver --tail=50 2>/dev/null | \
  grep -E "error|timeout|rate limit" | tail -20

# 9. Validate specific namespaces are listable
kubectl get pods -n kube-system --no-headers | wc -l
kubectl get pods -n observability --no-headers | wc -l

If the health endpoint returns degraded api=reachable deployments=inaccessible, the RBAC configuration for the kubectl-doctor service account was modified and the plugin no longer has the permissions needed for a complete scan. Re-apply the ClusterRoleBinding and verify with kubectl auth can-i list deployments --all-namespaces.

Step 7: Create a status page for Kubernetes workload scan visibility

Make kubectl-doctor's scan infrastructure health visible to the platform and SRE teams:

  1. Go to Status Pages → New Status Page
  2. Name it: "Workload Configuration Scanning (kubectl-doctor)"
  3. Add the kubectl-doctor scan health monitor, API livez/readyz monitors, and the workload API monitor
  4. Share the URL with your platform engineering and SRE teams

During a scheduled scan failure investigation, engineers checking this page know immediately whether the API infrastructure is the blocker and whether to wait for recovery or escalate to cluster operators.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on kubectl-doctor health endpoint | API degradation, workload listing failures, RBAC gaps | | HTTP monitor on API livez/readyz | API server degradation affecting all workload scans | | HTTP monitor on apps/v1 workload API | Specific workload API group failures | | TCP monitor on port 6443 | Network failures preventing all scan operations | | Email + webhook alert channels | Immediate notification when configuration scanning is unreliable | | Status page | Platform team visibility into scan infrastructure health |

kubectl-doctor prevents configuration debt from becoming availability incidents — but only if its scans are running reliably and returning complete results. External monitoring ensures you know when the Kubernetes API infrastructure is degraded, so you can trust that a clean scan means "no findings" rather than "scan failed silently."

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles