tutorial

How to Monitor Kubent with Vigilmon

Kubent (kube-no-trouble) is a tool that scans your Kubernetes cluster and reports any resources that use API versions which have been deprecated or completel...

Kubent (kube-no-trouble) is a tool that scans your Kubernetes cluster and reports any resources that use API versions which have been deprecated or completely removed in newer versions of Kubernetes. In production environments, a missed deprecated API causes a hard failure when the cluster control plane is upgraded — a deployment that uses extensions/v1beta1 Ingress will stop working the moment that API group is removed. Kubent is typically run as a Kubernetes CronJob that continuously checks the cluster and alerts engineers before a planned upgrade.

But Kubent's own scanning job can fail silently. If the CronJob misses its schedule, the scan pod crashes on startup, or the result exporter goes down, your team loses its early-warning system for API deprecation — without knowing the warning system itself is broken. In this tutorial you'll set up uptime monitoring for your Kubent deployment using Vigilmon — free tier, no credit card required.

Why Kubent deployments need external monitoring

A CronJob-based Kubent deployment has several failure modes that produce no outward signal:

  • CronJob suspended or missing — if the CronJob is accidentally deleted or suspended by a GitOps reconciler, scans stop entirely; the absence of scan results looks identical to "nothing deprecated found"
  • Pod startup failures — the scan pod may fail to start due to image pull errors, resource limits, or missing RBAC permissions, leaving the last scan stale while the cluster accumulates new deprecated resources
  • Kubeconfig or service account expiry — Kubent needs API access to scan the cluster; an expired or rotated credential causes every scan to exit with an authentication error, silently producing no output
  • Result exporter crash — when Kubent output is forwarded to a Prometheus metric or an HTTP status endpoint, a crash in the exporter means metrics disappear without affecting the CronJob's reported success
  • Scan result file not updated — the job may exit zero (success) but fail to write its output when a permissions issue affects the mounted volume or reporting channel

External monitoring from Vigilmon watches the HTTP status endpoint your Kubent deployment exposes and alerts you when scans stop completing.

What you'll need

  • A Kubernetes cluster with Kubent installed or accessible
  • A CronJob that runs Kubent scans (see Step 1)
  • An HTTP status endpoint that reports recent scan results
  • A free Vigilmon account

Step 1: Deploy Kubent as a CronJob with a status exporter

Kubent does not expose HTTP endpoints natively. The standard pattern runs Kubent as a CronJob and writes the result to a shared volume read by a small HTTP status server.

# kubent-deployment.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: kubent
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubent
  namespace: kubent
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubent-reader
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubent-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubent-reader
subjects:
  - kind: ServiceAccount
    name: kubent
    namespace: kubent
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kubent-results
  namespace: kubent
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 100Mi
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kubent-scan
  namespace: kubent
spec:
  schedule: "0 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kubent
          restartPolicy: OnFailure
          containers:
            - name: kubent
              image: ghcr.io/doitintl/kube-no-trouble:latest
              args: ["-o", "json", "-t"]
              env:
                - name: SCAN_TIMESTAMP
                  value: ""
              command:
                - /bin/sh
                - -c
                - |
                  TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
                  kubent -o json 2>&1 > /results/latest.json
                  EXIT_CODE=$?
                  echo "{\"timestamp\":\"$TIMESTAMP\",\"exit_code\":$EXIT_CODE}" > /results/meta.json
                  echo "Scan completed at $TIMESTAMP with exit code $EXIT_CODE"
              volumeMounts:
                - name: results
                  mountPath: /results
          volumes:
            - name: results
              persistentVolumeClaim:
                claimName: kubent-results
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubent-status
  namespace: kubent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubent-status
  template:
    metadata:
      labels:
        app: kubent-status
    spec:
      containers:
        - name: status-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                if [ -f /results/meta.json ]; then
                  TS=$(grep -o '"timestamp":"[^"]*"' /results/meta.json | cut -d'"' -f4)
                  EXIT=$(grep -o '"exit_code":[0-9]*' /results/meta.json | cut -d: -f2)
                  # Check if scan is within the last 2 hours
                  NOW=$(date -u +%s)
                  SCAN_S=$(date -u -d "$TS" +%s 2>/dev/null || echo 0)
                  AGE=$((NOW - SCAN_S))
                  if [ "$EXIT" = "0" ] && [ "$AGE" -lt 7200 ]; then
                    STATUS="healthy last_scan=$TS"
                    CODE="200"
                  elif [ "$AGE" -ge 7200 ]; then
                    STATUS="stale last_scan=$TS age_seconds=$AGE"
                    CODE="503"
                  else
                    STATUS="scan_failed exit_code=$EXIT last_scan=$TS"
                    CODE="503"
                  fi
                else
                  STATUS="no_scan_results"
                  CODE="503"
                fi
                printf "HTTP/1.1 $CODE OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: results
              mountPath: /results
      volumes:
        - name: results
          persistentVolumeClaim:
            claimName: kubent-results
---
apiVersion: v1
kind: Service
metadata:
  name: kubent-status
  namespace: kubent
spec:
  type: NodePort
  selector:
    app: kubent-status
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30082

kubectl apply -f kubent-deployment.yaml

# Trigger a manual scan to populate initial results
kubectl create job -n kubent --from=cronjob/kubent-scan kubent-manual-scan

# Wait for the job and verify the status endpoint
kubectl wait -n kubent --for=condition=complete job/kubent-manual-scan --timeout=120s
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:30082/
# healthy last_scan=2024-01-15T10:00:00Z

Step 2: Verify Kubent scan results and check for deprecated APIs

Before monitoring, run a manual scan and review the output to establish your baseline:

# Run Kubent manually to see current deprecated API usage
kubectl run kubent-check -n kubent \
  --image=ghcr.io/doitintl/kube-no-trouble:latest \
  --restart=Never \
  --serviceaccount=kubent \
  --rm -it -- kubent

# Review the JSON output from the last automated scan
kubectl exec -n kubent \
  $(kubectl get pod -n kubent -l app=kubent-status -o name | head -1) \
  -- cat /results/latest.json | python3 -m json.tool

# Check the CronJob schedule and last run time
kubectl get cronjob kubent-scan -n kubent
kubectl get jobs -n kubent --sort-by=.metadata.creationTimestamp

# Inspect recent job failures
kubectl describe jobs -n kubent | grep -A 10 "Events:"

This confirms the scan is running, results are being written, and any deprecated APIs already present in the cluster are documented before you enable alerting.

Step 3: Set up HTTP monitoring in Vigilmon

With the status endpoint running, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add monitors for your endpoints:

| Monitor name | URL | Expected status | |---|---|---| | Kubent scan health | http://your-node-ip:30082/ | 200 | | Kubent CronJob last run | http://your-node-ip:30082/ | 200 (body contains healthy) |

  1. Set the check interval to 5 minutes (scans run hourly, so a 5-minute interval catches staleness quickly)
  2. Under Expected response, set status code 200 and match healthy in the response body
  3. Save the monitor

Step 4: Monitor the Kubent status TCP port

Add a TCP monitor to detect networking issues independently of the HTTP layer:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your node IP or Ingress hostname and port 30082
  4. Save the monitor

A TCP failure while the status pod is Running indicates a NodePort firewall rule change or a network policy that was applied to the kubent namespace.

Step 5: Configure alert channels

A silent Kubent failure means your team will discover deprecated APIs only when a cluster upgrade hard-fails a running workload. Alert immediately when scans stop completing.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your platform engineering or cluster admin team address
  3. Assign the channel to all Kubent monitors

Webhook alerts for incident routing

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
  3. The payload Vigilmon sends:
{
  "monitor_name": "Kubent scan health",
  "status": "down",
  "url": "http://your-node-ip:30082/",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Route this alert to a runbook that checks the CronJob schedule, inspects recent job failures, and triggers a manual scan to restore current scan results.

Step 6: Correlate Vigilmon alerts with Kubent diagnostics

When you receive a Kubent downtime alert, run these checks:

# 1. Check if the CronJob is suspended
kubectl get cronjob kubent-scan -n kubent -o jsonpath='{.spec.suspend}'

# 2. Check recent job history
kubectl get jobs -n kubent --sort-by=.metadata.creationTimestamp | tail -5

# 3. Inspect failed job pods
kubectl get pods -n kubent --field-selector=status.phase=Failed

# 4. Check the most recent job logs
LATEST_JOB=$(kubectl get jobs -n kubent --sort-by=.metadata.creationTimestamp -o name | tail -1)
kubectl logs -n kubent -l "job-name=${LATEST_JOB##*/}" --tail=50

# 5. Verify the service account permissions are intact
kubectl auth can-i list deployments --as=system:serviceaccount:kubent:kubent -A
kubectl auth can-i list customresourcedefinitions --as=system:serviceaccount:kubent:kubent

# 6. Check the status deployment is healthy
kubectl get deployment kubent-status -n kubent
kubectl logs -n kubent -l app=kubent-status --tail=20

# 7. Check the PVC is still bound
kubectl get pvc kubent-results -n kubent

# 8. Manually trigger a new scan
kubectl create job -n kubent --from=cronjob/kubent-scan kubent-recovery-scan-$(date +%s)

If Vigilmon shows a 503, the most common causes are a suspended CronJob (re-enable with kubectl patch cronjob kubent-scan -n kubent -p '{"spec":{"suspend":false}}') or an expired service account token.

Step 7: Create a status page for API deprecation scanning

Make Kubent's scan health visible to the broader platform team so everyone knows when deprecation scanning is active:

  1. Go to Status Pages → New Status Page
  2. Name it: "Kubernetes API Deprecation Scanner"
  3. Add your monitors: Kubent scan health
  4. Share the URL with cluster administrators and the platform engineering team

This makes it easy to confirm that deprecation scanning is running before a planned Kubernetes version upgrade.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on status endpoint | Stale scans, job failures, service account expiry, CronJob suspension | | TCP monitor on status port | Networking issues blocking access to scan results | | Email + webhook alert channels | Immediate notification when deprecation scanning stops | | Status page | Team visibility into API deprecation scanner health before upgrades |

Kubent is your last line of defense against deprecated API failures during Kubernetes upgrades. External monitoring is the only reliable way to know when that defense has silently stopped working.

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles