tutorial

Monitoring KubePug with Vigilmon: Preflight Job Health, API Server Connectivity, Deprecation Scan Results & Cluster Readiness Alerts

How to monitor KubePug with Vigilmon — preflight job health checks, Kubernetes API server connectivity, deprecation scan result endpoints, and cluster readiness alerts to ensure your API deprecation checks always run before cluster upgrades.

KubePug is the Kubernetes preflight checker that scans your cluster for API group deprecations and removals before you upgrade Kubernetes versions — the tool that prevents the discovery, in production, that your Helm charts still use networking.k8s.io/v1beta1 Ingress after the control plane upgrade to 1.22 removed it. In cluster upgrade workflows, KubePug runs as a mandatory preflight gate: the upgrade proceeds only after KubePug reports zero deprecated API usages in the target version. When the preflight CronJob fails silently, the upgrade gate becomes a rubber stamp. When KubePug cannot reach the Kubernetes API server to enumerate deployed resources, it returns a clean bill of health that is actually a scan failure — not a real result. Vigilmon gives you external visibility into KubePug's job health, the Kubernetes API connectivity it depends on, and the endpoint that exposes scan results, so your cluster upgrade process is always backed by real preflight data.

What You'll Build

  • A Heartbeat monitor that verifies the KubePug preflight CronJob completes successfully on schedule
  • HTTP monitors on the Kubernetes API server's liveness and readiness endpoints
  • An HTTP monitor on a scan-results endpoint that exposes KubePug's latest findings
  • SSL certificate monitoring for the Kubernetes API server certificate
  • An alert runbook mapping Vigilmon findings to specific kubectl and kubepug remediation commands

Prerequisites

  • KubePug (v1.0+) installed as a kubectl plugin or Helm chart in your cluster
  • A CronJob or CI/CD pipeline that runs KubePug scans on a schedule before planned upgrades
  • kubectl access to the cluster
  • A free account at vigilmon.online

Step 1: Understand KubePug's Execution Model

KubePug does not run as a persistent server — it is a batch tool that queries the Kubernetes API, checks deployed resources against a deprecation data file, and produces a report. Monitoring KubePug means monitoring its execution pipeline:

# Install the kubectl plugin
kubectl krew install deprecations

# Run a deprecation scan against the next Kubernetes version
kubectl deprecations --k8s-version v1.30

# Run KubePug as a standalone binary
kubepug --k8s-version=v1.30 --output=json > kubepug-results.json

# Check the results for deprecated APIs
cat kubepug-results.json | jq '.DeprecatedAPIs[] | {group: .Group, version: .Version, kind: .Kind}'

# Check for deleted APIs (breaking changes on upgrade)
cat kubepug-results.json | jq '.DeletedAPIs[] | {group: .Group, version: .Version, kind: .Kind}'

The critical distinction:

  • DeprecatedAPIs: APIs that still work in the target version but are planned for removal — warnings, not blockers.
  • DeletedAPIs: APIs that have been removed in the target version — hard blockers that will break workloads on upgrade.

A scan that returns empty results because KubePug could not connect to the API server is indistinguishable from a scan that returned empty results because there are no deprecated APIs. External monitoring fills this gap.

Step 2: Create a KubePug Preflight CronJob with Heartbeat Reporting

Deploy KubePug as a CronJob that runs weekly (or before each planned maintenance window) and reports its completion status to Vigilmon:

# kubepug-preflight-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kubepug-preflight
  namespace: monitoring
spec:
  schedule: "0 6 * * 1"  # Every Monday at 06:00 UTC
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kubepug-scanner
          restartPolicy: OnFailure
          containers:
          - name: kubepug
            image: ghcr.io/kubepug/kubepug:v1.7.1
            args:
            - --k8s-version=v1.30
            - --output=json
            - --filename=/results/scan.json
            volumeMounts:
            - name: results
              mountPath: /results
          - name: reporter
            image: curlimages/curl:8.5.0
            command:
            - sh
            - -c
            - |
              # Wait for kubepug to complete
              sleep 30
              DELETED=$(cat /results/scan.json | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d.get('DeletedAPIs', [])))")
              if [ "$DELETED" -gt "0" ]; then
                echo "BLOCKING: $DELETED deleted API(s) found — upgrade will break workloads"
                curl -m 10 https://vigilmon.online/api/heartbeats/YOUR_HEARTBEAT_ID/fail
              else
                echo "CLEAR: no deleted APIs for target version"
                curl -m 10 https://vigilmon.online/api/heartbeats/YOUR_HEARTBEAT_ID
              fi
            volumeMounts:
            - name: results
              mountPath: /results
          volumes:
          - name: results
            emptyDir: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubepug-scanner
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubepug-reader
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubepug-scanner
subjects:
- kind: ServiceAccount
  name: kubepug-scanner
  namespace: monitoring
roleRef:
  kind: ClusterRole
  name: kubepug-reader
  apiGroup: rbac.authorization.k8s.io

Apply the resources:

kubectl apply -f kubepug-preflight-cronjob.yaml

# Trigger a manual run to verify the setup
kubectl create job --from=cronjob/kubepug-preflight kubepug-manual-test -n monitoring
kubectl logs -n monitoring job/kubepug-manual-test -c reporter

Step 3: Create a Vigilmon Heartbeat Monitor

  1. Log in to Vigilmon → Add Monitor → Heartbeat.
  2. Expected interval: 1 week (10080 minutes) if running weekly.
  3. Grace period: 120 minutes (allows for cluster load variance).
  4. Label: kubepug preflight scan.
  5. Click Save and copy the generated heartbeat URL into the CronJob manifest above.

This monitor catches:

  • CronJob pod failing to schedule due to cluster resource pressure
  • KubePug binary failing to connect to the Kubernetes API (scan returns no results but exits non-zero)
  • RBAC misconfiguration preventing the scanner from listing cluster resources
  • Image pull failures when the KubePug container image is unavailable
  • Silent job failures where the CronJob completes but the reporter container does not run

Why heartbeat instead of polling: KubePug runs on a schedule, not continuously. A heartbeat monitor is the correct pattern — you expect a ping every N hours and alert if it stops arriving, rather than checking a persistent endpoint.

Step 4: Expose Scan Results via an HTTP Endpoint

For integration with upgrade approval workflows, expose the latest KubePug scan results via a simple HTTP endpoint so Vigilmon and other systems can query them:

# scan-results-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubepug-results-server
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubepug-results-server
  template:
    metadata:
      labels:
        app: kubepug-results-server
    spec:
      containers:
      - name: server
        image: python:3.12-alpine
        command:
        - python3
        - -m
        - http.server
        - "8080"
        - --directory
        - /results
        volumeMounts:
        - name: results
          mountPath: /results
      volumes:
      - name: results
        configMap:
          name: kubepug-latest-results

The CronJob writes results to this ConfigMap after each scan. Then in Vigilmon:

  1. Add Monitor → HTTP.
  2. URL: https://kubepug-results.example.com/scan.json.
  3. Expected status: 200.
  4. Keyword: ScanTime (present in every valid KubePug JSON output).
  5. Label: kubepug scan results endpoint.
  6. Click Save.

Upgrade gate integration: Your upgrade approval workflow can query this endpoint before proceeding. If the endpoint returns stale results (timestamp older than 7 days), the upgrade gate should block until a fresh scan is available.

Step 5: Monitor the Kubernetes API Server

KubePug's scan quality depends entirely on the Kubernetes API server being available and returning accurate resource state. Monitor the API server's health endpoints directly:

# Find your API server address
kubectl cluster-info
# Kubernetes control plane is running at https://k8s-api.example.com:6443

# Check API server liveness
curl -k https://k8s-api.example.com:6443/livez
# Expected: "ok"

# Check API server readiness (confirms it's serving requests)
curl -k https://k8s-api.example.com:6443/readyz
# Expected: "ok"

In Vigilmon:

API server liveness:

  1. Add Monitor → HTTP.
  2. URL: https://k8s-api.example.com:6443/livez.
  3. Expected status: 200.
  4. Keyword: ok.
  5. Label: kubernetes api server liveness.
  6. Click Save.

API server readiness:

  1. Add Monitor → HTTP.
  2. URL: https://k8s-api.example.com:6443/readyz.
  3. Expected status: 200.
  4. Keyword: ok.
  5. Label: kubernetes api server readiness.
  6. Click Save.

TCP check on port 6443:

  1. Add Monitor → TCP.
  2. Host: k8s-api.example.com.
  3. Port: 6443.
  4. Label: kubernetes api server tcp.
  5. Click Save.

Step 6: Monitor the API Server SSL Certificate

Kubernetes API server certificates are time-limited. An expired certificate causes all kubectl and KubePug operations to fail with TLS errors — the preflight scan cannot run and the upgrade gate is broken at the worst possible time:

# Check the API server certificate expiry
openssl s_client -connect k8s-api.example.com:6443 2>/dev/null | \
  openssl x509 -noout -dates

# For kubeadm-managed clusters, check all component certificates
kubeadm certs check-expiration

In Vigilmon:

  1. Add Monitor → SSL Certificate.
  2. Domain: k8s-api.example.com.
  3. Port: 6443.
  4. Alert when expiry is within: 30 days.
  5. Alert again: 14 days, 7 days, 3 days, 1 day.
  6. Click Save.

kubeadm certificate rotation: kubeadm-managed clusters issue 1-year certificates by default. Clusters running for more than 11 months without certificate rotation are at risk. Run kubeadm certs renew all before the certificates expire, not after — rotating an expired API server certificate requires restarting the control plane.

Step 7: Configure Alerting and Upgrade Runbook

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Immediate Action | |---|---|---| | Preflight heartbeat | Missed heartbeat | kubectl get cronjob -n monitoring kubepug-preflight; check last job run: kubectl get jobs -n monitoring | | Scan results endpoint | Non-200 or keyword missing | Check results server pod; verify ConfigMap was updated by last scan job | | API server liveness | Non-200 | Check control plane pods: kubectl get pods -n kube-system -l tier=control-plane | | API server readiness | Non-200 | API is live but not ready; check etcd connectivity and leader election | | API server TCP | Connection refused | Check API server load balancer; verify control plane node status | | SSL certificate | < 30 days to expiry | kubeadm certs check-expiration; schedule certificate renewal window |

Upgrade gate policy: Never proceed with a Kubernetes version upgrade unless:

  1. The preflight heartbeat fired successfully within the last 7 days.
  2. The scan results endpoint returns results with zero DeletedAPIs.
  3. The API server SSL certificate has more than 30 days remaining.

Common KubePug Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | CronJob pod fails to schedule due to node pressure | Preflight heartbeat missed at next expected interval | | RBAC role removed; scanner cannot list resources | Scan completes with empty results; heartbeat sent but scan data is invalid | | API server certificate expired; kubepug fails to connect | SSL monitor catches 30 days early; API TCP check detects connection drop at expiry | | etcd leader election failure makes API server not ready | /readyz returns non-200; TCP check still succeeds (important distinction) | | KubePug image version pinned to deleted tag | Pod fails to start; heartbeat missed | | Cluster upgrade proceeds without preflight scan | Heartbeat missed; upgrade blocked by Vigilmon alert if policy enforced | | Namespace quota prevents scanner job from running | Job queues indefinitely; heartbeat missed at expected interval | | Results ConfigMap not updated after scan job completes | Scan results endpoint returns stale data; keyword check catches wrong ScanTime |

Kubernetes version upgrades remove APIs without warning at the cluster level — your workloads simply stop working after the control plane upgrade if they reference a deleted API group. KubePug is the safety check that prevents this outcome, but it only works if it actually runs and actually reaches the API server. A silently failed preflight scan is worse than no scan at all, because it creates false confidence. Vigilmon closes this blind spot by monitoring the KubePug CronJob heartbeat, the Kubernetes API server health, and the scan results endpoint, so your upgrade preflight process is always independently verified and you never upgrade on stale or missing data.

Start monitoring your KubePug preflight pipeline in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles