tutorial

Monitoring kube-bench CIS Benchmark Runs: Scan Scheduling, Report Generation, and Kubernetes Compliance Posture Continuity

kube-bench runs the CIS Kubernetes Benchmark against your cluster components — the API server, scheduler, controller manager, etcd, kubelet, and more — produ...

kube-bench runs the CIS Kubernetes Benchmark against your cluster components — the API server, scheduler, controller manager, etcd, kubelet, and more — producing a structured report of passing and failing security controls. A passing baseline with scheduled re-runs gives you continuous compliance posture; a broken scan pipeline gives you stale data while the posture drifts.

The challenge: kube-bench doesn't run as a persistent service. It runs as a Job, completes, and exits. If the Job fails to schedule, fails to complete, or produces a report that nobody consumes, your compliance posture becomes unknown — and in regulated environments, unknown is worse than failing.

This guide covers how to monitor kube-bench scan scheduling, report generation success, finding regression detection, and how to use Vigilmon to maintain visibility into the infrastructure that keeps compliance scans running.

Understanding kube-bench Failure Modes

kube-bench runs typically fail in one of these ways:

  • Job never schedules — a tainted node, missing ServiceAccount, or RBAC misconfiguration prevents the Job from starting
  • Job starts but fails — insufficient permissions to read kubelet config or control plane component flags; the Job exits non-zero
  • Job completes but report isn't consumed — the scan runs, but no system reads the output or archives the report; failures go undetected
  • Finding regression — a previously passing control now fails due to a cluster upgrade, config change, or new CIS benchmark version

None of these generate an alert by default. You need explicit monitoring for each layer.

Monitoring kube-bench Job Scheduling

CronJob Health

The most common production pattern wraps kube-bench in a CronJob that runs on a schedule:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-bench
  namespace: kube-system
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true
          hostIPC: true
          hostNetwork: true
          serviceAccountName: kube-bench
          containers:
          - name: kube-bench
            image: aquasec/kube-bench:latest
            command: ["kube-bench", "--json", "--outputfile", "/output/report.json"]
            volumeMounts:
            - name: output
              mountPath: /output
            securityContext:
              privileged: true
          restartPolicy: Never
          volumes:
          - name: output
            emptyDir: {}

# Check CronJob execution history
kubectl get cronjob kube-bench -n kube-system

# View recent job runs and their status
kubectl get jobs -n kube-system -l app=kube-bench \
  --sort-by=.metadata.creationTimestamp | tail -7

# Check how long ago the last run completed
kubectl get cronjob kube-bench -n kube-system \
  -o jsonpath='{.status.lastSuccessfulTime}'

Alert if the last successful run is more than 25 hours ago (accounting for scheduling drift):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kube-bench-alerts
  namespace: kube-system
spec:
  groups:
  - name: kube-bench
    rules:
    - alert: KubeBenchScanOverdue
      expr: time() - kube_cronjob_status_last_successful_time{cronjob="kube-bench", namespace="kube-system"} > 90000
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "kube-bench scan is overdue"
        description: "The kube-bench CronJob has not completed successfully in over 25 hours. Compliance posture data is stale."

    - alert: KubeBenchJobFailed
      expr: kube_job_status_failed{job_name=~"kube-bench-.*", namespace="kube-system"} > 0
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: "kube-bench scan job failed"
        description: "A kube-bench job has exited with a failure status. Check pod logs for permission or configuration errors."

Job Pod Scheduling Issues

# Check if kube-bench pods are pending (scheduling failures)
kubectl get pods -n kube-system -l app=kube-bench | grep -v "Running\|Completed"

# Describe a stuck pod to find the scheduling failure reason
kubectl describe pod -n kube-system -l app=kube-bench | \
  grep -A5 "Events:"

# Common failure: missing node tolerations for control plane taints
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'

Control plane nodes in managed clusters are often tainted. kube-bench needs to run on those nodes to check control plane components:

# Add tolerations to the Job template for control plane nodes
tolerations:
- key: node-role.kubernetes.io/control-plane
  operator: Exists
  effect: NoSchedule
- key: node-role.kubernetes.io/master
  operator: Exists
  effect: NoSchedule
nodeSelector:
  node-role.kubernetes.io/control-plane: ""

Monitoring Report Generation Success

Output Verification

A completed kube-bench Job doesn't guarantee a useful report. Verify the output file is non-empty and valid JSON:

# Get the most recent kube-bench pod name
POD=$(kubectl get pods -n kube-system -l app=kube-bench \
  --sort-by=.metadata.creationTimestamp | tail -1 | awk '{print $1}')

# View the scan summary from the completed pod logs
kubectl logs -n kube-system "$POD" | tail -30

# If using JSON output to a PersistentVolume, validate the report
kubectl exec -n kube-system "$POD" -- \
  sh -c 'jq ".Totals" /output/report.json 2>/dev/null || echo "REPORT INVALID"'

Report Archival and Freshness

For long-term compliance posture tracking, export reports to object storage and verify freshness:

# Check that today's report was uploaded to S3
TODAY=$(date +%Y-%m-%d)
aws s3 ls "s3://your-compliance-bucket/kube-bench/${TODAY}/" 2>/dev/null || \
  echo "WARNING: No kube-bench report found for today"

# Get the most recent report and check its timestamp
LATEST=$(aws s3 ls s3://your-compliance-bucket/kube-bench/ | \
  sort | tail -1 | awk '{print $4}')
echo "Latest report: $LATEST"

A CronJob that archives reports and fails visibly if archival fails:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-bench-with-archive
  namespace: kube-system
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true
          serviceAccountName: kube-bench
          initContainers:
          - name: kube-bench
            image: aquasec/kube-bench:latest
            command:
            - /bin/sh
            - -c
            - |
              kube-bench --json > /output/report.json
              FAILED=$(jq '.Totals.total_fail // 0' /output/report.json)
              echo "kube-bench scan complete: $FAILED controls failing"
            volumeMounts:
            - name: output
              mountPath: /output
          containers:
          - name: archive
            image: amazon/aws-cli:latest
            command:
            - /bin/sh
            - -c
            - |
              DATE=$(date +%Y-%m-%d)
              aws s3 cp /output/report.json \
                "s3://your-compliance-bucket/kube-bench/${DATE}/report.json" || {
                echo "ERROR: Failed to archive kube-bench report"
                exit 1
              }
              echo "Report archived successfully"
            volumeMounts:
            - name: output
              mountPath: /output
          volumes:
          - name: output
            emptyDir: {}
          restartPolicy: Never

Monitoring for Finding Regressions

Tracking Pass/Fail Counts Over Time

The most important signal from kube-bench is not absolute pass/fail — it's whether the counts change. A control that was passing last week but is failing today indicates a regression.

Parse the JSON output to extract counts:

# Extract totals from the latest report
jq '{
  pass: .Totals.total_pass,
  fail: .Totals.total_fail,
  warn: .Totals.total_warn,
  info: .Totals.total_info
}' /output/report.json

A minimal regression detection script:

#!/bin/sh
# Compare today's report against yesterday's
TODAY_FAIL=$(jq '.Totals.total_fail' /reports/$(date +%Y-%m-%d).json 2>/dev/null || echo 0)
YESTERDAY_FAIL=$(jq '.Totals.total_fail' /reports/$(date -d "yesterday" +%Y-%m-%d).json 2>/dev/null || echo 0)

if [ "$TODAY_FAIL" -gt "$YESTERDAY_FAIL" ]; then
  REGRESSION=$(( TODAY_FAIL - YESTERDAY_FAIL ))
  echo "REGRESSION: $REGRESSION new failing controls detected"
  exit 1
fi

echo "No regression: $TODAY_FAIL failing controls (same as yesterday: $YESTERDAY_FAIL)"

Finding-Level Alerting

Track specific high-severity findings that should never fail:

# Extract specific failing controls by ID
jq -r '.Controls[] | .Tests[]? | .Results[]? |
  select(.status == "FAIL") |
  "\(.test_number): \(.test_desc)"' /output/report.json | \
  grep -E "^1\.(1|2)\." # API server checks

# Alert on critical control failures
CRITICAL_FAILS=$(jq -r '.Controls[] | .Tests[]? | .Results[]? |
  select(.status == "FAIL" and (.test_number | test("^1\\.(1|2)\\."))) |
  .test_number' /output/report.json | wc -l)

if [ "$CRITICAL_FAILS" -gt 0 ]; then
  echo "CRITICAL: $CRITICAL_FAILS API server CIS controls failing"
  exit 1
fi

Prometheus Metrics from kube-bench Reports

Export kube-bench results to Prometheus for dashboarding and alerting:

# Push kube-bench totals to Prometheus pushgateway
FAIL=$(jq '.Totals.total_fail' /output/report.json)
PASS=$(jq '.Totals.total_pass' /output/report.json)
WARN=$(jq '.Totals.total_warn' /output/report.json)

cat <<EOF | curl --data-binary @- http://pushgateway:9091/metrics/job/kube-bench
# HELP kube_bench_controls_total Number of CIS benchmark controls by status
# TYPE kube_bench_controls_total gauge
kube_bench_controls_total{status="fail"} ${FAIL}
kube_bench_controls_total{status="pass"} ${PASS}
kube_bench_controls_total{status="warn"} ${WARN}
EOF

Then alert in Prometheus:

- alert: KubeBenchFailCountIncreased
  expr: kube_bench_controls_total{status="fail"} > kube_bench_controls_total{status="fail"} offset 24h
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: "kube-bench failing control count has increased"
    description: "More CIS benchmark controls are failing than 24 hours ago. A recent cluster change may have introduced a security regression."

- alert: KubeBenchCriticalFails
  expr: kube_bench_controls_total{status="fail"} > 10
  for: 1h
  labels:
    severity: critical
  annotations:
    summary: "kube-bench reports more than 10 failing CIS controls"
    description: "Your Kubernetes cluster has significant CIS benchmark findings. Review the latest kube-bench report."

Integrating Vigilmon for Compliance Posture Continuity

Compliance posture is only as good as the scan pipeline that produces it. Vigilmon monitors your kube-bench infrastructure from outside the cluster so you know when the scanning system itself is broken:

Monitor your compliance status API. Deploy a lightweight endpoint that exposes the latest kube-bench results:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-bench-status-api
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-bench-status-api
  template:
    metadata:
      labels:
        app: kube-bench-status-api
    spec:
      containers:
      - name: api
        image: python:3.12-alpine
        command:
        - python3
        - -c
        - |
          import json, os, http.server, time
          class Handler(http.server.BaseHTTPRequestHandler):
            def do_GET(self):
              try:
                with open('/reports/latest.json') as f:
                  report = json.load(f)
                status = {
                  "scan_age_seconds": int(time.time()) - int(os.path.getmtime('/reports/latest.json')),
                  "fail": report['Totals']['total_fail'],
                  "pass": report['Totals']['total_pass'],
                  "warn": report['Totals']['total_warn'],
                  "status": "ok"
                }
                body = json.dumps(status).encode()
                self.send_response(200)
                self.send_header('Content-Type', 'application/json')
                self.end_headers()
                self.wfile.write(body)
              except Exception as e:
                self.send_response(503)
                self.end_headers()
                self.wfile.write(b'{"status":"error"}')
          http.server.HTTPServer(('', 8080), Handler).serve_forever()
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: reports
          mountPath: /reports
      volumes:
      - name: reports
        persistentVolumeClaim:
          claimName: kube-bench-reports
---
apiVersion: v1
kind: Service
metadata:
  name: kube-bench-status
  namespace: kube-system
spec:
  selector:
    app: kube-bench-status-api
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer

Then in Vigilmon:

  1. Log in to Vigilmon and go to Monitors → New Monitor.
  2. Add an HTTP / HTTPS monitor targeting https://kube-bench-status.yourdomain.com.
  3. Set check interval to 5 minutes.
  4. Add a keyword assertion for "status":"ok" to validate the API is returning real scan data.
  5. Add a second keyword assertion for "fail" to confirm the report is populated.
  6. Configure alert channels so your compliance or security team is paged if the endpoint goes down.

This gives you a 24/7 external confirmation that your kube-bench scanning pipeline is alive. If the CronJob stops running, the PVC unmounts, or the status API crashes, Vigilmon alerts you before your next compliance audit.

Key Metrics Summary

| Signal | What to Monitor | Alert Threshold | |--------|----------------|-----------------| | CronJob last success | kube_cronjob_status_last_successful_time | > 25h without a run | | Job failure | kube_job_status_failed | Any failure | | Failing controls | kube_bench_controls_total{status="fail"} | Increase vs 24h ago | | Critical API server controls | Check 1.1.x, 1.2.x failures | Any failure | | Report archival | S3/GCS object freshness | No upload in > 25h | | Status API | Vigilmon HTTP + keyword | Any failure for > 10m | | Pod scheduling | Pending pods in kube-system | Pending > 10m |

Conclusion

kube-bench gives you CIS benchmark compliance visibility — but only when the scan pipeline is running correctly. By monitoring CronJob execution health, verifying report generation success, tracking finding counts over time for regressions, and exposing a status API that Vigilmon checks externally, you convert kube-bench from a point-in-time audit tool into a continuous compliance monitoring system. The external Vigilmon check is the safety net: if the entire scanning infrastructure breaks, you know within minutes — not when an auditor asks for your latest compliance report and you discover it's three months old.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles