Monitoring Kyverno with Vigilmon: Policy Engine Health, Admission Webhook Alerts & Controller Status

Kyverno is the Kubernetes-native policy engine that validates, mutates, and generates resources at admission time. Every kubectl apply, every Helm deployment, every ArgoCD sync passes through Kyverno's admission webhook before Kubernetes accepts it. When Kyverno is healthy, it's invisible. When it fails, one of two things happens: either Kubernetes rejects every workload write because the webhook is unavailable (catastrophic), or Kyverno silently stops enforcing policies because background processing has stalled (a security gap you won't notice until audit time). Vigilmon monitors Kyverno's health endpoints from outside the cluster and alerts your team the moment either failure mode appears.

What You'll Build

• HTTP monitors for Kyverno's admission webhook health endpoint
• HTTP monitors for each Kyverno controller replica
• A heartbeat monitor to verify background policy scans are running
• Tiered alert routing: webhook failures page on-call, controller degradation notifies Slack

Prerequisites

• Kyverno installed in a Kubernetes cluster (version 1.10+)
• Kyverno deployed in HA mode (3 replicas recommended for production)
• An Ingress controller exposing Kyverno health endpoints over HTTPS
• A free account at vigilmon.online

Why Kyverno Monitoring Is Not Optional

Kyverno's failure modes are uniquely risky:

Webhook downtime blocks cluster operations. Kyverno registers itself as a ValidatingWebhookConfiguration and MutatingWebhookConfiguration with failurePolicy: Fail by default. If the webhook service is unreachable, Kubernetes rejects all matching resource writes — no new pods, no new deployments, nothing. A 5-minute Kyverno outage during a production deploy can halt your entire CI/CD pipeline.

Silent policy bypass is a security risk. If Kyverno's background controller crashes, existing resources that violate policies won't be detected. New violations that slip through during webhook degradation won't be backfilled. Your security posture erodes invisibly.

HA doesn't protect against all failure modes. Even with 3 replicas, a misconfigured policy that panics the admission controller or a memory exhaustion event can take down all replicas within seconds. Internal Kubernetes health probes catch this eventually — but external monitoring catches it faster, from the perspective of the cluster's actual users.

Step 1: Verify Kyverno's Health Endpoints

Kyverno exposes health endpoints on each pod. The admission controller (which handles webhook requests) uses port 9443, while the separate metrics and health ports use 8000:

# Check admission controller health inside the cluster
kubectl exec -n kyverno deploy/kyverno-admission-controller -- \
  wget -qO- http://localhost:8000/health/liveness

# Check readiness
kubectl exec -n kyverno deploy/kyverno-admission-controller -- \
  wget -qO- http://localhost:8000/health/readiness

Both return HTTP 200 when healthy. Check all three Kyverno components:

# Admission controller
kubectl get pods -n kyverno -l app.kubernetes.io/component=admission-controller

# Background controller (runs background policy scans)
kubectl get pods -n kyverno -l app.kubernetes.io/component=background-controller

# Reports controller (generates PolicyReport resources)
kubectl get pods -n kyverno -l app.kubernetes.io/component=reports-controller

Step 2: Expose Health Endpoints via Ingress

Create a Service and Ingress for each Kyverno component you want to monitor:

# kyverno-health-services.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: kyverno-admission-health
  namespace: kyverno
spec:
  selector:
    app.kubernetes.io/component: admission-controller
  ports:
    - name: health
      port: 8000
      targetPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: kyverno-background-health
  namespace: kyverno
spec:
  selector:
    app.kubernetes.io/component: background-controller
  ports:
    - name: health
      port: 8000
      targetPort: 8000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kyverno-health
  namespace: kyverno
spec:
  rules:
    - host: kyverno-health.internal.yourdomain.com
      http:
        paths:
          - path: /admission/liveness
            pathType: Prefix
            backend:
              service:
                name: kyverno-admission-health
                port:
                  number: 8000
          - path: /admission/readiness
            pathType: Prefix
            backend:
              service:
                name: kyverno-admission-health
                port:
                  number: 8000
          - path: /background/liveness
            pathType: Prefix
            backend:
              service:
                name: kyverno-background-health
                port:
                  number: 8000

Apply and verify:

kubectl apply -f kyverno-health-services.yaml
curl https://kyverno-health.internal.yourdomain.com/admission/liveness
# Expected: 200 OK

Step 3: Add Vigilmon HTTP Monitors

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Configure the following monitors:

| Monitor Name | URL | Priority | |---|---|---| | kyverno admission liveness | https://kyverno-health.yourdomain.com/admission/liveness | Critical | | kyverno admission readiness | https://kyverno-health.yourdomain.com/admission/readiness | Critical | | kyverno background controller | https://kyverno-health.yourdomain.com/background/liveness | High |

Settings for all monitors:

• Check interval: 1 minute
• Timeout: 10 seconds
• Expected status code: 200
• Alert after: 2 consecutive failures

Kyverno's admission health is your most critical alert — treat it the same as a production API being down, because that's effectively what it means for your cluster.

Step 4: Monitor the Webhook Endpoint Directly

Beyond health checks, you can verify the webhook is responding to synthetic requests. This catches TLS certificate issues and service routing failures that /health wouldn't reveal:

# Test the webhook TLS endpoint (will get a 400 — no valid AdmissionRequest)
# but TLS must complete for this to work
curl -k https://kyverno-svc.kyverno.svc.cluster.local:443/validate/fail-bindable-volumes

From outside the cluster, create a minimal webhook probe endpoint. Add this to your Ingress and configure a Vigilmon monitor that asserts on the TLS handshake:

# Monitor settings in Vigilmon:
# URL: https://kyverno.internal.yourdomain.com/health
# Check: Status code is 200
# SSL cert expiry alert: 14 days before expiration

Enable SSL certificate monitoring on the kyverno webhook domain in Vigilmon — certificate expiry on the webhook TLS cert causes x509: certificate has expired or is not yet valid errors across all admission requests, which is an outage equivalent to the webhook being down.

Step 5: Heartbeat Monitor for Background Policy Scans

Kyverno's background controller continuously rescans existing resources against current policies and generates PolicyReport objects. If it stalls, you lose visibility into existing policy violations without any error signal.

Create a CronJob that verifies recent PolicyReports are being generated:

# kyverno-scan-heartbeat.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kyverno-scan-heartbeat
  namespace: kyverno
spec:
  schedule: "*/30 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: kyverno-scan-heartbeat
          containers:
            - name: heartbeat
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: kyverno-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Verify background controller is producing PolicyReports
                  REPORT_COUNT=$(kubectl get policyreports -A --no-headers 2>/dev/null | wc -l)
                  if [ "$REPORT_COUNT" -eq 0 ]; then
                    echo "WARN: No PolicyReports found — background scanner may be stalled"
                    exit 1
                  fi
                  # Verify background controller pod is running
                  RUNNING=$(kubectl get pods -n kyverno \
                    -l app.kubernetes.io/component=background-controller \
                    --field-selector=status.phase=Running \
                    --no-headers | wc -l)
                  if [ "$RUNNING" -eq 0 ]; then
                    echo "FAIL: Background controller has no running pods"
                    exit 1
                  fi
                  wget -qO- "$HEARTBEAT_URL"
                  echo "Kyverno background scan heartbeat sent"

Create the RBAC resources and secret:

# Create ServiceAccount with read access to policyreports and pods
kubectl create serviceaccount kyverno-scan-heartbeat -n kyverno
kubectl create clusterrolebinding kyverno-heartbeat-reader \
  --clusterrole=view \
  --serviceaccount=kyverno:kyverno-scan-heartbeat

# Store heartbeat URL
kubectl create secret generic vigilmon-secrets \
  --from-literal=kyverno-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  -n kyverno

In Vigilmon, create a Heartbeat monitor:

• Name: kyverno background policy scans
• Expected interval: 30 minutes
• Grace period: 10 minutes

Step 6: Policy Violation Alerting via Kyverno Webhooks

Kyverno can send notifications when policies are violated. Wire these to Vigilmon's webhook-to-status mechanism by setting up a Kyverno Policy that triggers an external notification:

# kyverno-vigilmon-notification-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: notify-on-critical-violations
spec:
  rules:
    - name: alert-privileged-containers
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        failureAction: Audit
        message: "Privileged container detected"
        pattern:
          spec:
            containers:
              - =(securityContext):
                  =(privileged): "false"

Pair this with Kyverno's notification integrations (via kyverno-notifier or a custom controller watching PolicyReports) to push violation counts to a Vigilmon webhook. This lets you track policy drift on your Vigilmon status page alongside uptime metrics.

Step 7: Configure Alert Routing

Route alerts by severity:

Critical — Admission controller down (page on-call):

1. Alert Channels → Add Channel → PagerDuty
2. Assign to: kyverno admission liveness, kyverno admission readiness

High — Background controller or scan stalled (notify Slack):

1. Alert Channels → Add Channel → Slack Webhook
2. Paste your #security-alerts webhook URL
3. Assign to: kyverno background controller, kyverno background policy scans heartbeat

This mirrors the operational reality: admission controller downtime is an immediate production incident; background controller issues are serious but don't block deployments.

What You're Now Monitoring

| Component | Monitor Type | Failure Detected | |---|---|---| | Admission controller | HTTP liveness | Webhook pod crash | | Admission controller | HTTP readiness | Pod running but not ready to serve | | Webhook TLS | SSL cert expiry | Certificate expiry blocks admission | | Background controller | HTTP liveness | Policy backfill stalled | | Background scans | Heartbeat | PolicyReports not being generated |

Conclusion

Kyverno sits at the intersection of security and reliability — a failure isn't just an outage, it's either a deployment blocker or a security gap. Vigilmon gives your platform and security teams the external visibility they need to catch Kyverno problems within minutes, from the same perspective as the cluster operations that depend on it.

Start monitoring Kyverno for free at vigilmon.online. Your first monitor is running in under two minutes.