tutorial

Monitoring OPA Gatekeeper with Vigilmon

OPA Gatekeeper enforces policy-as-code across your Kubernetes cluster — here's how to monitor admission webhook health, policy violation counts, audit job completeness, and constraint sync with Vigilmon.

OPA Gatekeeper is the standard open-source solution for Policy-as-Code enforcement in Kubernetes. As a validating and mutating admission controller, it intercepts every resource creation and modification request to the Kubernetes API server and evaluates it against policies written in Rego — blocking privileged containers, enforcing image registry allowlists, requiring resource limits, and dozens of other compliance controls. Gatekeeper also runs periodic audit jobs that scan existing cluster resources for policy violations, surfacing drift that was approved before a policy was introduced.

When Gatekeeper's admission webhook goes down, every Kubernetes admission request — kubectl apply, Helm deployments, autoscaler pod creation — is either blocked or bypassed depending on your failurePolicy setting. Either outcome is bad. Vigilmon gives you continuous health monitoring for Gatekeeper's webhook, audit pipeline, and policy sync so you catch Gatekeeper problems before they become cluster-wide incidents.

What You'll Set Up

  • Gatekeeper webhook admission controller availability (deployment readiness)
  • Webhook response latency monitoring (P99 approaching Kubernetes timeout)
  • Audit job health and last-run timestamp
  • Gatekeeper Prometheus metrics endpoint (/metrics on port 8888)
  • ConstraintTemplate sync (policy compilation health)
  • Total cluster violations trend monitoring
  • Mutation webhook health (if using mutation policies)

Prerequisites

  • OPA Gatekeeper installed in your Kubernetes cluster (version 3.x)
  • kubectl access to the gatekeeper-system namespace
  • A free Vigilmon account

Why Monitoring Gatekeeper Is Critical

Gatekeeper sits in the critical path of the Kubernetes control plane. A misconfigured or crashed admission webhook can:

  • Block all resource creation in the cluster if failurePolicy: Fail is set
  • Allow all policy-violating resources to be created if failurePolicy: Ignore is set and the webhook is down
  • Silently miss audit violations if the audit deployment crashes

Neither failure mode is safe. Fail causes an operational outage; Ignore creates a security window. Monitoring webhook availability, response latency, and audit completeness is essential for both reliability and compliance.

Step 1: Monitor Webhook Admission Controller Availability

The gatekeeper-controller-manager deployment runs the admission webhook. Check that the expected number of replicas are ready:

Set up a Vigilmon cron heartbeat that checks the deployment readiness:

#!/bin/bash
DESIRED=$(kubectl get deployment gatekeeper-controller-manager \
  -n gatekeeper-system -o jsonpath='{.spec.replicas}')
READY=$(kubectl get deployment gatekeeper-controller-manager \
  -n gatekeeper-system -o jsonpath='{.status.readyReplicas}')

if [ "$READY" = "$DESIRED" ] && [ -n "$READY" ]; then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Schedule every 1 minute. Set the Vigilmon heartbeat interval to 3 minutes — if the check fails for 3 consecutive minutes, you have a real availability problem.

Name this heartbeat "Gatekeeper Webhook Deployment" and set it as your highest-priority Gatekeeper alert. A failure here means admission control is degraded.

Alternative: Monitor the Gatekeeper health endpoint directly.

If Gatekeeper's health endpoint is exposed inside the cluster, you can probe it from within the cluster:

kubectl run gatekeeper-health-check \
  --image=curlimages/curl \
  --restart=Never \
  --rm \
  -it \
  -- curl -sk https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/healthz

For persistent monitoring, expose the health endpoint via a Kubernetes Service and monitor it with Vigilmon's HTTP monitor (if accessible outside the cluster) or via a pod-based cron heartbeat.

Step 2: Monitor Webhook Response Latency

Kubernetes imposes a maximum webhook timeout (default 10 seconds). If your Gatekeeper admission webhook takes longer than this to respond, Kubernetes applies the failurePolicy. Webhook latency spikes under load or when Rego policy complexity grows. Alert before P99 latency approaches the timeout.

Gatekeeper exposes webhook duration metrics at the /metrics endpoint on port 8888:

gatekeeper_webhook_duration_seconds{quantile="0.99"}

Set up a Prometheus scrape rule and alert when P99 exceeds 7 seconds (70% of the 10-second timeout). For a Vigilmon cron heartbeat without a Prometheus stack:

#!/bin/bash
# Port-forward the metrics endpoint (run this from a pod or use kubectl exec)
METRICS=$(kubectl exec -n gatekeeper-system \
  deploy/gatekeeper-controller-manager \
  -- wget -qO- http://localhost:8888/metrics 2>/dev/null)

P99=$(echo "$METRICS" | \
  grep 'gatekeeper_webhook_duration_seconds{.*quantile="0.99"' | \
  tail -1 | awk '{print $NF}')

# Alert if P99 > 7 seconds (approaching 10s Kubernetes timeout)
if [ -n "$P99" ] && (( $(echo "$P99 < 7.0" | bc -l) )); then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Schedule every 5 minutes.

Step 3: Monitor the Gatekeeper Metrics Endpoint

Gatekeeper exposes Prometheus metrics on port 8888. This endpoint is the source of truth for violations, latency, and sync health. If it goes dark, all visibility into Gatekeeper's behavior disappears.

Add a Vigilmon cron heartbeat that checks the metrics endpoint is reachable and returning data:

#!/bin/bash
METRICS=$(kubectl exec -n gatekeeper-system \
  deploy/gatekeeper-controller-manager \
  -- wget -qO- http://localhost:8888/metrics 2>/dev/null | wc -l)

# Expect at least 50 metric lines from a healthy Gatekeeper instance
if [ "$METRICS" -gt 50 ]; then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Schedule every 2 minutes.

Step 4: Monitor Audit Job Health

The gatekeeper-audit deployment periodically scans all existing cluster resources against your Constraints and reports violations. If the audit deployment crashes or its job gets stuck, you lose continuous compliance visibility — the violation counts on your Constraints freeze at the last successful audit value, making it look like no new violations exist.

Check that the audit deployment is healthy and that a recent audit has completed:

#!/bin/bash
# Check audit deployment readiness
AUDIT_READY=$(kubectl get deployment gatekeeper-audit \
  -n gatekeeper-system -o jsonpath='{.status.readyReplicas}' 2>/dev/null)

if [ -z "$AUDIT_READY" ] || [ "$AUDIT_READY" -lt 1 ]; then
  exit 0  # Don't send heartbeat
fi

# Check the last audit timestamp from Constraint status
# The audit updates .status.byPod[].auditTimestamp on each Constraint
LAST_AUDIT=$(kubectl get constraint -A -o json 2>/dev/null | \
  python3 -c "
import sys, json
from datetime import datetime, timezone
data = json.load(sys.stdin)
timestamps = []
for item in data.get('items', []):
    for pod in item.get('status', {}).get('byPod', []):
        ts = pod.get('auditTimestamp', '')
        if ts:
            timestamps.append(ts)
if not timestamps:
    sys.exit(1)
latest = max(timestamps)
print(latest)
" 2>/dev/null)

if [ -z "$LAST_AUDIT" ]; then
  exit 0
fi

# Alert if last audit is older than 15 minutes
LAST_AUDIT_EPOCH=$(date -d "$LAST_AUDIT" +%s 2>/dev/null || date -j -f "%Y-%m-%dT%H:%M:%SZ" "$LAST_AUDIT" +%s 2>/dev/null)
NOW_EPOCH=$(date +%s)
AGE=$(( NOW_EPOCH - LAST_AUDIT_EPOCH ))

if [ "$AGE" -lt 900 ]; then  # 15 minutes
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Schedule every 5 minutes. Set the Vigilmon heartbeat interval to 20 minutes. If the audit heartbeat has not been received in 20 minutes, the audit pipeline is stalled and you should investigate.

Step 5: Track Policy Violation Counts

Gatekeeper reports violations through two channels: admission-time (new violations blocked or audited at the API server) and audit-time (existing violations in the cluster). Monitor both:

Admission-time violations per Constraint:

rate(gatekeeper_violations_total[5m])

Alert when the rate spikes, indicating a new workload or deployment is violating policy at scale.

Audit-time total violations:

#!/bin/bash
# Count total audit violations across all Constraints
TOTAL_VIOLATIONS=$(kubectl get constraint -A -o json 2>/dev/null | \
  python3 -c "
import sys, json
data = json.load(sys.stdin)
total = sum(
  item.get('status', {}).get('totalViolations', 0)
  for item in data.get('items', [])
)
print(total)
" 2>/dev/null || echo 0)

# Log the violation count to a file for trend tracking
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) violations=$TOTAL_VIOLATIONS" >> /var/log/gatekeeper-violations.log

# Send heartbeat regardless (this tracks audit health, not violation threshold)
curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null

Use a separate Vigilmon heartbeat for violation trend tracking. The key signal is not the absolute count but the direction: rising violations after a stable period indicates policy drift or a new workload pattern.

Step 6: ConstraintTemplate Sync Health

ConstraintTemplates contain the Rego policies that Gatekeeper compiles into WebAssembly and loads into the OPA engine. A syntax error in Rego or a bad Gatekeeper upgrade can leave templates in a Failed state — the policy simply stops enforcing without any obvious signal to end users.

Monitor template compilation health:

#!/bin/bash
# Check for ConstraintTemplates in Failed state
FAILED=$(kubectl get constrainttemplates -o json 2>/dev/null | \
  python3 -c "
import sys, json
data = json.load(sys.stdin)
failed = [
  item['metadata']['name']
  for item in data.get('items', [])
  if any(
    c.get('type') == 'Ready' and c.get('status') == 'False'
    for c in item.get('status', {}).get('conditions', [])
  )
]
print(len(failed))
" 2>/dev/null || echo 0)

if [ "$FAILED" -eq 0 ]; then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Schedule every 5 minutes. A ConstraintTemplate failure is a silent policy enforcement gap — alert immediately and investigate the template's .status.conditions for the Rego compilation error.

Step 7: Mutation Webhook Health (If Using Mutations)

If you use Gatekeeper's mutation capabilities (injecting labels, modifying resource fields), the MutatingWebhookConfiguration is equally critical. A failed mutation webhook can allow resources to be created without required labels or annotations.

#!/bin/bash
# Check that the mutating webhook configuration exists and is active
WEBHOOK_CONFIGURED=$(kubectl get mutatingwebhookconfiguration \
  gatekeeper-mutating-webhook-configuration \
  -o jsonpath='{.metadata.name}' 2>/dev/null)

if [ "$WEBHOOK_CONFIGURED" = "gatekeeper-mutating-webhook-configuration" ]; then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Schedule every 2 minutes. Combine this check with the deployment readiness check (Step 1) for a comprehensive webhook health view.

Alerting Configuration

| Monitor | Alert condition | Recommended channel | |---------|----------------|---------------------| | Webhook deployment heartbeat | Missed heartbeat | PagerDuty | | Audit job heartbeat | Missed (>20 min) | Slack | | Metrics endpoint heartbeat | Missed heartbeat (2×) | Slack | | ConstraintTemplate sync heartbeat | Missed heartbeat | PagerDuty | | Webhook latency heartbeat | Missed heartbeat | Slack | | Mutation webhook heartbeat | Missed heartbeat | Slack |

Webhook availability and ConstraintTemplate failures should go to PagerDuty or equivalent on-call. Audit and metrics gaps can go to Slack for async follow-up — they don't immediately break cluster operations but create compliance blind spots.

Interpreting Gatekeeper Alerts

Webhook deployment not ready: The gatekeeper-controller-manager pod(s) have crashed or are pending. Check kubectl describe pod -n gatekeeper-system -l control-plane=controller-manager for OOMKill events or image pull errors. Gatekeeper needs enough memory to hold all compiled Rego policies in WASM.

Webhook latency approaching timeout: Usually caused by complex Rego evaluation (nested iterations over large external data), a slow OPA external data provider, or excessive policy count. Profile with gatekeeper_webhook_duration_seconds broken down by constraint kind.

Audit job stalled: The gatekeeper-audit pod may be restarting or stuck on a large cluster. Check kubectl logs -n gatekeeper-system -l control-plane=audit-controller. The --audit-chunk-size flag controls how many resources are audited per batch — reduce it for large clusters.

ConstraintTemplate in Failed state: Run kubectl describe constrainttemplate <name> and look at .status.conditions. The Rego policy has a compilation error. Fix the syntax and re-apply the template.

Total violations spiking: A new deployment or namespace is violating policy at scale. Check kubectl get constraint -A -o yaml | grep -A5 violations to identify which constraint is flagging resources, then trace the workload from the admission log or audit violation list.

Conclusion

OPA Gatekeeper is your Kubernetes cluster's policy enforcement layer, and it is uniquely dangerous when it fails silently — either blocking everything or enforcing nothing, depending on your failurePolicy. Vigilmon heartbeat monitors for webhook availability, audit job completeness, ConstraintTemplate sync, and latency give you the early warning you need to catch Gatekeeper problems before they affect cluster operations or create compliance gaps.

Get started at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles