tutorial

Monitoring OPA Gatekeeper: Policy Enforcement Observability in Kubernetes

--- title: "Monitoring OPA Gatekeeper: Policy Enforcement Observability in Kubernetes" description: "How to monitor OPA Gatekeeper constraint violations, au...

title: "Monitoring OPA Gatekeeper: Policy Enforcement Observability in Kubernetes" description: "How to monitor OPA Gatekeeper constraint violations, audit results, webhook latency, and integrate policy enforcement observability with Vigilmon for Kubernetes platform teams." date: 2026-07-01 tags: [kubernetes, opa, gatekeeper, policy, monitoring, vigilmon]

OPA Gatekeeper enforces custom policies on Kubernetes resources via admission webhooks. When Gatekeeper is working correctly, policy violations are blocked at admission time and surfaced via audit. When it is not — webhook timeouts, crashing controller pods, misconfigured constraints — your cluster either enforces policy incorrectly or blocks legitimate workloads entirely. This guide covers how to observe Gatekeeper's behavior in production.

Gatekeeper Architecture

Gatekeeper has three main components:

  • Admission webhook — validates resource creates/updates against active Constraint objects; failurePolicy: Ignore or Fail determines what happens when the webhook is down
  • Audit controller — periodically scans existing resources against constraints and writes violations to Constraint status
  • Gatekeeper controller manager — reconciles ConstraintTemplate and Constraint objects, compiles Rego policies

Failure modes:

  • Webhook latency exceeding the Kubernetes API server timeout (default 10s) triggers failurePolicy
  • Controller manager OOM kills stall constraint compilation
  • Audit controller falling behind means violations go undetected on existing resources
  • Constraint compilation errors leave policies silently unenforced

Prometheus Metrics

Gatekeeper exposes metrics at :8888/metrics on the controller manager pod.

Scrape Configuration

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gatekeeper-controller-manager
  namespace: gatekeeper-system
spec:
  selector:
    matchLabels:
      control-plane: controller-manager
      gatekeeper.sh/system: "yes"
  endpoints:
    - port: metrics
      path: /metrics

Admission Webhook Metrics

# Total admission requests handled
rate(gatekeeper_validations_total[5m])

# Admission denials by constraint
rate(gatekeeper_validations_total{admission_status="deny"}[5m]) by (constraint)

# Webhook request latency (p99)
histogram_quantile(0.99,
  rate(gatekeeper_webhook_request_duration_seconds_bucket[5m])
)

High p99 webhook latency is the leading indicator of timeout-induced admission failures. Alert on it before the Kubernetes API server starts timing out requests.

Constraint Violation Metrics

# Total violations detected by audit
gatekeeper_violations{enforcement_action="deny"}

# Violations per constraint across all namespaces
gatekeeper_violations by (constraint, enforcement_action)

# Audit duration (long audits mean stale violation data)
gatekeeper_audit_duration_seconds

Constraint Compilation

# Constraints that failed to compile (active policy gap)
gatekeeper_constraint_template_ingestion_count{status="error"}

# Time to compile a constraint template
histogram_quantile(0.95,
  rate(gatekeeper_constraint_template_ingestion_duration_seconds_bucket[5m])
)

A constraint_template_ingestion_count with status="error" means a Rego policy failed to compile — that constraint is silently not enforced.

Alerting Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gatekeeper-alerts
  namespace: gatekeeper-system
spec:
  groups:
    - name: gatekeeper
      rules:
        - alert: GatekeeperWebhookHighLatency
          expr: |
            histogram_quantile(0.99,
              rate(gatekeeper_webhook_request_duration_seconds_bucket[5m])
            ) > 5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Gatekeeper webhook p99 latency {{ $value }}s — risk of API server timeouts"

        - alert: GatekeeperConstraintCompileError
          expr: gatekeeper_constraint_template_ingestion_count{status="error"} > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "Gatekeeper constraint template failed to compile — policy not enforced"

        - alert: GatekeeperHighViolationCount
          expr: sum(gatekeeper_violations{enforcement_action="deny"}) > 100
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "{{ $value }} Gatekeeper violations detected — audit results need review"

        - alert: GatekeeperControllerDown
          expr: up{job="gatekeeper-controller-manager"} == 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Gatekeeper controller manager is down — policies may not be enforced"

        - alert: GatekeeperAuditStale
          expr: time() - gatekeeper_audit_last_run_time > 3600
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Gatekeeper audit has not run in >1 hour — violations may be undetected"

Inspecting Violations with kubectl

# List all constraints and their violation counts
kubectl get constraints -A

# Detailed violations for a specific constraint
kubectl describe constraint require-resource-limits

# All violations across all constraint types
kubectl get constraints -A -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
for item in data['items']:
    violations = item.get('status', {}).get('violations', [])
    if violations:
        print(f\"\n=== {item['kind']}/{item['metadata']['name']} ===\")
        for v in violations[:5]:
            print(f\"  {v.get('namespace','cluster')}/{v.get('name')}: {v.get('message')}\")
"

# Check constraint template compilation status
kubectl get constrainttemplate -o wide

# Gatekeeper controller logs
kubectl logs -n gatekeeper-system -l control-plane=controller-manager --tail=100

# Webhook configuration
kubectl get validatingwebhookconfigurations gatekeeper-validating-webhook-configuration -o yaml | \
  grep -E "failurePolicy|timeoutSeconds"

The failurePolicy field is critical: Ignore means Gatekeeper downtime lets unenforced resources through; Fail means Gatekeeper downtime blocks all resource creation. Know which mode you are in and alert accordingly.

Audit Controller Observability

Audit runs on a configurable interval (default: 60 seconds) and writes violation counts to each Constraint's .status.violations field. Check audit health:

# Last audit timestamp on a constraint
kubectl get constraint require-labels -o jsonpath='{.status.auditTimestamp}'

# Total violations found in last audit
kubectl get constraint require-labels -o jsonpath='{.status.totalViolations}'

# Audit log from controller
kubectl logs -n gatekeeper-system deploy/gatekeeper-audit --tail=100 | grep -E "audit|violation"

Testing Policy Enforcement

After deploying a new constraint, verify it enforces correctly before trusting it in production:

# Attempt to create a resource that should be denied
kubectl run test-pod --image=nginx --namespace=production --dry-run=server

# If the constraint is working, you get:
# Error from server ([constraint-name] denied the request: ...)

# If it passes silently, check constraint compilation status:
kubectl describe constrainttemplate <template-name>

Vigilmon Integration

Monitor Webhook Availability

The most critical Gatekeeper component is the admission webhook. Add a Vigilmon TCP monitor:

  1. Go to Monitors → Add Monitor → TCP/Port
  2. Set host to the gatekeeper-webhook-service ClusterIP (or use a LoadBalancer with external access)
  3. Set port to 8443
  4. Set check interval to 30 seconds

If the webhook pod crashes, this monitor fires within 30 seconds — before Kubernetes starts timing out admission requests.

Heartbeat for Audit Pipeline

If you run a custom audit export job (e.g., shipping violations to a SIEM or ticketing system), register a Vigilmon heartbeat:

#!/bin/bash
# audit-export.sh — runs every 5 minutes via CronJob

# Export Gatekeeper violations to SIEM
kubectl get constraints -A -o json | ./ship-violations-to-siem.py

# Signal health
curl -fsS "https://vigilmon.online/heartbeat/YOUR-HEARTBEAT-ID"

Configure the heartbeat period to 10 minutes — if the export job stops running, violations are no longer being tracked and compliance reporting silently goes dark.

Policy Compliance Status Page

Create a Vigilmon status page with a component per enforcement tier:

  • Admission Enforcement — green when webhook is reachable and latency is normal
  • Audit Coverage — green when heartbeat received from audit export within period
  • Policy Compilation — green when no ConstraintTemplate errors detected

This gives your security and platform teams a shared view of policy enforcement health without requiring direct cluster access.

HTTPS Monitor for Policy Documentation

If you publish a human-readable policy catalog (e.g., a docs site or Confluence page describing what each constraint enforces), add a Vigilmon HTTPS monitor for that endpoint — broken documentation pages are a frequent operational gap during incident response.

Summary

| Signal | Metric / Resource | Alert Threshold | |---|---|---| | Webhook latency | gatekeeper_webhook_request_duration_seconds | p99 > 5s | | Constraint compile failures | gatekeeper_constraint_template_ingestion_count{status="error"} | > 0 immediately | | Violation count | gatekeeper_violations | Business-defined threshold | | Audit freshness | gatekeeper_audit_last_run_time | > 1 hour since last run | | Controller availability | up{job="gatekeeper-controller-manager"} | == 0 for 2m |

The highest-severity gap is a Fail mode webhook with the controller down — it blocks your entire cluster. Pair the GatekeeperControllerDown alert with a Vigilmon TCP probe on the webhook port for independent, out-of-band detection that does not depend on the same Prometheus stack running inside the cluster.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles