tutorial

How to Monitor Konstraint + OPA Gatekeeper with Vigilmon

Konstraint is a CLI tool that simplifies authoring and managing OPA Gatekeeper constraints and constraint templates. It generates `ConstraintTemplate` and `C...

Konstraint is a CLI tool that simplifies authoring and managing OPA Gatekeeper constraints and constraint templates. It generates ConstraintTemplate and Constraint CRDs from annotated Rego files, keeping your policy definitions DRY and version-controlled. Combined with OPA Gatekeeper — which enforces those constraints as a Kubernetes admission webhook — Konstraint is part of a policy pipeline that sits directly in the path of all resource creation.

In this tutorial you'll set up monitoring for OPA Gatekeeper and the services that support your Konstraint-based policy workflow using Vigilmon — free tier, no credit card required.

Why your Gatekeeper + Konstraint pipeline needs monitoring

Konstraint itself is a stateless CLI. What needs monitoring is the runtime infrastructure:

  • Gatekeeper audit controller down — the audit job that continuously evaluates existing resources against constraints stops running; policy violations accumulate silently without anyone knowing
  • Gatekeeper admission webhook unavailable — new resources bypass all policy checks (fail open) or all deployments are blocked (fail closed), depending on your failurePolicy
  • Constraint template sync broken — your CI pipeline that runs konstraint doc and applies templates via kubectl apply fails silently; the cluster runs outdated policies
  • OPA metrics endpoint unreachable — you lose visibility into constraint evaluation rates, violation counts, and audit duration
  • Webhook certificate expired — Gatekeeper manages its own TLS certificate; after expiry, the API server refuses to send admission requests to the webhook

What you'll need

  • A Kubernetes cluster with OPA Gatekeeper installed
  • Konstraint-generated constraint templates applied to the cluster
  • A free Vigilmon account

Step 1: Verify Gatekeeper is installed and healthy

Gatekeeper runs as several pods in the gatekeeper-system namespace:

kubectl get pods -n gatekeeper-system
# NAME                                             READY   STATUS    RESTARTS   AGE
# gatekeeper-audit-xxx-yyy                         1/1     Running   0          5d
# gatekeeper-controller-manager-xxx-yyy            1/1     Running   0          5d
# gatekeeper-controller-manager-xxx-zzz            1/1     Running   0          5d

Check the existing services:

kubectl get svc -n gatekeeper-system
# NAME                                 TYPE        CLUSTER-IP     PORT(S)    AGE
# gatekeeper-webhook-service           ClusterIP   10.96.10.100   443/TCP    5d
# gatekeeper-metrics                   ClusterIP   10.96.10.101   8888/TCP   5d

Gatekeeper exposes metrics on port 8888 by default. Expose this externally for Vigilmon:

# gatekeeper-monitor-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: gatekeeper-health-monitor
  namespace: gatekeeper-system
  labels:
    app: gatekeeper-monitoring
spec:
  type: LoadBalancer
  selector:
    control-plane: controller-manager
    gatekeeper.sh/system: "yes"
  ports:
    - name: metrics
      port: 8888
      targetPort: 8888
      protocol: TCP

kubectl apply -f gatekeeper-monitor-svc.yaml
kubectl get svc gatekeeper-health-monitor -n gatekeeper-system
# NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
# gatekeeper-health-monitor  LoadBalancer   10.96.10.110   203.0.113.55    8888:31800/TCP   2m

Verify:

curl http://203.0.113.55:8888/metrics
# # HELP gatekeeper_violations Total number of audited violations
# gatekeeper_violations{...} 0

Also expose the audit controller metrics:

# gatekeeper-audit-monitor-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: gatekeeper-audit-monitor
  namespace: gatekeeper-system
spec:
  type: LoadBalancer
  selector:
    control-plane: audit-controller
    gatekeeper.sh/system: "yes"
  ports:
    - name: metrics
      port: 8888
      targetPort: 8888

kubectl apply -f gatekeeper-audit-monitor-svc.yaml
kubectl get svc gatekeeper-audit-monitor -n gatekeeper-system

Step 2: Add a health proxy for comprehensive checks

Gatekeeper doesn't expose a dedicated /health endpoint in all versions. Add a thin proxy that checks both the controller and audit components:

# gatekeeper-health-proxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gatekeeper-health-proxy
  namespace: gatekeeper-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gatekeeper-health-proxy
  template:
    metadata:
      labels:
        app: gatekeeper-health-proxy
    spec:
      containers:
        - name: health-proxy
          image: node:20-alpine
          command: ["/bin/sh", "-c"]
          args:
            - |
              cat > /app.cjs << 'JSEOF'
              const http = require('http');
              const { execSync } = require('child_process');

              function checkMetrics(host, port, cb) {
                const req = http.get({ host, port, path: '/metrics', timeout: 3000 }, (res) => {
                  cb(null, res.statusCode);
                });
                req.on('error', (e) => cb(e));
                req.on('timeout', () => { req.destroy(); cb(new Error('timeout')); });
              }

              http.createServer((req, res) => {
                if (req.url !== '/health') { res.writeHead(404); res.end(); return; }
                checkMetrics('gatekeeper-metrics.gatekeeper-system.svc', 8888, (err, code) => {
                  const ok = !err && code === 200;
                  res.writeHead(ok ? 200 : 503, { 'Content-Type': 'application/json' });
                  res.end(JSON.stringify({
                    status: ok ? 'ok' : 'error',
                    gatekeeper_metrics: ok ? 'reachable' : (err ? err.message : code)
                  }));
                });
              }).listen(9090, () => console.log('Health proxy on :9090'));
              JSEOF
              node /app.cjs
          ports:
            - containerPort: 9090
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
---
apiVersion: v1
kind: Service
metadata:
  name: gatekeeper-health-proxy
  namespace: gatekeeper-system
spec:
  type: LoadBalancer
  selector:
    app: gatekeeper-health-proxy
  ports:
    - port: 9090
      targetPort: 9090

kubectl apply -f gatekeeper-health-proxy.yaml
kubectl get svc gatekeeper-health-proxy -n gatekeeper-system
# NAME                      TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)         AGE
# gatekeeper-health-proxy   LoadBalancer   10.96.10.120  203.0.113.56    9090:31900/TCP  1m

curl http://203.0.113.56:9090/health
# {"status":"ok","gatekeeper_metrics":"reachable"}

Step 3: Set up HTTP monitoring in Vigilmon

Add your Gatekeeper endpoints to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add monitors:

| Monitor name | URL | Expected status | |---|---|---| | Gatekeeper Controller Metrics | http://203.0.113.55:8888/metrics | 200 | | Gatekeeper Audit Metrics | http://AUDIT_EXTERNAL_IP:8888/metrics | 200 | | Gatekeeper Health Proxy | http://203.0.113.56:9090/health | 200 | | Konstraint CI Pipeline Health | https://ci.yourdomain.com/health | 200 |

  1. Set check interval to 1 minute
  2. Under Expected response, set status code 200; for the health proxy, add body contains "status":"ok"
  3. Save each monitor

The audit metrics monitor is especially important: if the audit controller pod goes down, you lose continuous compliance scanning. Existing violations won't be detected. New violations from already-running workloads won't be reported. Vigilmon will catch this immediately.

Step 4: Monitor the admission webhook TCP port

The Gatekeeper admission webhook is served over HTTPS on port 443. The API server calls it for every resource operation. Monitor at the TCP level:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter:
    • Host: your Gatekeeper webhook service's external hostname (or the LoadBalancer IP if you expose it)
    • Port: 443
  4. Save the monitor

If the webhook service becomes unreachable at the TCP level, the API server will start timing out on admission calls — manifesting as very slow kubectl apply operations that eventually fail or succeed depending on your failurePolicy.

Step 5: Configure alert channels

Gatekeeper outages have two distinct blast radii:

  • Controller down: new resources bypass policy checks (fail open) or all creates/updates are blocked (fail closed)
  • Audit controller down: existing policy violations go undetected, compliance posture silently drifts

Both are critical but require different responses. Configure alerts for both:

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Add your security and platform team emails
  3. Assign to all Gatekeeper monitors

Webhook alerts

  1. Go to Alert Channels → Add Channel → Webhook
  2. Use your PagerDuty or Slack webhook URL
  3. Vigilmon payload:
{
  "monitor_name": "Gatekeeper Audit Metrics",
  "status": "down",
  "url": "http://AUDIT_IP:8888/metrics",
  "started_at": "2024-01-15T09:00:00Z",
  "duration_seconds": 300
}

Map this to different alert severity levels: controller downtime → P1 (deployments blocked or unprotected), audit downtime → P2 (compliance scanning paused).

Step 6: Monitor your Konstraint CI pipeline

Konstraint runs in CI to generate and apply constraint templates. If this pipeline breaks, your policy changes stop reaching the cluster. Add a health endpoint to your policy CI pipeline:

#!/usr/bin/env bash
# konstraint-pipeline-health.sh
# Outputs a JSON health check based on when konstraint last ran successfully

LAST_RUN_FILE="/var/state/konstraint-last-success"

if [ ! -f "$LAST_RUN_FILE" ]; then
  echo '{"status":"unknown","reason":"no run recorded"}'
  exit 1
fi

LAST_RUN=$(cat "$LAST_RUN_FILE")
NOW=$(date +%s)
AGE_MINUTES=$(( (NOW - LAST_RUN) / 60 ))

if [ "$AGE_MINUTES" -gt 120 ]; then
  echo "{\"status\":\"stale\",\"last_run_minutes_ago\":$AGE_MINUTES}"
  exit 1
fi

echo "{\"status\":\"ok\",\"last_run_minutes_ago\":$AGE_MINUTES}"

Expose this via a simple HTTP service in your CI infrastructure and add it as a Vigilmon monitor. If konstraint hasn't run successfully in two hours, you have a policy drift risk.

Step 7: Correlate Vigilmon alerts with Gatekeeper diagnostics

When a Gatekeeper alert fires:

# 1. Check all Gatekeeper pods
kubectl get pods -n gatekeeper-system

# 2. Check recent events
kubectl get events -n gatekeeper-system --sort-by='.lastTimestamp' | tail -20

# 3. Verify constraint templates are loaded
kubectl get constrainttemplates

# 4. Check constraints and their enforcement status
kubectl get constraints --all-namespaces 2>/dev/null || \
  kubectl get $(kubectl get crds -o name | grep constraints.gatekeeper.sh | head -5 | \
    xargs -I{} kubectl get {} --all-namespaces --no-headers -o name | head -20 | \
    tr '\n' ',' | sed 's/,$//') 2>/dev/null

# 5. Check audit log for violation counts
kubectl logs -n gatekeeper-system -l control-plane=audit-controller --tail=100 | \
  grep -E 'violations|audit|error'

# 6. Check webhook config is still pointing to a live service
kubectl describe validatingwebhookconfiguration gatekeeper-validating-webhook-configuration | \
  grep -A5 'Service:'

# 7. Run konstraint to verify policy files are valid
konstraint create --output ./generated ./policy/
konstraint doc --output POLICIES.md ./policy/

Step 8: Create a status page for policy governance

  1. Go to Status Pages → New Status Page
  2. Name it: "OPA Gatekeeper Policy Governance"
  3. Add monitors: Controller Metrics, Audit Metrics, Health Proxy, Konstraint CI
  4. Share with security, compliance, and platform engineering teams

Security and compliance teams can use this page to verify that policy enforcement is active — especially important during security audits or compliance reviews.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on controller metrics | Gatekeeper controller pod failures | | HTTP monitor on audit metrics | Audit controller down (silent compliance drift) | | HTTP monitor on health proxy | Composite availability check | | HTTP monitor on Konstraint CI | Policy template update pipeline failures | | TCP monitor on webhook port 443 | Certificate / LoadBalancer failures | | Multi-region consensus | Fewer false positives from transient blips | | Separate alert severities | P1 for admission control, P2 for audit |

The combination of Konstraint and OPA Gatekeeper gives you policy-as-code governance across your entire cluster. Vigilmon closes the loop by ensuring the governance infrastructure itself is always healthy — because the worst time to find out your audit controller has been down for a week is during a SOC2 audit.

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles