Monitoring Policy Controller with Vigilmon: Kubernetes Admission Control, Supply Chain Policy Health, and Alert Coverage

Policy Controller is Sigstore's Kubernetes admission controller that enforces supply chain security policies — verifying that container images are signed, attested, and comply with your ClusterImagePolicy rules before any pod is admitted to your cluster. It's the enforcement layer that makes container image signing mean something: without it, unsigned images can run regardless of your signing workflow. When Policy Controller's webhook crashes, Kubernetes falls back to its configured failure policy — either blocking all deployments (Fail) or admitting everything silently (Ignore). Either outcome is a supply chain security incident. Vigilmon gives you external monitoring of Policy Controller's webhook health, the policy validation endpoint, Fulcio/Rekor connectivity, and SSL certificates.

What You'll Build

• A monitor on Policy Controller's webhook readiness endpoint
• An HTTP monitor for the webhook server's liveness probe
• Monitors for Fulcio and Rekor connectivity (the Sigstore services Policy Controller depends on)
• SSL certificate monitors for the webhook's TLS certificate
• An alerting setup calibrated to the criticality of your failurePolicy setting

Prerequisites

• Policy Controller deployed in your Kubernetes cluster (via Helm chart or manifest)
• Policy Controller's webhook endpoint accessible from outside the cluster (via Ingress or LoadBalancer), or port-forwarded for testing
• A free account at vigilmon.online

Step 1: Verify Policy Controller's Webhook Health

Policy Controller runs as a validating admission webhook. Kubernetes calls this webhook for every pod creation and update in namespaces where Policy Controller is active. The webhook server exposes readiness and liveness endpoints:

# Port-forward the Policy Controller webhook service
kubectl port-forward svc/policy-controller-webhook -n cosign-system 8443:443

# Check the health endpoint
curl -k https://localhost:8443/healthz

A healthy Policy Controller returns HTTP 200:

{"status": "ok"}

To expose this for external monitoring, create an Ingress or use your cluster's monitoring reverse proxy to route traffic to the policy-controller webhook service:

# Check the webhook registration
kubectl get validatingwebhookconfigurations | grep policy-controller

# Describe the webhook to see the caBundle and service configuration
kubectl describe validatingwebhookconfiguration policy-controller-validating-webhook-configuration

Step 2: Create a Vigilmon Monitor for the Webhook Health

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://policy-controller-health.example.com/healthz (your exposed health endpoint).
3. Check interval: 30 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: Policy Controller webhook health.
8. Click Save.

This monitor catches:

• Policy Controller process crashes or pod evictions
• OOMKilled events when the webhook server runs out of memory during policy evaluation
• Failed Helm upgrades that leave the webhook in a broken state
• Kubernetes API server connectivity issues that affect the webhook server startup

Alert sensitivity: Set alerts after 1 consecutive failure. Policy Controller is on the critical path for every pod deployment in enforced namespaces. A single failure of this webhook (combined with failurePolicy: Fail) blocks the entire deployment pipeline.

failurePolicy is your blast radius: failurePolicy: Fail means Policy Controller downtime = zero deployments cluster-wide. failurePolicy: Ignore means downtime = zero enforcement. Neither is acceptable without monitoring. Vigilmon gives you the early warning to respond before either scenario becomes prolonged.

Step 3: Monitor the Webhook TLS Certificate

Policy Controller's webhook server uses a TLS certificate that Kubernetes's API server validates before sending admission requests. If this certificate expires, the Kubernetes API server rejects the webhook call — and depending on failurePolicy, either all deployments fail or policy enforcement silently stops:

# Check the webhook certificate expiry
kubectl get secret policy-controller-webhook-server-cert -n cosign-system \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: policy-controller-health.example.com (or any domain where you've exposed the webhook health endpoint via HTTPS).
3. Alert when expiry is within: 14 days.
4. Alert again: 7 days, 3 days, 1 day.
5. Click Save.

Cert-manager auto-rotation: If you use cert-manager to issue the webhook's TLS certificate, the rotation is automatic but cert-manager itself can fail. Monitor your cert-manager health separately to catch cert-manager issues before they cause webhook certificate expiry.

Step 4: Monitor Fulcio Connectivity

Policy Controller verifies image signatures by checking certificates issued by Fulcio (Sigstore's certificate authority). If Policy Controller can't reach Fulcio, signature verification for keyless signatures fails:

# Check Fulcio's health (public instance)
curl https://fulcio.sigstore.dev/api/v2/configuration

# Or your self-hosted Fulcio instance
curl https://fulcio.example.com/api/v2/configuration

A healthy Fulcio returns its configuration JSON:

{
  "oidcIssuers": {...},
  "metaIssuers": {...}
}

1. Add Monitor → HTTP.
2. URL: https://fulcio.sigstore.dev/api/v2/configuration (or your self-hosted Fulcio URL).
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: oidcIssuers.
7. Label: Fulcio CA (Policy Controller dependency).
8. Click Save.

Step 5: Monitor Rekor Connectivity

Policy Controller also connects to Rekor to verify transparency log entries when enforcing policies that require log inclusion. Without Rekor connectivity, policies with includeSpec.rekor requirements fail verification:

# Check Rekor's log API (public instance)
curl https://rekor.sigstore.dev/api/v1/log

# Or your self-hosted Rekor instance
curl https://rekor.example.com/api/v1/log

1. Add Monitor → HTTP.
2. URL: https://rekor.sigstore.dev/api/v1/log.
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: treeSize.
7. Label: Rekor transparency log (Policy Controller dependency).
8. Click Save.

Step 6: Monitor ClusterImagePolicy Enforcement (Canary Check)

The most reliable way to verify that Policy Controller is actively enforcing policies is a canary check — attempting to deploy an unsigned image in a test namespace and verifying that it's rejected. This can be wrapped in a cron-based monitoring endpoint:

# A simple test: try to create a pod with an unsigned image in an enforced namespace
# This should be rejected if Policy Controller is working
kubectl run policy-test --image=busybox:latest \
  --restart=Never \
  --namespace=policy-enforced-ns \
  --dry-run=server 2>&1

# Expected output when Policy Controller is healthy:
# Error from server: admission webhook "policy.sigstore.dev" denied the request:
# validation failed: ...

Wrap this in a small health check service that returns 200 when the rejection works correctly:

#!/bin/bash
result=$(kubectl run policy-test-$(date +%s) \
  --image=alpine:latest \
  --restart=Never \
  --namespace=policy-enforced-ns \
  --dry-run=server 2>&1)

if echo "$result" | grep -q "denied the request"; then
  echo '{"status": "enforcing"}'
  exit 0
else
  echo '{"status": "not-enforcing"}'
  exit 1
fi

1. Add Monitor → HTTP.
2. URL: https://policy-canary.example.com/enforcement-check.
3. Check interval: 5 minutes.
4. Expected status: 200.
5. Keyword: enforcing.
6. Label: Policy Controller enforcement canary.
7. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Webhook health | Non-200 or ok missing | Controller down; check kubectl get pods -n cosign-system; assess failurePolicy impact | | Webhook SSL cert | < 14 days to expiry | Renew webhook cert; Kubernetes API server will reject webhook calls on expiry | | Fulcio connectivity | Non-200 | Keyless signature verification fails; new keyless-signed image deployments rejected | | Rekor connectivity | Non-200 | Transparency log verification fails; policies requiring log inclusion fail | | Enforcement canary | Non-200 or not-enforcing | Policy enforcement broken; unsigned images may be admitted silently |

Alert escalation: The enforcement canary should alert your security team, not just your ops team — a silent failure of enforcement is a supply chain security incident.

Understanding Policy Controller's ClusterImagePolicy

Policy Controller enforces policies defined as ClusterImagePolicy custom resources. Understanding what a healthy policy configuration looks like helps you interpret monitoring results:

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-signed-images
spec:
  images:
    - glob: "registry.example.com/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://accounts.google.com
            subjectRegExp: ".*@example.com"

When this policy is active:

• Images matching registry.example.com/** must have a keyless Sigstore signature
• The signature must be verifiable against Fulcio's CA
• The OIDC identity must match the specified issuer and subject pattern

Check policy status:

kubectl get clusterimagepolicies
kubectl describe clusterimagepolicy require-signed-images

Common Policy Controller Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Controller pod crash | Webhook health monitor fires within 30 s | | Webhook TLS cert expires | SSL monitor fires at 14 days; API server rejects webhook on expiry | | Fulcio unreachable | Fulcio monitor fires; keyless signature verification blocked | | Rekor unreachable | Rekor monitor fires; log-inclusion policies fail | | Policy misconfigured (no matching authorities) | Enforcement canary fires; all images rejected or all images admitted | | OOMKilled during complex policy evaluation | Health monitor fires; controller restarts under load | | cert-manager fails to rotate webhook cert | SSL monitor catches approaching expiry | | ClusterImagePolicy CRD not installed | Controller crashes on startup; health monitor fires | | Network policy blocks controller egress | Fulcio/Rekor monitors fire; verification calls time out | | Helm upgrade breaks webhook registration | Health check OK but webhook deregistered; canary monitor catches this |

Policy Controller is the last line of defense for container supply chain integrity — it's the enforcement point that turns image signing from a best-effort practice into a mandatory gate. When it's healthy and its dependencies (Fulcio, Rekor) are reachable, your cluster enforces your supply chain policy automatically on every deployment. When it fails, that enforcement stops silently. Vigilmon gives you continuous external monitoring of the webhook health, TLS certificates, Fulcio and Rekor connectivity, and an active enforcement canary so you always know your supply chain policy is being enforced.

Start monitoring Policy Controller in under 5 minutes — register free at vigilmon.online.