Monitoring CUE-Powered Infrastructure with Vigilmon: Admission Webhooks, Validation APIs, and Configuration Pipeline Health

CUE is a data constraint language and toolchain for defining, validating, and generating configuration at scale. Platform engineering teams use CUE to validate Kubernetes resource schemas against organizational policies, generate environment-specific configurations from a single source of truth, and enforce API contract constraints across microservices. The most critical CUE infrastructure in production is the admission webhook that validates every Kubernetes manifest before it reaches the API server — if that webhook is down, deployments are blocked. Configuration validation APIs that serve CUE evaluations are equally critical: when they fail, CI pipelines stall waiting for schema checks that never complete. Vigilmon gives you external, agent-free visibility into every CUE infrastructure availability surface.

What You'll Build

• An HTTP monitor on a CUE-based Kubernetes admission webhook to detect validation failures
• A schema validation API health check for CUE evaluation services
• A TCP port monitor on the webhook admission port
• SSL certificate monitoring for webhook TLS (required by Kubernetes)
• An alerting setup that distinguishes admission webhook failures from CI validation failures

Prerequisites

• A CUE-based Kubernetes admission webhook (e.g., built with cue def + a webhook server) accessible over HTTPS
• A CUE schema validation API service with a health endpoint
• Kubernetes 1.21+ with ValidatingWebhookConfiguration or MutatingWebhookConfiguration pointing to your webhook
• A free account at vigilmon.online

Step 1: Understand CUE's Production Availability Surfaces

CUE in production manifests in three main ways:

1. Kubernetes Admission Webhooks — A webhook server that runs cue vet or a CUE evaluator against incoming Kubernetes resources. The Kubernetes API server calls this webhook synchronously on every kubectl apply. If the webhook returns an error or times out, the resource is rejected (or, for mutating webhooks with failurePolicy: Ignore, silently bypassed).

# Test the webhook health endpoint directly
curl -k https://cue-webhook.example.com:8443/healthz
# Returns: {"status":"ok","validatorVersion":"0.x.y"}

2. Schema Validation APIs — A REST service that wraps CUE evaluation, accepting JSON or YAML payloads and returning validation results. Used by CI pipelines to validate configuration before deployment.

curl -s https://cue-validate.example.com/health
# Returns: {"status":"healthy","cueVersion":"v0.8.x"}

3. Configuration Generation APIs — A service that takes CUE definitions and parameters and returns rendered YAML/JSON configuration for downstream systems.

Step 2: Monitor the CUE Admission Webhook

The admission webhook is the most critical CUE component — Kubernetes calls it synchronously. A failed webhook blocks all resource creation in the affected namespaces:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://cue-webhook.example.com:8443/healthz.
3. Check interval: 30 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok (present in the health response).
7. Label: CUE Admission Webhook.
8. Click Save.

This monitor catches:

• Webhook server pod crashes or OOM kills
• CUE evaluator goroutine panics that take down the webhook process
• Memory pressure causing webhook latency that exceeds Kubernetes' webhook timeout (default 10s)
• Kubernetes Service or endpoint routing failures that prevent the API server from reaching the webhook
• Deployment failures during webhook server upgrades that leave it in a broken state

Alert sensitivity: Trigger after 1 consecutive failure. When the CUE admission webhook fails and your ValidatingWebhookConfiguration uses failurePolicy: Fail, every kubectl apply in the webhook's scope is blocked immediately.

Step 3: Monitor the TCP Port for the Admission Webhook

Kubernetes calls the webhook over TCP before completing the TLS handshake. A TCP-level check confirms the port is open independently of HTTP health:

nc -zv cue-webhook.example.com 8443

1. Add Monitor → TCP.
2. Host: cue-webhook.example.com.
3. Port: 8443 (standard webhook port; adjust if using a non-standard port behind an ingress).
4. Check interval: 60 seconds.
5. Label: CUE Webhook TCP.
6. Click Save.

TCP port checks detect network policy changes or load balancer failures that prevent the Kubernetes API server from reaching the webhook, which would otherwise only manifest as deploy timeouts — not obvious 4xx errors.

Step 4: Monitor the CUE Schema Validation API

CI/CD pipelines call the schema validation API to check configuration before committing it. If this API is down, pipelines either fail hard (blocking all merges) or skip validation (allowing invalid configs to reach production):

curl -s https://cue-validate.example.com/health
# Returns: {"status":"healthy","cueVersion":"v0.8.x"}

1. Add Monitor → HTTP.
2. URL: https://cue-validate.example.com/health.
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: healthy.
7. Label: CUE Validation API.
8. Click Save.

For a deeper functional check that tests actual CUE evaluation:

curl -s -X POST https://cue-validate.example.com/validate \
  -H "Content-Type: application/json" \
  -d '{"schema":"#Config","value":{"replicas":1,"image":"nginx:latest"}}' \
| jq '.valid'
# Returns: true

Add a second HTTP monitor with a POST body if Vigilmon supports request body configuration, targeting this /validate endpoint with keyword: true.

Step 5: Monitor SSL Certificates for Webhook TLS

Kubernetes requires TLS for all admission webhook endpoints — it validates the webhook's certificate against the caBundle field in the ValidatingWebhookConfiguration. An expired certificate causes Kubernetes to reject the webhook handshake, making it equivalent to the webhook being down:

openssl s_client -connect cue-webhook.example.com:8443 2>/dev/null | openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: cue-webhook.example.com.
3. Port: 8443.
4. Alert when expiry is within: 30 days.
5. Alert again: 14 days, 7 days, 3 days, 1 day.
6. Click Save.

Webhook TLS certificates are often managed by cert-manager with a 90-day rotation. The caBundle in the ValidatingWebhookConfiguration must be updated each time the CA rotates. A 30-day alert window gives you time to verify that cert-manager's rotation and the webhook configuration update completed successfully before the old cert expires.

Step 6: Monitor Configuration Generation Endpoints

If your platform exposes a CUE-based configuration generation API — for example, a service that takes a service definition and returns rendered Kubernetes YAML — monitor its health separately:

curl -s https://cue-generate.example.com/healthz
# Returns: {"status":"ok","schemasLoaded":42}

1. Add Monitor → HTTP.
2. URL: https://cue-generate.example.com/healthz.
3. Check interval: 3 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: schemasLoaded (confirms CUE schemas were loaded, not just the HTTP server started).
7. Label: CUE Config Generator.
8. Click Save.

schemasLoaded in the keyword check is more meaningful than just checking for ok — it confirms the CUE definitions were successfully parsed and loaded into memory, catching the subtle case where the server starts but fails to read its schema files.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | CUE admission webhook HTTP | Non-200 or keyword missing | Webhook pod down; kubectl apply will block in affected namespaces | | CUE admission webhook TCP | Connection refused | Network policy or load balancer failure blocking API server → webhook | | CUE validation API | Non-200 or keyword missing | CI pipelines blocked or skipping validation | | SSL certificate (webhook) | < 30 days to expiry | Renew cert; verify caBundle in ValidatingWebhookConfiguration | | CUE config generator | Non-200 or keyword missing | Config generation failing; check schema file loading |

Alert after: 1 consecutive failure for the admission webhook and TCP monitors. 2 consecutive failures for the validation API and config generator.

Common CUE Infrastructure Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Webhook pod OOM killed | HTTP webhook monitor fires within 30 s; kubectl apply blocks | | Kubernetes network policy blocks webhook | TCP monitor fires; HTTP monitor cannot connect | | Webhook cert expires | SSL monitor alerts 30 days ahead; Kubernetes rejects TLS at expiry | | CUE evaluation panic / goroutine leak | Webhook health returns 500; monitor fires immediately | | Schema file corrupt after ConfigMap update | Generator monitor catches schemasLoaded:0 | | Validation API overloaded under CI burst | Response time spikes; Vigilmon timeout alert fires | | Webhook timeout under heavy load | Kubernetes apply hangs; TCP monitor stays green, HTTP times out | | cert-manager rotation fails (caBundle not updated) | SSL monitor catches cert expiry before kubectl apply failures begin |

CUE's power in production comes from enforcing configuration contracts at the admission layer and in CI pipelines. When the admission webhook is down, kubectl apply blocks. When the validation API is unavailable, pipelines skip checks or stall. When TLS certificates expire, Kubernetes silently rejects every webhook call. Vigilmon gives you external visibility into each layer — webhook liveness, TCP connectivity, schema validation availability, and certificate expiry — so CUE keeps enforcing your platform standards without unexpected gaps.

Start monitoring your CUE infrastructure in under 5 minutes — register free at vigilmon.online.