How to Monitor Conftest Policy Services with Vigilmon

Conftest is a tool for writing tests against structured configuration data — Kubernetes manifests, Terraform plans, Dockerfile instructions, and more — using Open Policy Agent (OPA) Rego policies. It runs at development time, in CI, and as a pre-deployment gate, enforcing your organization's standards before bad config reaches production.

Because Conftest integrates into critical deployment pipelines, any failure in its infrastructure blocks deployments. In this tutorial you'll set up monitoring for Conftest-related services and OPA backends using Vigilmon — free tier, no credit card required.

Why Conftest infrastructure needs external monitoring

Conftest itself is a stateless CLI — it reads policies from a local directory or from a remote OPA bundle server. The monitoring concern is the infrastructure around it:

• OPA bundle server down — if you use conftest pull to fetch remote policies, an unreachable bundle server means CI falls back to stale policies or fails entirely
• OPA REST API unavailable — teams that run a shared OPA server for policy queries face total validation failure if that server goes down
• Policy bundle update pipeline broken — the CI job that publishes updated Rego bundles to your OCI registry or bundle server may have failed silently; Conftest silently uses outdated policies
• HTTP policy registry unreachable — Conftest supports pulling policies from HTTP endpoints; if the registry is down, conftest pull fails and your gates are either skipped or hard-blocked

External monitoring from Vigilmon closes the gap between "CI passed" and "policy enforcement is actually working."

What you'll need

• Conftest running in CI against Kubernetes manifests or other config
• Optionally, a shared OPA server or bundle registry
• A free Vigilmon account

Step 1: Deploy a shared OPA server with a health endpoint

If your team runs a centralized OPA server that Conftest queries, it needs to be monitored. Here's a minimal production-ready deployment:

# opa-server-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: opa-server
  namespace: opa-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: opa-server
  template:
    metadata:
      labels:
        app: opa-server
    spec:
      containers:
        - name: opa
          image: openpolicyagent/opa:latest-rootless
          args:
            - run
            - --server
            - --addr=:8181
            - --log-level=info
            - --bundle
            - /bundles
          ports:
            - containerPort: 8181
          livenessProbe:
            httpGet:
              path: /health
              port: 8181
            initialDelaySeconds: 10
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /health?bundle=true
              port: 8181
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          volumeMounts:
            - name: bundles
              mountPath: /bundles
      volumes:
        - name: bundles
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: opa-server
  namespace: opa-system
spec:
  type: LoadBalancer
  selector:
    app: opa-server
  ports:
    - port: 8181
      targetPort: 8181

kubectl apply -f opa-server-deployment.yaml
kubectl get svc opa-server -n opa-system
# NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
# opa-server   LoadBalancer   10.96.30.100   203.0.113.70    8181:31700/TCP   2m

Verify the health endpoint:

curl http://203.0.113.70:8181/health
# {}
# An empty JSON object means OPA is healthy (HTTP 200)

curl http://203.0.113.70:8181/health?bundle=true
# {} — confirms the bundle was loaded successfully

Step 2: Monitor the OPA bundle registry

If you publish Conftest policies to an OCI registry or an HTTP bundle server, add a health check endpoint to that service:

// bundle-server.cjs — minimal bundle HTTP server with health endpoint
const http = require('http');
const fs = require('fs');
const path = require('path');

const BUNDLES_DIR = process.env.BUNDLES_DIR || './bundles';

const server = http.createServer((req, res) => {
  if (req.url === '/health') {
    // Check that at least one bundle file exists
    try {
      const files = fs.readdirSync(BUNDLES_DIR);
      const bundles = files.filter(f => f.endsWith('.tar.gz'));
      if (bundles.length === 0) {
        res.writeHead(503, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ status: 'error', reason: 'no bundles available' }));
        return;
      }
      res.writeHead(200, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ status: 'ok', bundles: bundles.length }));
    } catch (err) {
      res.writeHead(503, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ status: 'error', reason: err.message }));
    }
  } else if (req.url.startsWith('/bundles/')) {
    const file = path.join(BUNDLES_DIR, path.basename(req.url));
    if (fs.existsSync(file)) {
      res.writeHead(200, { 'Content-Type': 'application/gzip' });
      fs.createReadStream(file).pipe(res);
    } else {
      res.writeHead(404);
      res.end();
    }
  } else {
    res.writeHead(404);
    res.end();
  }
});

server.listen(8080, () => console.log('Bundle server on :8080'));

Step 3: Set up HTTP monitoring in Vigilmon

Add your OPA and bundle endpoints to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add monitors for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | OPA Server Health | http://203.0.113.70:8181/health | 200 | | OPA Bundle Loaded | http://203.0.113.70:8181/health?bundle=true | 200 | | Policy Bundle Registry | https://bundles.yourdomain.com/health | 200 |

4. Set check interval to 1 minute
5. Under Expected response, set status code 200
6. Save each monitor

The ?bundle=true OPA health check is especially important: a plain /health returning 200 only means OPA is running, not that it successfully loaded your policies. The bundle check confirms policy data is actually available for evaluation.

Step 4: Monitor the TCP layer

OPA listens on port 8181 by default. Add a TCP monitor as a low-level availability check:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter: host 203.0.113.70, port 8181
4. Save the monitor

If OPA crashes or the port stops listening, the TCP check fires before the HTTP check times out — giving you a few extra seconds of early warning.

Step 5: Configure alert channels

When your OPA server goes down, every conftest test run that queries it will fail, blocking deployments across potentially many teams.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter the platform team's email
3. Assign the channel to all Conftest/OPA monitors

Webhook alerts

1. Go to Alert Channels → Add Channel → Webhook
2. Use your Slack or PagerDuty webhook URL
3. The Vigilmon payload:

{
  "monitor_name": "OPA Server Health",
  "status": "down",
  "url": "http://203.0.113.70:8181/health",
  "started_at": "2024-01-15T11:00:00Z",
  "duration_seconds": 45
}

Wire this into an automated response that pages the on-call platform engineer and posts a notice in your engineering Slack channel.

Step 6: Verify your Conftest pipeline end-to-end

When Vigilmon alerts fire, use these diagnostics to pinpoint the failure:

# 1. Test Conftest policy fetch directly
conftest pull --policy ./policy https://bundles.yourdomain.com/bundles/k8s-policies.tar.gz

# 2. Run a policy test against a sample manifest
conftest test deployment.yaml --policy ./policy

# 3. Check OPA server health manually
curl -s http://203.0.113.70:8181/health?bundle=true | jq .

# 4. Query an OPA policy directly to verify evaluation works
curl -X POST http://203.0.113.70:8181/v1/data/kubernetes/admission \
  -H 'Content-Type: application/json' \
  -d '{"input": {"kind": "Deployment", "metadata": {"name": "test"}}}'

# 5. Check OPA pod logs for bundle fetch errors
kubectl logs -n opa-system -l app=opa-server --tail=50 | grep -E 'error|bundle|failed'

# 6. Check OPA pod restarts
kubectl get pods -n opa-system -l app=opa-server

If Vigilmon shows the endpoint down but pods are running, the LoadBalancer or network routing is the failure point. If pods are restarting, check for OOM or bundle parsing errors in the logs.

Step 7: Monitor bundle freshness with a custom check

Stale policies are a silent failure mode — Conftest is running, OPA is healthy, but your policies haven't updated in three days. Add a bundle age check to your health endpoint:

// Add to your bundle server health handler
const stat = fs.statSync(path.join(BUNDLES_DIR, 'k8s-policies.tar.gz'));
const ageMinutes = (Date.now() - stat.mtimeMs) / 60000;
if (ageMinutes > 60) {
  // Bundle not updated in over an hour — surface this in the health response
  res.writeHead(503, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify({
    status: 'stale',
    bundle_age_minutes: Math.round(ageMinutes),
    reason: 'bundle not updated recently'
  }));
  return;
}

Now Vigilmon will alert when the bundle update pipeline breaks, not just when the server crashes.

Step 8: Create a status page for policy infrastructure

1. Go to Status Pages → New Status Page
2. Name it: "Policy Enforcement — OPA / Conftest"
3. Add monitors: OPA Server Health, OPA Bundle Loaded, Policy Bundle Registry
4. Share the URL internally

Developers who see conftest test failing in CI can immediately check whether the policy infrastructure is the cause.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on /health | OPA server downtime | | HTTP monitor on /health?bundle=true | Policy bundle load failures | | HTTP monitor on bundle registry | Bundle publish pipeline failures | | Bundle staleness check | Silently outdated policies | | TCP monitor on port 8181 | Low-level port unavailability | | Alert channels | Immediate notification when deployments are blocked |

Conftest gives you policy-as-code confidence in CI. Vigilmon gives you confidence that the policy infrastructure itself is healthy — because the worst time to discover your OPA server has been down for two hours is when a security audit asks why the last fifty deployments bypassed policy checks.

Get started at vigilmon.online — free tier, no credit card required.