tutorial

How to Monitor Datree Policy Pipelines with Vigilmon

Datree is a Kubernetes misconfiguration prevention tool that lets you enforce policy as code across your CLI workflows and CI/CD pipelines. It catches miscon...

Datree is a Kubernetes misconfiguration prevention tool that lets you enforce policy as code across your CLI workflows and CI/CD pipelines. It catches misconfigurations — missing resource limits, insecure pod specs, deprecated API versions — before they ever reach your cluster.

But Datree itself is infrastructure: a policy server, a CLI that phones home to validate schemas, and a CI integration that can silently break if the policy service is unreachable. In this tutorial you'll set up uptime and response-time monitoring for the Datree policy service and any associated webhooks using Vigilmon — free tier, no credit card required.

Why Datree pipelines need external monitoring

When Datree runs in CI, it calls out to the Datree policy server to fetch the active policy ruleset and validate manifests. That external dependency creates failure modes your internal checks won't catch:

  • Policy server unreachable — CI jobs that run datree test silently time out or fail with a network error, blocking deployments for unclear reasons
  • Self-hosted policy server down — if you run Datree Enterprise with an on-prem policy server, an outage means all manifest validation stops
  • Webhook integration breaks — Datree can push policy-violation events to external webhooks; if the consumer endpoint is down, you lose the audit trail
  • CLI version drift — the Datree CLI may pin to a specific API version; if the backend rotates API schemas without notice, CI breaks across multiple pipelines simultaneously

External monitoring from Vigilmon catches these before your engineers start debugging why CI is blocked.

What you'll need

  • Datree installed in your CI pipeline (datree test running against manifests)
  • Optionally, a self-hosted Datree Enterprise policy server
  • A free Vigilmon account

Step 1: Identify your Datree endpoints

Depending on your deployment model you'll have one or more endpoints to monitor:

Cloud-managed Datree (default)

The Datree CLI connects to app.datree.io to fetch policies. You can monitor this for availability, but since it's a third-party SaaS endpoint you primarily want to know when it's down so you can pause CI or switch to offline mode.

Self-hosted Datree Enterprise

You control the policy server. Add a health endpoint to your deployment:

# datree-server-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: datree-policy-server
  namespace: datree-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: datree-policy-server
  template:
    metadata:
      labels:
        app: datree-policy-server
    spec:
      containers:
        - name: datree-policy-server
          image: datree/datree-enterprise:latest
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: datree-policy-server
  namespace: datree-system
spec:
  type: LoadBalancer
  selector:
    app: datree-policy-server
  ports:
    - port: 80
      targetPort: 8080

kubectl apply -f datree-server-deployment.yaml
kubectl get svc datree-policy-server -n datree-system
# NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)       AGE
# datree-policy-server   LoadBalancer   10.96.40.200   203.0.113.80    80:31500/TCP  2m

Step 2: Add a health endpoint to your CI validation wrapper

If you run datree test in a wrapper script, expose a simple HTTP health check that Vigilmon can probe:

#!/usr/bin/env bash
# datree-health-server.sh — runs alongside CI to expose health status
# Start with: node datree-health-server.cjs &

cat > datree-health-server.cjs << 'EOF'
const http = require('http');
const { execSync } = require('child_process');

const server = http.createServer((req, res) => {
  if (req.url === '/health') {
    try {
      // Check that datree CLI is available and can reach the policy server
      execSync('datree config get token', { timeout: 5000 });
      res.writeHead(200, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ status: 'ok', datree: 'reachable' }));
    } catch (err) {
      res.writeHead(503, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ status: 'error', message: err.message }));
    }
  } else {
    res.writeHead(404);
    res.end();
  }
});

server.listen(9090, () => console.log('Datree health server on :9090'));
EOF

node datree-health-server.cjs

Expose port 9090 and verify:

curl http://your-ci-host:9090/health
# {"status":"ok","datree":"reachable"}

Step 3: Set up HTTP monitoring in Vigilmon

With your endpoints identified, add them to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. For each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Datree Policy Server | https://policy.yourdomain.com/health | 200 | | Datree CI Health | http://ci-host:9090/health | 200 | | Datree Webhook Consumer | https://webhook.yourdomain.com/health | 200 |

  1. Set the check interval to 1 minute
  2. Under Expected response, set status code 200 and optionally "status":"ok" in the body
  3. Save each monitor

Vigilmon probes from multiple geographic regions simultaneously. If your self-hosted policy server goes down, every CI job that runs datree test will fail within minutes — you'll know why immediately instead of debugging phantom CI failures.

Step 4: Monitor the TCP layer

If your policy server uses a custom TCP port or you have a gRPC API:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter the host and port of your policy server gRPC interface (e.g., policy.yourdomain.com / 9443)
  4. Save the monitor

A TCP monitor catches TLS handshake failures and port-level outages that an HTTP check might not surface clearly.

Step 5: Configure alert channels

When the Datree policy server goes down, every deployment pipeline that enforces policy validation is blocked. You want to know immediately.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call address
  3. Assign the channel to your Datree monitors

Webhook alerts for CI notification

  1. Go to Alert Channels → Add Channel → Webhook
  2. Use a webhook URL that routes to your CI notification system (Slack, PagerDuty, etc.)
  3. The payload Vigilmon sends:
{
  "monitor_name": "Datree Policy Server",
  "status": "down",
  "url": "https://policy.yourdomain.com/health",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 90
}

Use this to trigger an automatic CI gate override that skips Datree validation while the server is down, or to notify engineers to switch to offline policy mode.

Step 6: Correlate Vigilmon alerts with Datree diagnostics

When you receive a Datree downtime alert, check these things in order:

# 1. Verify the Datree CLI can reach the policy server
datree config get token

# 2. Test policy validation directly
datree test path/to/manifests/ --policy my-policy

# 3. Check the Datree policy server pod status
kubectl get pods -n datree-system -l app=datree-policy-server

# 4. View recent policy server logs
kubectl logs -n datree-system -l app=datree-policy-server --tail=50

# 5. Check if the LoadBalancer external IP is still assigned
kubectl get svc datree-policy-server -n datree-system

# 6. Verify CRD and policy definitions are intact
kubectl get datreepolicies --all-namespaces 2>/dev/null || \
  datree policy ls

If Vigilmon shows the endpoint as down but pods are running, the issue is likely at the LoadBalancer or network routing layer — exactly the class of failure Kubernetes internal probes miss.

Step 7: Create a status page for your policy infrastructure

If multiple teams share your Datree policy server, a status page lets them self-serve when they notice CI failures:

  1. Go to Status Pages → New Status Page
  2. Name it: "Policy Enforcement Infrastructure"
  3. Add your monitors: Datree Policy Server, Datree Webhook Consumer
  4. Share the URL with your platform engineering channel

Teams can check the status page before filing a "CI is broken" ticket, reducing interruptions to your platform team.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on policy server /health | Policy server downtime, slow responses | | HTTP monitor on webhook consumer | Audit trail gaps from missed events | | TCP monitor on gRPC port | Low-level port or TLS failures | | Email + webhook alert channels | Immediate notification when CI validation is blocked | | Status page | Team self-service during incidents |

The goal is to make Datree's external dependencies as observable as the services Datree itself protects. When your policy enforcement infrastructure goes dark, you want an alert before developers start wondering why their deployments are stuck — not after.

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles