How to Monitor Conjur with Vigilmon

Conjur is CyberArk's open-source secrets manager built for DevOps and machine identity workflows. It provides a REST API for secret retrieval, policy-based access control, audit logging, and authentication via Kubernetes service accounts, LDAP, OIDC, JWT, and API keys. Conjur runs as a containerized service — typically with a leader node and one or more follower nodes for read-heavy environments.

But Conjur itself is critical infrastructure: when the Conjur API is unavailable, every application that fetches secrets at startup or rotation time will fail to authenticate or retrieve credentials. In this tutorial you'll set up uptime and response-time monitoring for your Conjur deployment using Vigilmon — free tier, no credit card required.

Why Conjur needs external monitoring

Conjur's own health checks only tell you what the container sees. External monitoring tells you what your applications see:

• Leader node unreachable — the primary Conjur node is down; all authentication requests and secret fetches fail immediately
• Follower lagging or down — read traffic fails over to the leader, overloading it; rotations and bulk secret fetches slow to a crawl
• Kubernetes authenticator webhook breaks — the Conjur Kubernetes authenticator sidecar loses contact with the server; Conjur-integrated pods can no longer start
• Data key corruption or rotation — after a data key rotation the follower nodes can no longer decrypt the database; secret retrieval returns 500 errors
• JWT or OIDC provider unreachable — if Conjur's auth is backed by an external OIDC provider and that provider is down, all JWT-authenticated workloads can't authenticate

External monitoring from Vigilmon surfaces these failure modes before your workloads start logging authentication errors.

What you'll need

• A running Conjur OSS or Conjur Enterprise deployment
• The Conjur REST API exposed (typically on port 443 or 8080)
• A free Vigilmon account

Step 1: Identify your Conjur endpoints

Conjur exposes a built-in health endpoint you can monitor directly:

# Check Conjur leader health
curl -s https://conjur.yourdomain.com/health | jq .

# Example response
{
  "ok": true,
  "version": "1.20.0",
  "services": {
    "possum": "ok",
    "ok": true
  }
}

# Check a follower node
curl -s https://conjur-follower.yourdomain.com/health | jq .

For Kubernetes-deployed Conjur, get the external endpoint:

kubectl get svc conjur-oss -n conjur
# NAME        TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)
# conjur-oss  LoadBalancer   10.96.120.50   203.0.113.55      443:30443/TCP

kubectl get svc conjur-follower -n conjur
# NAME              TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)
# conjur-follower   LoadBalancer   10.96.120.60   203.0.113.56      443:30444/TCP

Step 2: Verify the API authentication endpoint

Beyond the health endpoint, test that the authentication API is responding. Add a canary policy and test account:

# Authenticate with a host API key to verify the authn endpoint
curl -s -X POST \
  "https://conjur.yourdomain.com/authn/myorg/host%2Fmonitoring-canary/authenticate" \
  -H "Content-Type: text/plain" \
  --data "your-api-key-here"
# Returns a short-lived access token on success

# Wrap the authn check in a lightweight health probe
cat > conjur-authn-probe.cjs << 'EOF'
const http = require('http');
const https = require('https');

const CONJUR_URL = process.env.CONJUR_URL || 'https://conjur.yourdomain.com';
const CONJUR_ACCOUNT = process.env.CONJUR_ACCOUNT || 'myorg';
const HOST_ID = 'host%2Fmonitoring-canary';
const API_KEY = process.env.CONJUR_CANARY_API_KEY;

http.createServer((req, res) => {
  if (req.url !== '/health') { res.writeHead(404); res.end(); return; }

  const opts = new URL(`${CONJUR_URL}/authn/${CONJUR_ACCOUNT}/${HOST_ID}/authenticate`);
  const postReq = https.request({
    hostname: opts.hostname,
    path: opts.pathname,
    method: 'POST',
    headers: { 'Content-Type': 'text/plain', 'Content-Length': API_KEY.length },
  }, r => {
    const ok = r.statusCode === 200;
    res.writeHead(ok ? 200 : 503, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ status: ok ? 'ok' : 'error', authn: r.statusCode }));
  });
  postReq.on('error', e => {
    res.writeHead(503, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ status: 'error', message: e.message }));
  });
  postReq.write(API_KEY);
  postReq.end();
}).listen(9097, () => console.log('Conjur authn probe :9097'));
EOF

CONJUR_CANARY_API_KEY=your-key node conjur-authn-probe.cjs

Expose this probe via a Kubernetes Service so Vigilmon can reach it:

# conjur-probe-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: conjur-health-probe
  namespace: conjur
spec:
  replicas: 1
  selector:
    matchLabels:
      app: conjur-health-probe
  template:
    metadata:
      labels:
        app: conjur-health-probe
    spec:
      containers:
        - name: probe
          image: node:20-alpine
          env:
            - name: CONJUR_URL
              value: "https://conjur-oss.conjur.svc.cluster.local"
            - name: CONJUR_ACCOUNT
              value: "myorg"
            - name: CONJUR_CANARY_API_KEY
              valueFrom:
                secretKeyRef:
                  name: conjur-canary-key
                  key: api-key
          ports:
            - containerPort: 9097
---
apiVersion: v1
kind: Service
metadata:
  name: conjur-health-probe
  namespace: conjur
spec:
  type: LoadBalancer
  selector:
    app: conjur-health-probe
  ports:
    - port: 9097
      targetPort: 9097

Step 3: Set up HTTP monitoring in Vigilmon

With your endpoints identified, add them to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Conjur Leader Health | https://conjur.yourdomain.com/health | 200 | | Conjur Follower Health | https://conjur-follower.yourdomain.com/health | 200 | | Conjur Authn Probe | http://203.0.113.57:9097/health | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match "ok":true in the response body
6. Save each monitor

Vigilmon probes from multiple geographic regions. If your Conjur leader becomes unreachable from one availability zone while appearing healthy in another, the discrepancy is immediately visible on your Vigilmon dashboard.

Step 4: Monitor the TCP layer

Conjur's HTTPS API runs on port 443 by default. Add a TCP monitor to catch TLS-layer failures independently of the HTTP application layer:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter conjur.yourdomain.com and port 443
4. Save the monitor

Add a second TCP monitor for each follower node. If the TCP monitor goes red while the HTTP monitor is still probing (timing window), you know the TLS handshake itself is failing — useful for diagnosing certificate rotation issues.

Step 5: Configure alert channels

When Conjur goes down, every application that fetches secrets on startup or rotates credentials will fail. This tends to cascade: all pods on a deployment restart fail simultaneously.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your secrets management team's on-call address
3. Assign the channel to all Conjur monitors

Webhook alerts for incident automation

1. Go to Alert Channels → Add Channel → Webhook
2. Use a webhook URL that routes to PagerDuty, Opsgenie, or Slack
3. The payload Vigilmon sends:

{
  "monitor_name": "Conjur Leader Health",
  "status": "down",
  "url": "https://conjur.yourdomain.com/health",
  "started_at": "2024-01-15T14:05:00Z",
  "duration_seconds": 180
}

Wire this into a runbook that notifies your SRE team to check follower lag and leader state before attempting a failover.

Step 6: Correlate Vigilmon alerts with Conjur diagnostics

When you receive a Conjur downtime alert, check these in order:

# 1. Check Conjur pod status
kubectl get pods -n conjur -l app=conjur-oss
kubectl get pods -n conjur -l app=conjur-follower

# 2. Check Conjur leader logs for errors
kubectl logs -n conjur deploy/conjur-oss --tail=100 | grep -i "error\|fail\|fatal"

# 3. Verify the health endpoint manually
curl -sv https://conjur.yourdomain.com/health 2>&1 | grep -E "< HTTP|ok"

# 4. Check follower replication lag
curl -s https://conjur-follower.yourdomain.com/health | jq '.replication_status'

# 5. Test authentication directly
curl -s -X POST \
  "https://conjur.yourdomain.com/authn/myorg/host%2Fmonitoring-canary/authenticate" \
  -H "Content-Type: text/plain" \
  --data "$CONJUR_CANARY_API_KEY" | head -c 100

# 6. Check Conjur database connectivity (if self-hosted with external Postgres)
kubectl exec -n conjur deploy/conjur-oss -- \
  psql $DATABASE_URL -c "SELECT 1;" 2>&1

# 7. Inspect Kubernetes authenticator status
kubectl get pods -n conjur -l app=conjur-k8s-authenticator

If the health endpoint returns 200 but authentication fails, the issue is likely with the authenticator configuration or the data key. If the health endpoint itself is unreachable, check network policies and LoadBalancer state first.

Step 7: Create a status page for your secrets infrastructure

A secrets management outage affects every service that rotates credentials. A status page helps teams diagnose before escalating:

1. Go to Status Pages → New Status Page
2. Name it: "Secrets Management - Conjur"
3. Add your monitors: Conjur Leader Health, Conjur Follower Health, Conjur Authn Probe
4. Share the URL with your platform engineering and security teams

During an incident, teams can check the status page to confirm whether a connectivity error is a Conjur outage or an application misconfiguration — reducing noise in your incident channel.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on Conjur /health | Leader and follower outages, degraded state | | HTTP monitor on authn probe | Authentication endpoint failures | | TCP monitor on port 443 | TLS certificate failures, port-level outages | | Email + webhook alert channels | Immediate notification before cascading auth failures | | Status page | Team self-service during secrets outages |

The goal is to make your Conjur deployment as observable as the applications it secures. When Conjur goes dark, you want an alert before workloads start failing to authenticate — not after.

Get started at vigilmon.online — free tier, no credit card required.