How to Monitor Trust Manager with Vigilmon

Trust Manager is a cert-manager subproject that distributes trust bundles — collections of CA certificates — across Kubernetes namespaces. It introduces a Bundle custom resource that defines which CA certificates to collect and which namespaces to copy them into as ConfigMaps or Secrets. Applications that need to verify TLS connections to internal services consume these bundles automatically.

But Trust Manager is infrastructure for your PKI distribution layer: when it fails to propagate an updated CA bundle, workloads across multiple namespaces silently start trusting stale certificates. When a bundle update doesn't reach a namespace in time, TLS verification failures cascade across services. In this tutorial you'll set up uptime and response-time monitoring for Trust Manager using Vigilmon — free tier, no credit card required.

Why Trust Manager needs external monitoring

Trust Manager's failure modes are subtle and often discovered late:

• Controller crashes after cert-manager upgrade — Trust Manager depends on cert-manager CRDs; a version skew can crash the controller pod, silently halting all bundle propagation
• Bundle propagation stalls — a new CA is added to a Bundle source but ConfigMaps in target namespaces don't update; services that rotate to the new CA begin failing TLS verification without an obvious error
• Webhook admission controller down — Trust Manager's optional admission webhook rejects Bundle object changes when the webhook server is unreachable, blocking PKI updates
• RBAC misconfiguration — a namespace RBAC change prevents Trust Manager from writing to specific target namespaces; bundles silently diverge between namespaces
• Bundle source unavailable — if a Bundle references a cert-manager Certificate or an in-cluster ConfigMap that gets deleted, the bundle can't be assembled and all target namespaces remain at the old version

External monitoring from Vigilmon catches these before your TLS verification failures become customer-visible errors.

What you'll need

• cert-manager installed in your cluster
• Trust Manager installed (helm upgrade --install trust-manager jetstack/trust-manager --namespace cert-manager)
• A free Vigilmon account

Step 1: Expose a health endpoint for the Trust Manager controller

Trust Manager exposes Prometheus metrics and a health endpoint on port 9402 by default. Forward this to an external endpoint for Vigilmon to reach:

# Verify Trust Manager is running and healthy
kubectl get pods -n cert-manager -l app=trust-manager
# NAME                             READY   STATUS    RESTARTS   AGE
# trust-manager-5d9b47c6f9-xk8qr   1/1     Running   0          12d

# Check the health endpoint (port 9402 by default)
kubectl port-forward -n cert-manager deploy/trust-manager 9402:9402 &
curl http://localhost:9402/healthz
# ok

Expose the health endpoint externally via a Service and Ingress:

# trust-manager-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: trust-manager-health
  namespace: cert-manager
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: trust-manager
  ports:
    - name: health
      port: 9402
      targetPort: 9402

kubectl apply -f trust-manager-health-service.yaml
kubectl get svc trust-manager-health -n cert-manager
# NAME                    TYPE           EXTERNAL-IP     PORT(S)
# trust-manager-health    LoadBalancer   203.0.113.60    9402:31200/TCP

curl http://203.0.113.60:9402/healthz
# ok

Step 2: Add a bundle propagation probe

Beyond pod health, verify that Trust Manager is actually propagating bundles. Create a canary Bundle and check whether target namespaces receive the expected ConfigMap:

# canary-bundle.yaml
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: monitoring-canary-bundle
spec:
  sources:
    - useDefaultCAs: true
  target:
    configMap:
      key: "ca-bundle.crt"
    namespaceSelector:
      matchLabels:
        trust-manager-canary: "true"

kubectl apply -f canary-bundle.yaml

# Label a canary namespace
kubectl label namespace monitoring trust-manager-canary=true

# After a few seconds, verify the bundle was propagated
kubectl get configmap monitoring-canary-bundle -n monitoring -o jsonpath='{.data.ca-bundle\.crt}' | head -5
# -----BEGIN CERTIFICATE-----

Wrap this check in a health probe:

cat > trust-manager-probe.sh << 'EOF'
#!/bin/sh
# Returns 0 if the canary bundle exists and is non-empty, 1 otherwise
CM=$(kubectl get configmap monitoring-canary-bundle -n monitoring \
  -o jsonpath='{.data.ca-bundle\.crt}' 2>/dev/null)
if [ -n "$CM" ]; then
  echo '{"status":"ok","bundle":"propagated"}'
  exit 0
else
  echo '{"status":"error","bundle":"missing"}'
  exit 1
fi
EOF
chmod +x trust-manager-probe.sh

Expose this probe as an HTTP endpoint via a small in-cluster service so Vigilmon can poll it:

# trust-probe-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trust-manager-probe
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: trust-manager-probe
  template:
    metadata:
      labels:
        app: trust-manager-probe
    spec:
      serviceAccountName: trust-manager-probe
      containers:
        - name: probe
          image: bitnami/kubectl:latest
          command:
            - sh
            - -c
            - |
              apk add nodejs --quiet && node -e "
              const http = require('http');
              const { execSync } = require('child_process');
              http.createServer((req, res) => {
                if (req.url !== '/health') { res.writeHead(404); res.end(); return; }
                try {
                  const cm = execSync(
                    'kubectl get configmap monitoring-canary-bundle -n monitoring -o jsonpath={.data.ca-bundle\\.crt}',
                    { timeout: 5000 }
                  ).toString();
                  const ok = cm.length > 0;
                  res.writeHead(ok ? 200 : 503, {'Content-Type':'application/json'});
                  res.end(JSON.stringify({ status: ok ? 'ok' : 'error', bytes: cm.length }));
                } catch (e) {
                  res.writeHead(503, {'Content-Type':'application/json'});
                  res.end(JSON.stringify({ status: 'error', message: e.message }));
                }
              }).listen(9098);
              "
          ports:
            - containerPort: 9098
---
apiVersion: v1
kind: Service
metadata:
  name: trust-manager-probe
  namespace: monitoring
spec:
  type: LoadBalancer
  selector:
    app: trust-manager-probe
  ports:
    - port: 9098
      targetPort: 9098

Step 3: Set up HTTP monitoring in Vigilmon

With your endpoints ready, add them to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Trust Manager Health | http://203.0.113.60:9402/healthz | 200 | | Trust Manager Bundle Probe | http://203.0.113.61:9098/health | 200 | | Trust Manager Webhook | https://trust-manager-webhook.cert-manager.svc/healthz | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match ok or "status":"ok" in the body
6. Save each monitor

Vigilmon probes from multiple geographic regions. A Trust Manager outage that only affects certain cluster regions will be visible in the geographic breakdown on your dashboard.

Step 4: Monitor the TCP layer

Trust Manager's webhook admission controller runs on port 6443 by default. Add a TCP monitor to catch TLS failures at the admission layer:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your Trust Manager webhook host and port (e.g., trust-manager-webhook.cert-manager.svc / 6443)
4. Save the monitor

If the TCP monitor goes red, Bundle object changes will be rejected by the Kubernetes API server, blocking any PKI updates.

Step 5: Configure alert channels

A Trust Manager outage doesn't cause immediate visible failures, but it means your trust bundles are silently diverging. You need to catch this early — before the next CA rotation.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your PKI or platform team's address
3. Assign the channel to all Trust Manager monitors

Webhook alerts for PKI automation

1. Go to Alert Channels → Add Channel → Webhook
2. Use a webhook URL that routes to Slack or your incident management system
3. The payload Vigilmon sends:

{
  "monitor_name": "Trust Manager Bundle Probe",
  "status": "down",
  "url": "http://203.0.113.61:9098/health",
  "started_at": "2024-01-15T16:00:00Z",
  "duration_seconds": 300
}

Wire this into an alert that blocks any scheduled CA rotations until the bundle propagation is confirmed healthy — preventing a rotation from landing in a cluster where half the namespaces still trust the old CA.

Step 6: Correlate Vigilmon alerts with Trust Manager diagnostics

When you receive a Trust Manager alert, check these in order:

# 1. Check the Trust Manager controller pod
kubectl get pods -n cert-manager -l app.kubernetes.io/name=trust-manager
kubectl describe pod -n cert-manager -l app.kubernetes.io/name=trust-manager

# 2. Check Trust Manager logs for reconciliation errors
kubectl logs -n cert-manager deploy/trust-manager --tail=100 \
  | grep -i "error\|fail\|reconcil"

# 3. Inspect Bundle status conditions
kubectl get bundle monitoring-canary-bundle -o yaml | grep -A20 'status:'
# Look for Ready: True and Reason: Synced

# 4. Check all Bundles for sync failures
kubectl get bundles --all-namespaces -o wide

# 5. Verify target namespace ConfigMaps exist
kubectl get configmap -l trust.cert-manager.io/bundle=monitoring-canary-bundle \
  --all-namespaces

# 6. Check cert-manager CRD compatibility
kubectl get crd bundles.trust.cert-manager.io -o jsonpath='{.spec.versions[*].name}'

# 7. Look at webhook events
kubectl get events -n cert-manager --field-selector reason=FailedMount \
  | grep trust-manager

If the health endpoint returns 200 but the bundle probe fails, the controller is running but a specific Bundle is not reconciling — look at the Bundle's .status.conditions for the root cause.

Step 7: Create a status page for your PKI distribution layer

CA bundle distribution affects TLS verification across your entire cluster. A status page surfaces Trust Manager health alongside cert-manager:

1. Go to Status Pages → New Status Page
2. Name it: "PKI Distribution Infrastructure"
3. Add your monitors: Trust Manager Health, Trust Manager Bundle Probe
4. Share the URL with your platform team and security team

When a TLS verification failure is reported, teams can check whether it correlates with a Trust Manager outage before deep-diving into certificate debugging.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on /healthz | Controller crashes, pod restarts | | HTTP monitor on bundle probe | Bundle propagation stalls, namespace gaps | | TCP monitor on webhook port | Webhook admission failures blocking Bundle updates | | Email + webhook alert channels | Immediate notification before CA rotation windows | | Status page | Team self-service during PKI incidents |

The goal is to make your CA bundle distribution layer as observable as the certificates it distributes. When Trust Manager stops propagating bundles, you want an alert before the next CA rotation causes cluster-wide TLS failures — not after.

Get started at vigilmon.online — free tier, no credit card required.