How to Monitor Notary v2 with Vigilmon (Signing Service Uptime, Registry Integration, Policy Enforcement)

Notary v2 — implemented as the notation CLI and managed under the CNCF Notary Project — signs OCI artifacts and container images using X.509 certificates stored in notation trust stores. Ratify, or a custom admission webhook, enforces verification policy at deploy time: images without a valid signature are rejected before they can run in your Kubernetes cluster.

The failure modes cut in both directions. If your signing service is down, engineers can't sign new images and your CI/CD pipeline stalls. If the Ratify admission webhook is unhealthy or misconfigured, the enforcement gate is silently open — unverified images may be admitted. If your signing certificates expire, every previously signed image fails verification. None of these failures produce obvious alerts; they surface as mysterious pipeline blocks or, worse, as security policy violations that go undetected.

Vigilmon monitors the components that keep your signing and enforcement pipeline healthy — the signing roundtrip, the Ratify webhook, the certificate expiry date, and the registry integration that stores signatures — so you find out before your users or your security team do.

Why Notary v2 needs external monitoring

Notary v2 spans multiple systems, each of which can fail independently:

Signing service down — if you use a centralised signing service (a KMS-backed sidecar, a notation-server deployment, or a CI agent with access to private keys) and it becomes unreachable, notation sign hangs or exits non-zero; new image builds cannot be signed; your pipeline is blocked
Registry integration broken — notation pushes signatures as OCI referrers to the same registry that holds the image; if the registry rejects the referrer push (auth misconfiguration, storage quota, API version mismatch), the signature is lost even though notation sign may exit 0 when using some shim configurations
Ratify admission webhook unhealthy — Ratify runs as a Kubernetes ValidatingWebhookConfiguration; if its pods are down or the webhook service is unreachable, the Kubernetes API server falls back to the failurePolicy setting (Ignore in many tutorials) and lets unsigned images through without any error
Certificate expiry — notation trusts certificates by thumbprint via trust stores; when a signing certificate expires, notation verify fails for every artifact signed with it, regardless of when it was signed; this can block deployment of images that have been in production for months
TSA (timestamp authority) unreachable — notation supports RFC 3161 timestamp countersignatures so signatures remain valid after certificate expiry; if your TSA endpoint is down during signing, timestamps are omitted and you lose long-lived signature validity

What you'll need

notation CLI (v1.0.0 or later) installed and configured with a trust store and trust policy
A Kubernetes cluster running Ratify (or another Notary v2 admission webhook)
A container registry that supports OCI referrers (ORAS-compatible: ACR, ECR, GCR, Harbor, Zot, Docker Hub)
A free Vigilmon account — no credit card required

Step 1: Verify the notation signing pipeline health

The most direct test is a signing roundtrip: push a scratch image, sign it, verify it, then clean up. Wrap this in a health endpoint so Vigilmon can probe it on a schedule.

First, write a script that performs the roundtrip:

#!/bin/bash
# notation-health-check.sh
# Requires: notation, crane (or oras), and a test image in the registry

set -euo pipefail

REGISTRY="${NOTATION_REGISTRY:-registry.example.com}"
REPO="${NOTATION_REPO:-health/canary}"
TAG="$(date +%s)"
IMAGE_REF="${REGISTRY}/${REPO}:${TAG}"

cleanup() {
  # Remove the test tag regardless of success or failure
  crane delete "${IMAGE_REF}" 2>/dev/null || true
}
trap cleanup EXIT

# Push a minimal scratch image for signing
crane cp gcr.io/google-containers/pause:3.9 "${IMAGE_REF}" \
  --insecure 2>/dev/null || {
  echo "FAIL: could not push canary image"
  exit 1
}

# Sign the image
notation sign "${IMAGE_REF}" \
  --key "${NOTATION_KEY_NAME}" \
  --signature-format cose \
  --timestamp-url "${NOTATION_TSA_URL:-}" \
  2>&1 | tail -1 | grep -q "Successfully signed" || {
  echo "FAIL: notation sign failed"
  exit 1
}

# Verify the image against the trust policy
notation verify "${IMAGE_REF}" 2>&1 | grep -q "Successfully verified" || {
  echo "FAIL: notation verify failed"
  exit 1
}

echo "OK: sign and verify roundtrip succeeded for ${IMAGE_REF}"

Expose this script as an HTTP health endpoint using a minimal server. The following Python wrapper runs the script on each GET request and returns 200 on success or 503 on failure:

# notation_health_server.py
import subprocess
import os
from http.server import BaseHTTPRequestHandler, HTTPServer

SCRIPT_PATH = os.environ.get("NOTATION_HEALTH_SCRIPT", "/opt/notation/notation-health-check.sh")

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return

        try:
            result = subprocess.run(
                ["bash", SCRIPT_PATH],
                capture_output=True,
                text=True,
                timeout=60,  # notation sign can be slow under KMS load
            )
            if result.returncode == 0:
                self.send_response(200)
                self.end_headers()
                self.wfile.write(b'{"status":"ok","check":"notation-roundtrip"}\n')
            else:
                self.send_response(503)
                self.end_headers()
                body = f'{{"status":"error","detail":{result.stderr!r}}}\n'
                self.wfile.write(body.encode())
        except subprocess.TimeoutExpired:
            self.send_response(503)
            self.end_headers()
            self.wfile.write(b'{"status":"error","detail":"notation sign timed out"}\n')

    def log_message(self, fmt, *args):
        pass  # suppress access log noise

if __name__ == "__main__":
    port = int(os.environ.get("PORT", "8080"))
    HTTPServer(("0.0.0.0", port), HealthHandler).serve_forever()

Run both in your signing environment (CI agent, sidecar, or dedicated signing pod) and expose the endpoint externally so Vigilmon can reach it.

Step 2: Monitor the Ratify admission webhook

Ratify runs as a Kubernetes Deployment behind a Service, registered as a ValidatingWebhookConfiguration. If the Ratify pods crash or the Service endpoint goes stale, the webhook silently fails open (with failurePolicy: Ignore) or blocks all deployments (with failurePolicy: Fail). Either outcome is critical.

Check the current state of your Ratify deployment:

# Verify Ratify pods are running
kubectl get pods -n gatekeeper-system -l app=ratify

# Verify the webhook service has endpoints
kubectl get endpoints ratify-webhook-service -n gatekeeper-system

# Verify the webhook configuration
kubectl get validatingwebhookconfiguration ratify-constraint

# Check the failure policy
kubectl get validatingwebhookconfiguration ratify-constraint \
  -o jsonpath='{.webhooks[0].failurePolicy}'

Deploy a lightweight health proxy that checks the Ratify deployment and the webhook certificate validity:

# ratify_health_proxy.py
from http.server import BaseHTTPRequestHandler, HTTPServer
from kubernetes import client, config
from datetime import datetime, timezone
import json
import os
import ssl

def load_k8s():
    if os.path.exists("/var/run/secrets/kubernetes.io/serviceaccount/token"):
        config.load_incluster_config()
    else:
        config.load_kube_config()

def check_ratify(namespace="gatekeeper-system"):
    load_k8s()
    v1 = client.CoreV1Api()
    apps = client.AppsV1Api()
    checks = {}

    # Check Ratify deployment readiness
    try:
        deploy = apps.read_namespaced_deployment("ratify", namespace)
        ready = deploy.status.ready_replicas or 0
        desired = deploy.spec.replicas or 1
        checks["ratify_pods"] = f"ok ({ready}/{desired} ready)" if ready >= 1 else f"error: {ready}/{desired} ready"
    except Exception as e:
        checks["ratify_pods"] = f"error: {str(e)[:120]}"

    # Check that the webhook service has at least one endpoint
    try:
        ep = v1.read_namespaced_endpoints("ratify-webhook-service", namespace)
        addresses = []
        for subset in (ep.subsets or []):
            addresses.extend(subset.addresses or [])
        checks["webhook_endpoints"] = f"ok ({len(addresses)} addresses)" if addresses else "error: no ready endpoints"
    except Exception as e:
        checks["webhook_endpoints"] = f"error: {str(e)[:120]}"

    # Check webhook serving certificate expiry via the webhook TLS secret
    try:
        secret = v1.read_namespaced_secret("ratify-tls", namespace)
        cert_pem = secret.data.get("tls.crt", "")
        if cert_pem:
            import base64
            import cryptography.x509 as x509
            from cryptography.hazmat.backends import default_backend
            cert_bytes = base64.b64decode(cert_pem)
            cert = x509.load_pem_x509_certificate(cert_bytes, default_backend())
            expiry = cert.not_valid_after_utc
            days_left = (expiry - datetime.now(timezone.utc)).days
            if days_left < 14:
                checks["webhook_cert"] = f"error: expires in {days_left} days"
            else:
                checks["webhook_cert"] = f"ok (expires in {days_left} days)"
        else:
            checks["webhook_cert"] = "error: tls.crt not found in secret"
    except Exception as e:
        checks["webhook_cert"] = f"error: {str(e)[:120]}"

    return checks

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return

        checks = check_ratify()
        errors = {k: v for k, v in checks.items() if str(v).startswith("error")}
        status_code = 503 if errors else 200
        body = json.dumps({
            "status": "error" if errors else "ok",
            "checks": checks,
        }).encode()
        self.send_response(status_code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        pass

if __name__ == "__main__":
    port = int(os.environ.get("PORT", "8081"))
    HTTPServer(("0.0.0.0", port), HealthHandler).serve_forever()

Deploy this proxy with a ServiceAccount that has get access to deployments, endpoints, and secrets in the Ratify namespace:

# ratify-monitor-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ratify-monitor
  namespace: gatekeeper-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ratify-monitor
  namespace: gatekeeper-system
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["endpoints", "secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ratify-monitor
  namespace: gatekeeper-system
subjects:
  - kind: ServiceAccount
    name: ratify-monitor
roleRef:
  kind: Role
  name: ratify-monitor
  apiGroup: rbac.authorization.k8s.io

Step 3: Check signing certificate expiry

notation stores signing certificates in trust stores under ~/.config/notation/truststore/ (Linux) or the platform equivalent. Certificate expiry is silent — notation will happily sign with an expired certificate in some configurations, and notation verify will reject every artifact signed with it once the certificate is expired.

The following script reads the leaf certificate from your notation trust store and returns 503 if expiry is within 30 days:

#!/usr/bin/env python3
# cert_expiry_check.py — expose as /health endpoint
import os
import glob
import json
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, HTTPServer
from cryptography import x509
from cryptography.hazmat.backends import default_backend

TRUSTSTORE_BASE = os.environ.get(
    "NOTATION_TRUSTSTORE",
    os.path.expanduser("~/.config/notation/truststore/x509/ca")
)
EXPIRY_WARN_DAYS = int(os.environ.get("EXPIRY_WARN_DAYS", "30"))

def check_certificates():
    results = {}
    cert_files = glob.glob(os.path.join(TRUSTSTORE_BASE, "**", "*.crt"), recursive=True)
    cert_files += glob.glob(os.path.join(TRUSTSTORE_BASE, "**", "*.pem"), recursive=True)

    if not cert_files:
        return {"error": f"no certificates found under {TRUSTSTORE_BASE}"}

    for cert_path in cert_files:
        label = os.path.relpath(cert_path, TRUSTSTORE_BASE)
        try:
            with open(cert_path, "rb") as f:
                cert_data = f.read()
            cert = x509.load_pem_x509_certificate(cert_data, default_backend())
            expiry = cert.not_valid_after_utc
            days_left = (expiry - datetime.now(timezone.utc)).days
            subject = cert.subject.rfc4514_string()

            if days_left < 0:
                results[label] = f"error: EXPIRED {abs(days_left)} days ago ({subject})"
            elif days_left < EXPIRY_WARN_DAYS:
                results[label] = f"error: expires in {days_left} days ({subject})"
            else:
                results[label] = f"ok: expires in {days_left} days"
        except Exception as e:
            results[label] = f"error: {str(e)[:120]}"

    return results

class CertHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return

        checks = check_certificates()
        errors = {k: v for k, v in checks.items() if str(v).startswith("error")}
        code = 503 if errors else 200
        body = json.dumps({
            "status": "error" if errors else "ok",
            "certificates": checks,
        }).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        pass

if __name__ == "__main__":
    port = int(os.environ.get("PORT", "8082"))
    HTTPServer(("0.0.0.0", port), CertHandler).serve_forever()

Run this on the same host or pod that holds your notation trust store. If you manage signing certificates centrally (e.g., via cert-manager or Vault PKI), the same approach works — adjust NOTATION_TRUSTSTORE to point at your certificate storage location.

Step 4: Monitor registry integration

Notary v2 stores signatures as OCI referrers attached to the signed manifest. If signature pushes silently fail — due to a registry auth misconfiguration, a quota error, or a missing referrers API implementation — signed images will show no signatures when Ratify queries the registry.

Test the referrers API directly against your registry. The OCI Distribution Spec v1.1 referrers API endpoint is:

GET /v2/<name>/referrers/<digest>?artifactType=application/vnd.cncf.notary.signature

# Set your registry and image details
REGISTRY="registry.example.com"
REPO="myapp/backend"
# Get the digest of a recently signed image
DIGEST=$(crane digest "${REGISTRY}/${REPO}:latest")

# Query the referrers API for Notary signatures
curl -s -u "${REGISTRY_USER}:${REGISTRY_PASS}" \
  "https://${REGISTRY}/v2/${REPO}/referrers/${DIGEST}?artifactType=application%2Fvnd.cncf.notary.signature" \
  | jq '.manifests | length'
# Should return 1 or more — 0 means no signatures found

Wrap this as a health probe:

#!/bin/bash
# registry-referrers-health.sh
set -euo pipefail

REGISTRY="${NOTATION_REGISTRY}"
REPO="${NOTATION_REPO}"
REF="${NOTATION_HEALTH_TAG:-latest}"

DIGEST=$(crane digest "${REGISTRY}/${REPO}:${REF}" 2>/dev/null) || {
  echo "FAIL: could not resolve digest for ${REGISTRY}/${REPO}:${REF}"
  exit 1
}

REFERRER_COUNT=$(curl -sf \
  -u "${REGISTRY_USER}:${REGISTRY_PASS}" \
  "https://${REGISTRY}/v2/${REPO}/referrers/${DIGEST}?artifactType=application%2Fvnd.cncf.notary.signature" \
  | jq '.manifests | length' 2>/dev/null) || {
  echo "FAIL: referrers API request failed"
  exit 1
}

if [ "${REFERRER_COUNT}" -ge 1 ]; then
  echo "OK: ${REFERRER_COUNT} Notary signature(s) found for ${DIGEST}"
else
  echo "FAIL: no Notary signatures found for ${DIGEST}"
  exit 1
fi

Expose this script through the same pattern used in Step 1 — a Python HTTP wrapper returns 200 when the script exits 0 and 503 otherwise. Point Vigilmon at this endpoint to detect registry integration failures before they affect production deployments.

Step 5: Add Vigilmon monitors

With your health endpoints running, add them to Vigilmon:

Signing pipeline roundtrip

Log in to vigilmon.online → Monitors → New Monitor
Choose HTTP / HTTPS
URL: https://your-signing-service.internal/health (the endpoint from Step 1)
Check interval: 1 minute — signing failures block CI immediately; you want fast detection
Expected response: status code 200, body contains "status":"ok"
Response time threshold: 90000ms — the notation roundtrip can take 60+ seconds under KMS load; set the threshold high enough to avoid false positives while still catching hangs
Assign your on-call alert channel
Save

Ratify admission webhook

Monitors → New Monitor → HTTP / HTTPS
URL: https://ratify-monitor.your-domain.com/health (the proxy from Step 2)
Check interval: 1 minute
Expected response: status code 200, body contains "status":"ok"
Save

Certificate expiry check

Monitors → New Monitor → HTTP / HTTPS
URL: https://your-cert-check-service.internal/health (the endpoint from Step 3)
Check interval: 5 minutes — certificate state changes slowly; 5 minutes is sufficient
Expected response: status code 200
Save

Registry referrers integration

Monitors → New Monitor → HTTP / HTTPS
URL: https://your-registry-health-service.internal/health (the endpoint from Step 4)
Check interval: 5 minutes
Expected response: status code 200, body contains "status":"ok"
Save

Recommended alert thresholds

| Monitor | Check interval | Alert condition | Priority | |---------|---------------|-----------------|----------| | notation sign/verify roundtrip | 1 minute | Any failure or timeout > 90s | P1 — blocks CI | | Ratify admission webhook | 1 minute | Any failure | P1 — security gate | | Signing certificate expiry | 5 minutes | < 30 days to expiry | P2 — plan rotation | | Signing certificate expiry | 5 minutes | < 7 days to expiry | P1 — rotate now | | Registry referrers API | 5 minutes | No signatures found | P2 | | TSA endpoint | 1 minute | HTTP non-200 | P2 |

For certificate expiry, configure two separate alert thresholds if your alerting system supports it: a low-urgency warning at 30 days out (time to schedule a rotation sprint) and a high-urgency alert at 7 days (rotate now or signatures will break). In Vigilmon, you can achieve the 7-day threshold by running a second cert-check endpoint that uses EXPIRY_WARN_DAYS=7.

For the Ratify webhook, set the failure policy to Fail in your ValidatingWebhookConfiguration and ensure your Vigilmon alerts are routed to a P1 channel. A failing admission webhook with failurePolicy: Ignore is operationally invisible — Vigilmon is what surfaces it:

# ValidatingWebhookConfiguration excerpt
webhooks:
  - name: validation.gatekeeper.sh
    failurePolicy: Fail   # do NOT use Ignore in production
    namespaceSelector:
      matchExpressions:
        - key: admission.gatekeeper.sh/ignore
          operator: DoesNotExist

Conclusion

Notary v2 and Ratify introduce a signing and verification pipeline with several independent failure points. The notation signing roundtrip, the Ratify admission webhook, the signing certificates, and the registry referrers API each need their own external health check — because none of these components will tell you proactively when they break.

The setup described here — a handful of HTTP endpoints backed by lightweight Python or bash scripts — gives you full external observability over the pipeline in under an hour. Once the endpoints are running, Vigilmon handles the rest: probing from multiple regions, multi-probe consensus to eliminate false positives, and alert routing to your existing channels.

Get started free at vigilmon.online — your first monitor is running in under two minutes, and the free tier covers the full set of endpoints described in this tutorial.