Monitoring Notation with Vigilmon: Image Signing Health, OCI Registry Verification, and Supply Chain Alert Coverage

Notation is the CNCF standard for signing and verifying container images and OCI artifacts — the layer that lets you prove that an image in your registry is exactly what your CI pipeline produced and hasn't been tampered with. When Notation's signing infrastructure is unavailable, CI pipelines can't sign new images. When verification policies are misconfigured, admission controllers reject every new pod. When the signing certificate expires, every notation verify call fails and deployments halt. Vigilmon gives you external visibility into the endpoints and services that underpin Notation-based supply chain security: your OCI registry, Notation policy endpoints, certificate authority health, and your trust policy verification workflow.

What You'll Build

• A monitor on your OCI registry API endpoint that Notation uses for signature storage
• An HTTP monitor for your Notation-compatible Timestamping Authority (TSA) if you use RFC 3161 timestamps
• SSL certificate monitors for your signing certificates and registry domains
• A monitor for any Notation policy service or verification API you expose internally
• An alerting setup that fires before signing infrastructure issues break deployments

Prerequisites

• A Notation-configured OCI registry (e.g., Azure Container Registry, AWS ECR, or self-hosted Harbor/Zot)
• Optionally, a Timestamping Authority endpoint for RFC 3161 timestamps
• A free account at vigilmon.online

Step 1: Verify Your OCI Registry's Notation-Relevant Endpoints

Notation stores signatures as OCI artifacts alongside the image in the registry. The registry API endpoints that Notation uses are the standard OCI Distribution Spec endpoints:

# Check that the registry API is responding
curl -I https://registry.example.com/v2/

# List referrers (signatures) for a specific image digest
curl https://registry.example.com/v2/myapp/referrers/sha256:<digest>?artifactType=application/vnd.cncf.notary.signature

A healthy registry returns 200 (or 401 for unauthenticated access to /v2/). A 200 from the referrers endpoint with a JSON manifest list confirms signatures are accessible.

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.cncf.notary.signature",
      "artifactType": "application/vnd.cncf.notary.signature",
      "digest": "sha256:...",
      "size": 1234
    }
  ]
}

Step 2: Monitor the OCI Registry API

The registry is the foundation of Notation's supply chain — it stores both images and their associated signatures. If the registry is unreachable, Notation cannot sign, verify, or distribute any artifacts:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://registry.example.com/v2/.
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 401 (unauthenticated access returns 401, confirming the API is up and requiring auth).
6. Label: OCI Registry API (Notation).
7. Click Save.

For public registries or those with anonymous read enabled, the expected status is 200. Check your registry's anonymous access settings to pick the right expected status.

This monitor catches:

• Registry process crashes that prevent all notation sign and notation verify operations
• Network partition between your Kubernetes cluster and the registry
• Registry backend storage failures that cause API errors
• Deployment failures after registry upgrades

Alert sensitivity: Set alerts after 1 consecutive failure. Registry unavailability means Notation verification in your admission policy will start rejecting all new pods if you enforce it strictly.

Step 3: Monitor the Timestamping Authority (TSA)

Notation supports RFC 3161 timestamping to prove that a signature was created before a certificate expired — this extends the validity of signatures beyond the certificate's lifetime. If your signing workflow uses a TSA, its availability is critical for sign operations:

# Verify your TSA is responding (most TSAs accept GET requests)
curl -I https://tsa.example.com/api/v1/timestamp

1. Add Monitor → HTTP.
2. URL: https://tsa.example.com/api/v1/timestamp (adjust to your TSA's health endpoint).
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200 or 405 (some TSAs return Method Not Allowed on GET but confirm the service is up).
6. Label: Notation TSA (timestamping authority).
7. Click Save.

Popular public TSAs you might monitor:

• DigiCert: http://timestamp.digicert.com
• Sectigo: http://timestamp.sectigo.com
• Sigstore Rekor (also acts as a transparency log): https://rekor.sigstore.dev/api/v1/log

Step 4: Monitor SSL Certificates

Notation relies on X.509 certificates at multiple levels:

• Signing certificate: Used to sign image digests. If expired, notation sign fails.
• Registry TLS certificate: If expired, Notation can't connect to the OCI registry.
• TSA certificate: If expired, timestamping fails and signatures without trusted timestamps may be rejected.

Add SSL certificate monitors for each domain involved in your Notation workflow:

Registry SSL Certificate

1. Add Monitor → SSL Certificate.
2. Domain: registry.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days.
5. Click Save.

TSA SSL Certificate (if using a private TSA)

1. Add Monitor → SSL Certificate.
2. Domain: tsa.example.com.
3. Alert when expiry is within: 60 days (TSA certs often need broader lead time for rotation).
4. Click Save.

Signing certificate rotation: The X.509 certificates stored in your Notation trust store (used to verify signatures) have their own expiry separate from TLS certificates. Use Vigilmon's SSL monitor on any HTTPS endpoint that serves your signing CA's CRL or OCSP responder to catch CA infrastructure issues early.

Step 5: Monitor a Notation Policy Verification Endpoint

Many teams expose an internal HTTP endpoint that wraps notation verify — used by webhook admission controllers or CI gates to verify image signatures before promoting artifacts. If you expose such an endpoint, monitor it:

# Example: internal verification proxy
curl -X POST https://verify.internal.example.com/verify \
  -H 'Content-Type: application/json' \
  -d '{"image": "registry.example.com/myapp:latest"}'

1. Add Monitor → HTTP.
2. URL: https://verify.internal.example.com/health.
3. Check interval: 60 seconds.
4. Expected status: 200.
5. Keyword: ok or healthy.
6. Label: Notation verification proxy.
7. Click Save.

This catches cases where the verification service itself crashes or loses connectivity to the registry, even while the registry API remains up.

Step 6: Test the Full Signing Workflow

Verify that Notation can complete a round-trip sign and verify against your registry:

# Sign an image
notation sign registry.example.com/myapp@sha256:<digest> \
  --key my-signing-key

# Verify the signature
notation verify registry.example.com/myapp@sha256:<digest>

# List signatures stored in the registry
notation list registry.example.com/myapp@sha256:<digest>

If notation verify exits 0, the registry, trust store, and certificate chain are all healthy. If it fails, check:

1. Registry connectivity: curl -I https://registry.example.com/v2/
2. Certificate validity: notation certificate show --type ca --name my-ca
3. Trust policy configuration: cat ~/.config/notation/trustpolicy.json

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | OCI Registry API | Non-200/401 response | Registry down; all sign/verify ops fail; check registry process | | TSA endpoint | Unreachable or non-200 | Timestamping fails; notation sign --timestamp-url calls fail | | Registry SSL | < 30 days to expiry | Renew registry TLS cert before notation clients reject connections | | TSA SSL | < 60 days to expiry | Coordinate TSA cert rotation with TSA provider | | Verification proxy | Non-200 or keyword missing | Admission controller may start rejecting all pods; check proxy logs |

Alert channels: Configure both email and a Slack/webhook channel for certificate expiry alerts — these require human coordination with certificate authorities and can't be auto-remediated.

Common Notation Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Registry downtime | OCI registry API monitor fires; all sign/verify ops blocked | | Signing certificate expires | SSL monitor fires 30 days early; notation sign starts failing | | TSA unreachable | TSA monitor fires; signing with timestamps fails, CI pipelines break | | Registry TLS cert expires | Registry SSL monitor fires; all Notation clients reject TLS connections | | Trust policy misconfigured | Verification proxy monitor catches rejected verifications | | Network partition to registry | Registry API monitor fires; signature fetches fail in Kubernetes nodes | | OCI referrers API unavailable | Registry API response indicates API regression after upgrade | | Signing key rotated without trust store update | Verification proxy returns non-200 for newly signed images |

Notation is the enforcement layer for container image integrity — when it's healthy, every image running in production is provably from your CI pipeline. When its infrastructure fails, either deployments halt (strict enforcement) or unsigned images slip through (lenient mode). Vigilmon gives you external monitoring of every component Notation depends on: the OCI registry, the timestamping authority, SSL certificates, and your verification endpoints — so you catch infrastructure drift before it becomes a supply chain incident.

Start monitoring your Notation supply chain in under 5 minutes — register free at vigilmon.online.