tutorial

Monitoring OCI Distribution (CNCF Distribution) with Vigilmon: Keeping Your Container Registry Healthy

How to monitor OCI Distribution (the reference implementation of the OCI Distribution Specification) with Vigilmon — tracking registry API health, storage backend availability, push and pull pipelines, and garbage collection from outside your infrastructure.

OCI Distribution — maintained by the CNCF as the reference implementation of the OCI Distribution Specification — is the open-source container registry that powers Docker Hub, GitHub Container Registry, Amazon ECR Public, and many self-hosted deployments. It implements the full OCI Distribution API: image push/pull, manifest storage, blob uploads, content-addressable layer storage, and garbage collection. Teams run self-hosted OCI Distribution instances as internal artifact stores, CI/CD artifact staging areas, and air-gapped registry mirrors. When OCI Distribution goes down — the registry process crashes, the storage backend becomes unavailable, or the filesystem fills up — every image push from CI and every pod start from Kubernetes fails simultaneously. Vigilmon gives you external monitoring that watches your OCI Distribution instance from outside the process and alerts before registry outages cascade into build and deployment failures.

What You'll Build

  • Vigilmon HTTP monitors for the OCI Distribution API endpoints
  • A push/pull validation heartbeat that verifies end-to-end registry operations work
  • A storage backend health probe
  • A garbage collection job monitor
  • Alert channels routed to your platform team

Prerequisites

  • A running OCI Distribution instance (v2 registry, typically distribution/distribution Docker image)
  • The registry exposed over HTTPS (required for Docker and containerd clients in production)
  • At least one storage backend configured (filesystem, S3, Azure Blob, or GCS)
  • A free account at vigilmon.online

Why Monitoring OCI Distribution Matters

OCI Distribution is a synchronous critical path dependency for your entire container infrastructure. Every component that starts, builds, or deploys containers depends on it:

Registry process unavailability is the most visible failure mode. When the registry process crashes or the API becomes unreachable, every Kubernetes pod using imagePullPolicy: Always fails to start, every CI job that pushes artifacts fails at the push step, and any developer docker pull returns a connection error. The blast radius is the entire container workload.

Storage backend failure is more dangerous than process failure because it may not immediately surface as a registry error. If the registry uses an S3 backend and the S3 bucket becomes inaccessible, the registry API may continue responding to manifest HEAD requests (served from cache) while all blob fetches return 500. Containers appear to be pulling but receive incomplete layers.

Filesystem exhaustion on filesystem-backend registries causes all writes to fail with 500 errors while reads continue to succeed. Push operations from CI fail; the registry appears healthy to read-only monitors.

Garbage collection (GC) lag causes the registry storage to grow unbounded. Unreferenced layers accumulate over time from CI builds that push new images but never clean up old ones. Eventually storage exhaustion triggers a write failure cascade.

Token authentication service failures affect registries using registry:2 token auth. The registry API returns 401 with a Www-Authenticate header pointing to the token service; if the token service is down, no client can authenticate and all operations fail.

External monitoring with Vigilmon catches these from the perspective of clients that depend on the registry.

Step 1: Monitor the OCI Distribution API Endpoint

OCI Distribution exposes a /v2/ base endpoint that all compliant clients probe first. This is the universal registry health signal:

# For anonymous/public registries — expect 200
curl -I https://registry.yourdomain.com/v2/
# Expected: HTTP/2 200

# For authenticated registries — expect 401 with Www-Authenticate header
curl -I https://registry.yourdomain.com/v2/
# Expected: HTTP/2 401
# Www-Authenticate: Bearer realm="https://registry.yourdomain.com/auth/token",service="registry.yourdomain.com"

Both 200 and 401 confirm the registry process is alive. A 503, connection refused, or timeout means the registry is down.

In the Vigilmon dashboard:

  1. Go to Add Monitor → HTTP
  2. Configure:

| Field | Value | |---|---| | Name | OCI Distribution /v2/ | | URL | https://registry.yourdomain.com/v2/ | | Method | GET | | Expected status | 200, 401 | | Check interval | 30 seconds | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

Use 30-second intervals for the /v2/ check — registry outages have an immediate and large blast radius, so faster detection pays off.

Step 2: Monitor the Registry Debug/Health Endpoint

OCI Distribution ships with a debug server that exposes a health endpoint. Enable it in your registry configuration:

# config.yml snippet
http:
  addr: :5000
  debug:
    addr: :5001
    prometheus:
      enabled: true
      path: /metrics
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

Then monitor the health endpoint:

curl http://registry.yourdomain.com:5001/debug/health
# Expected: {}  (empty JSON = healthy)

In Vigilmon:

| Field | Value | |---|---| | Name | OCI Distribution /debug/health | | URL | http://registry.yourdomain.com:5001/debug/health | | Method | GET | | Expected status | 200 | | Response body contains | {} | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

The health endpoint checks storage backend connectivity — if storage is unhealthy, the health endpoint returns a non-empty JSON body with the error details.

Step 3: Monitor Prometheus Metrics Endpoint

If you've enabled Prometheus metrics (common in production deployments), add a metrics monitor:

curl http://registry.yourdomain.com:5001/metrics | head -20
# Expected: Prometheus text format with registry_* metrics

In Vigilmon:

| Field | Value | |---|---| | Name | OCI Distribution /metrics | | URL | http://registry.yourdomain.com:5001/metrics | | Method | GET | | Expected status | 200 | | Response body contains | registry_ | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

Step 4: End-to-End Push/Pull Validation Heartbeat

The /v2/ check confirms the API is alive but doesn't validate that push and pull actually work end-to-end. Add a CronJob that performs a real push and pull cycle:

# oci-distribution-e2e-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: oci-distribution-e2e-probe
  namespace: monitoring
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: e2e-probe
              image: alpine:latest
              env:
                - name: REGISTRY
                  value: "registry.yourdomain.com"
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: oci-e2e-heartbeat-url
                - name: REGISTRY_USER
                  valueFrom:
                    secretKeyRef:
                      name: registry-credentials
                      key: username
                - name: REGISTRY_PASS
                  valueFrom:
                    secretKeyRef:
                      name: registry-credentials
                      key: password
              command:
                - /bin/sh
                - -c
                - |
                  apk add --quiet curl skopeo

                  PROBE_TAG="monitoring/probe:$(date +%s)"

                  # Push a minimal image manifest directly via the OCI Distribution API
                  # Get auth token first
                  TOKEN=$(curl -sf -u "$REGISTRY_USER:$REGISTRY_PASS" \
                    "https://$REGISTRY/auth/token?service=$REGISTRY&scope=repository:monitoring/probe:push,pull" \
                    | grep -o '"token":"[^"]*"' | cut -d'"' -f4)

                  if [ -z "$TOKEN" ]; then
                    echo "ERROR: Failed to obtain registry token" >&2
                    exit 1
                  fi

                  # Push an empty blob (minimal test payload)
                  EMPTY_BLOB='sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
                  UPLOAD_URL=$(curl -sf -D - \
                    -H "Authorization: Bearer $TOKEN" \
                    -X POST \
                    "https://$REGISTRY/v2/monitoring/probe/blobs/uploads/" \
                    | grep -i "^location:" | tr -d '\r' | cut -d' ' -f2)

                  if [ -z "$UPLOAD_URL" ]; then
                    echo "ERROR: Failed to initiate blob upload" >&2
                    exit 1
                  fi

                  # Check the tag list endpoint to confirm pull access works
                  TAG_RESPONSE=$(curl -sf \
                    -H "Authorization: Bearer $TOKEN" \
                    "https://$REGISTRY/v2/monitoring/probe/tags/list" || echo "empty")

                  echo "E2E registry probe completed: push initiation OK, tag list accessible"
                  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null
                  echo "E2E heartbeat sent"

For a simpler approach using skopeo:

#!/bin/bash
# oci-distribution-skopeo-probe.sh
set -e

REGISTRY="registry.yourdomain.com"
HEARTBEAT_URL="${VIGILMON_OCI_E2E_HEARTBEAT_URL}"

# Copy a minimal public image into the registry as a probe
if skopeo copy \
    --dest-creds "$REGISTRY_USER:$REGISTRY_PASS" \
    docker://alpine:latest \
    "docker://$REGISTRY/monitoring/probe:latest" 2>&1; then

    echo "Push probe succeeded"

    # Verify pull works
    if skopeo inspect \
        --creds "$REGISTRY_USER:$REGISTRY_PASS" \
        "docker://$REGISTRY/monitoring/probe:latest" > /dev/null 2>&1; then

        curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null
        echo "Pull probe succeeded, heartbeat sent"
    else
        echo "ERROR: Pull probe failed after successful push" >&2
        exit 1
    fi
else
    echo "ERROR: Push probe failed" >&2
    exit 1
fi

Create the heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat
  2. Name: OCI Distribution end-to-end push/pull
  3. Expected interval: 5 minutes
  4. Grace period: 5 minutes

Step 5: Monitor Garbage Collection Jobs

OCI Distribution GC must run periodically to reclaim storage from deleted images. Monitor that GC is completing on schedule:

#!/bin/bash
# registry-gc-probe.sh
set -e

HEARTBEAT_URL="${VIGILMON_OCI_GC_HEARTBEAT_URL}"
REGISTRY_CONFIG="/etc/docker/registry/config.yml"
STORAGE_DIR="/var/lib/registry"

# Run garbage collection
registry garbage-collect "$REGISTRY_CONFIG" --delete-untagged 2>&1

# Check that the storage directory is accessible and not full
USAGE=$(df "$STORAGE_DIR" | awk 'NR==2 {print $5}' | tr -d '%')
THRESHOLD=90

if [ "$USAGE" -gt "$THRESHOLD" ]; then
    echo "WARN: Registry storage at ${USAGE}% (threshold: ${THRESHOLD}%) after GC" >&2
    exit 1
fi

echo "GC complete, storage at ${USAGE}%"
curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null
echo "GC heartbeat sent"

Deploy as a Kubernetes CronJob that runs daily (adjust to your GC schedule):

# registry-gc-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: registry-gc
  namespace: registry
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: gc
              image: registry:2
              command:
                - /bin/sh
                - -c
                - |
                  /entrypoint.sh garbage-collect /etc/docker/registry/config.yml --delete-untagged
                  wget -qO- "$VIGILMON_GC_HEARTBEAT_URL"
              env:
                - name: VIGILMON_GC_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: oci-gc-heartbeat-url
              volumeMounts:
                - name: registry-storage
                  mountPath: /var/lib/registry
                - name: registry-config
                  mountPath: /etc/docker/registry
          volumes:
            - name: registry-storage
              persistentVolumeClaim:
                claimName: registry-pvc
            - name: registry-config
              configMap:
                name: registry-config

Create the heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat
  2. Name: OCI Distribution garbage collection
  3. Expected interval: 24 hours
  4. Grace period: 2 hours

Step 6: Configure Alert Channels

Route OCI Distribution alerts by severity:

For the /v2/ API monitor (highest severity — entire container infrastructure blocked):

  1. In Vigilmon: Alert Channels → Add Channel → PagerDuty
  2. Add your infrastructure on-call rotation
  3. Assign to OCI Distribution /v2/ — this is a P1 incident

For storage and GC monitors (important but not immediate):

  1. Alert Channels → Add Channel → Slack Webhook
  2. Add your #infrastructure or #platform-engineering webhook
  3. Assign to health, metrics, GC, and E2E heartbeat monitors

For the end-to-end push/pull heartbeat (CI impact):

  1. Alert Channels → Add Channel → Slack Webhook
  2. Add your #ci-cd-alerts webhook
  3. Assign to the OCI Distribution end-to-end push/pull heartbeat

Create the Kubernetes secrets:

kubectl create secret generic vigilmon-secrets \
  --from-literal=oci-e2e-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_E2E_TOKEN' \
  --from-literal=oci-gc-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_GC_TOKEN' \
  -n monitoring

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | /v2/ API | HTTP GET (30s) | Registry process down — P1 for all container workloads | | /debug/health | HTTP GET (1 min) | Storage backend failure, internal health degraded | | /metrics | HTTP GET (1 min) | Metrics server and internal instrumentation unavailable | | End-to-end push/pull | Heartbeat (5 min) | Push or pull broken at application layer | | Garbage collection | Heartbeat (24 hr) | GC not running, storage growing unbounded |

Conclusion

OCI Distribution is the foundation of your container artifact supply chain — when it fails, everything fails simultaneously. The /v2/ endpoint is the single most important signal (it reflects the registry process status immediately) but it misses the subtler failures: storage backend unavailability, filesystem exhaustion, and GC lag. The combination of API probes, end-to-end push/pull validation, and GC heartbeats gives you complete visibility into the registry's health as a system, not just as a process.

Get started free at vigilmon.online. The /v2/ HTTP probe and end-to-end push/pull heartbeat together give you the two most critical OCI Distribution signals in under ten minutes: is the API alive, and can clients actually push and pull images?

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles