tutorial

Monitoring Zot Registry with Vigilmon

Zot is the only OCI-native container registry — built from scratch on the OCI Distribution Spec without Docker Registry compatibility overhead. But push/pull failures, storage backend pressure, replication sync lag, and garbage collection stalls can silently degrade your artifact supply chain. Here's how to monitor Zot with Vigilmon.

Zot is an open-source container registry built entirely on the OCI Distribution Specification, with no Docker Registry v2 compatibility layer and no non-OCI baggage. It's the only registry that's both OCI-native and CNCF-conformant, making it popular for air-gapped environments, GitOps pipelines, and supply-chain-secure container deployments. Zot supports pluggable storage backends (local disk, S3), image replication to peer registries, and online garbage collection — but when any of these subsystems stalls, the failure surface is broad: push operations return 500 errors, pulls serve stale layers, replication drifts silently, and garbage collection eats disk I/O without finishing. Vigilmon gives you external monitoring for Zot API health, push/pull throughput, storage backend health, replication sync status, and garbage collection metrics so you catch registry failures before they block CI/CD pipelines.

What You'll Set Up

  • Zot API health and readiness via HTTP monitors
  • Push and pull throughput tracking
  • Storage backend health monitoring
  • Replication sync status checks
  • Garbage collection activity monitoring

Prerequisites

  • Zot v2.0+ running (via binary, Docker, or Kubernetes)
  • Zot API accessible at your registry URL (default port 5000)
  • zli CLI or oras CLI available for push/pull tests
  • A free Vigilmon account

Step 1: Monitor Zot API Health and Readiness

Zot exposes a health endpoint at /_zot/metrics (Prometheus format) and a standard OCI Distribution Spec health check at /v2/. The /v2/ endpoint returns 200 OK with an empty JSON object when the registry is ready to serve requests. Use an HTTP monitor for immediate alerting on registry downtime:

  1. In Vigilmon, click Add Monitor → HTTP Monitor.
  2. Set the URL to https://your-zot-registry.example.com/v2/.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Add an Authorization header if your registry requires authentication: Basic <base64(user:password)>.

For a more thorough health check via cron heartbeat:

#!/bin/bash
# /usr/local/bin/zot-health-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
ZOT_URL="${ZOT_URL:-http://localhost:5000}"
ZOT_AUTH="${ZOT_AUTH:-}"  # Set to "user:password" if auth required

AUTH_ARGS=""
if [ -n "$ZOT_AUTH" ]; then
  AUTH_ARGS="-u $ZOT_AUTH"
fi

# Check the OCI distribution spec v2 endpoint
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" $AUTH_ARGS \
  "${ZOT_URL}/v2/")

if [ "$HTTP_STATUS" = "200" ] || [ "$HTTP_STATUS" = "401" ]; then
  # 401 means auth is required but the registry is up
  echo "Zot API healthy: HTTP $HTTP_STATUS"
  curl -s "$HEARTBEAT_URL"
else
  echo "Zot API unhealthy: HTTP $HTTP_STATUS"
  exit 1
fi

Install and enable:

chmod +x /usr/local/bin/zot-health-check.sh
(crontab -l 2>/dev/null; echo "*/1 * * * * /usr/local/bin/zot-health-check.sh >> /var/log/zot-health.log 2>&1") | crontab -

The HTTP monitor catches the most common failure modes: Zot process crash, port binding failure, and TLS certificate expiry. The cron heartbeat adds authentication and response body validation.

Step 2: Track Push and Pull Throughput

Push and pull are Zot's primary workload. Slow pushes block CI artifact publishing; slow pulls delay container startup in your deployment pipeline. Monitor the full push/pull round-trip with a test image:

#!/bin/bash
# /usr/local/bin/zot-push-pull-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
ZOT_URL="${ZOT_URL:-http://localhost:5000}"
ZOT_USER="${ZOT_USER:-admin}"
ZOT_PASS="${ZOT_PASS:-admin}"
TEST_REPO="vigilmon/healthcheck"
TEST_TAG="probe-$(date +%s)"
PUSH_THRESHOLD=15  # seconds
PULL_THRESHOLD=10  # seconds

# Requires 'oras' CLI: https://oras.land/
if ! command -v oras >/dev/null 2>&1; then
  echo "oras CLI not found — install from https://oras.land/"
  exit 1
fi

# Create a small test artifact
TEST_FILE="/tmp/zot-probe-$$"
echo "vigilmon-probe-$(date -u +%Y-%m-%dT%H:%M:%SZ)" > "$TEST_FILE"

# Login to registry
echo "$ZOT_PASS" | oras login --username "$ZOT_USER" --password-stdin \
  "${ZOT_URL#*//}" >/dev/null 2>&1

# Measure push latency
START=$(date +%s)
if oras push --no-tls "${ZOT_URL#*//}/${TEST_REPO}:${TEST_TAG}" \
    "$TEST_FILE:text/plain" >/dev/null 2>&1; then
  END=$(date +%s)
  PUSH_ELAPSED=$(( END - START ))
else
  echo "Push failed for ${TEST_REPO}:${TEST_TAG}"
  rm -f "$TEST_FILE"
  exit 1
fi

# Measure pull latency
PULL_DIR="/tmp/zot-pull-$$"
mkdir -p "$PULL_DIR"
START=$(date +%s)
if oras pull --no-tls --output "$PULL_DIR" \
    "${ZOT_URL#*//}/${TEST_REPO}:${TEST_TAG}" >/dev/null 2>&1; then
  END=$(date +%s)
  PULL_ELAPSED=$(( END - START ))
else
  echo "Pull failed for ${TEST_REPO}:${TEST_TAG}"
  rm -rf "$TEST_FILE" "$PULL_DIR"
  exit 1
fi

echo "Push: ${PUSH_ELAPSED}s, Pull: ${PULL_ELAPSED}s"
rm -f "$TEST_FILE"
rm -rf "$PULL_DIR"

if [ "$PUSH_ELAPSED" -lt "$PUSH_THRESHOLD" ] && \
   [ "$PULL_ELAPSED" -lt "$PULL_THRESHOLD" ]; then
  curl -s "$HEARTBEAT_URL"
else
  echo "Throughput degraded: push=${PUSH_ELAPSED}s pull=${PULL_ELAPSED}s"
  exit 1
fi

Schedule this every 10 minutes with a matching Vigilmon heartbeat interval. Slow pushes above 15 seconds for a tiny artifact indicate storage backend I/O saturation or blob deduplication taking longer than expected on a large layer store. Slow pulls often indicate the manifest and layer cache is cold, triggering storage reads that compete with ongoing pushes.

Step 3: Monitor Storage Backend Health

Zot's storage backend is its most critical dependency. Whether using local disk or S3-compatible object storage, storage failures cause silent data corruption, failed uploads, and incomplete pulls. Monitor storage health separately from API health:

#!/bin/bash
# /usr/local/bin/zot-storage-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
ZOT_STORAGE_ROOT="${ZOT_STORAGE:-/var/lib/registry}"
DISK_THRESHOLD=80  # percent — alert before Zot hits its own configurable limit
INODE_THRESHOLD=85  # percent

# Check disk usage at the storage root
DISK_USED=$(df "$ZOT_STORAGE_ROOT" 2>/dev/null | awk 'NR==2 {gsub(/%/, ""); print $5}')

if [ -z "$DISK_USED" ]; then
  echo "Storage root not accessible: $ZOT_STORAGE_ROOT"
  exit 1
fi

if [ "$DISK_USED" -ge "$DISK_THRESHOLD" ]; then
  echo "Storage critical: ${DISK_USED}% used at $ZOT_STORAGE_ROOT"
  exit 1
fi

# Check inode exhaustion — many small OCI layer blobs can exhaust inodes
# before disk bytes are full
INODE_USED=$(df -i "$ZOT_STORAGE_ROOT" 2>/dev/null | awk 'NR==2 {gsub(/%/, ""); print $5}')

if [ -n "$INODE_USED" ] && [ "$INODE_USED" -ge "$INODE_THRESHOLD" ]; then
  echo "Inode exhaustion: ${INODE_USED}% used at $ZOT_STORAGE_ROOT"
  exit 1
fi

# Verify write access to the storage root
TEST_FILE="$ZOT_STORAGE_ROOT/.vigilmon-write-test-$$"
if touch "$TEST_FILE" 2>/dev/null; then
  rm -f "$TEST_FILE"
else
  echo "Storage write test failed — filesystem may be read-only"
  exit 1
fi

# Count blobs to verify the repository structure is intact
BLOB_DIR="$ZOT_STORAGE_ROOT"
if [ -d "$BLOB_DIR" ]; then
  REPO_COUNT=$(find "$BLOB_DIR" -maxdepth 1 -type d 2>/dev/null | tail -n +2 | wc -l)
  echo "Storage OK: ${DISK_USED}% disk, ${INODE_USED:-N/A}% inodes, ${REPO_COUNT} repos"
fi

curl -s "$HEARTBEAT_URL"

Set the Vigilmon heartbeat to 10 minutes. Inode exhaustion is a particularly common failure mode for Zot because each OCI layer blob is a separate file — a registry with 500 images and 10 layers each creates 5,000+ files before manifests and index files. Monitor inodes separately from bytes.

If you use S3 as a backend, add an S3 reachability check:

# Check S3 backend connectivity (requires aws CLI)
aws s3 ls "s3://${ZOT_S3_BUCKET}/" --region "${ZOT_S3_REGION}" \
  >/dev/null 2>&1 && echo "S3 backend OK" || echo "S3 backend unreachable"

Step 4: Monitor Replication Sync Status

Zot supports OCI image replication to peer registries via its sync extension. Replication lag means your disaster recovery registry is serving stale images, and geo-distributed pulls are serving stale layers. Monitor replication health via Zot's metrics endpoint:

#!/bin/bash
# /usr/local/bin/zot-replication-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
ZOT_URL="${ZOT_URL:-http://localhost:5000}"
ZOT_AUTH="${ZOT_AUTH:-}"
MAX_SYNC_LAG_MINUTES=30  # Alert if last sync was more than 30 minutes ago

AUTH_ARGS=""
if [ -n "$ZOT_AUTH" ]; then
  AUTH_ARGS="-u $ZOT_AUTH"
fi

# Check Zot metrics for sync error counters
METRICS=$(curl -s $AUTH_ARGS "${ZOT_URL}/_zot/metrics" 2>/dev/null)

if [ -z "$METRICS" ]; then
  echo "Metrics endpoint unreachable — is _zot.extensions.metrics enabled?"
  # Don't fail the heartbeat if metrics aren't configured
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

# Check for sync error counts
SYNC_ERRORS=$(echo "$METRICS" | grep "^zot_sync_errors_total" | \
  awk '{sum+=$2} END {print sum+0}')

if [ "${SYNC_ERRORS:-0}" -gt 0 ]; then
  echo "Replication errors detected: sync_errors_total=$SYNC_ERRORS"
  exit 1
fi

# Check for sync repo errors specifically
REPO_ERRORS=$(echo "$METRICS" | grep "sync_repo" | grep -v "^#" | \
  awk '{sum+=$2} END {print sum+0}')

echo "Replication status: errors=$SYNC_ERRORS repo_errors=${REPO_ERRORS:-0}"
curl -s "$HEARTBEAT_URL"

Also configure a Vigilmon HTTP monitor directly against your replica registry:

  1. Add an HTTP monitor for https://your-replica-registry.example.com/v2/.
  2. Set Expected HTTP status to 200.
  3. Set Check interval to 2 minutes.

If the replica goes down while the primary is up, you'll get separate alerts for each — giving you clear signal about whether the failure is primary vs. replica.

Step 5: Monitor Garbage Collection Activity

Zot supports online garbage collection that runs in the background to reclaim storage from deleted manifests and unreferenced blobs. GC that runs continuously and never finishes (due to a large blob store or slow storage) causes I/O saturation that degrades push/pull performance. GC that never runs causes unbounded disk growth. Monitor GC health via metrics:

#!/bin/bash
# /usr/local/bin/zot-gc-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
ZOT_URL="${ZOT_URL:-http://localhost:5000}"
ZOT_STORAGE="${ZOT_STORAGE:-/var/lib/registry}"
ZOT_AUTH="${ZOT_AUTH:-}"
GC_LOG="/var/log/zot-gc.log"

AUTH_ARGS=""
if [ -n "$ZOT_AUTH" ]; then
  AUTH_ARGS="-u $ZOT_AUTH"
fi

# Query metrics for GC-related counters
METRICS=$(curl -s $AUTH_ARGS "${ZOT_URL}/_zot/metrics" 2>/dev/null)

if [ -n "$METRICS" ]; then
  # Extract GC run count and duration metrics
  GC_RUNS=$(echo "$METRICS" | grep "^zot_gc_runs_total" | \
    awk '{print $2+0}' | head -1)
  GC_ERRORS=$(echo "$METRICS" | grep "^zot_gc_errors_total" | \
    awk '{print $2+0}' | head -1)
  GC_BLOBS=$(echo "$METRICS" | grep "^zot_gc_blobs_deleted_total" | \
    awk '{print $2+0}' | head -1)

  echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) gc_runs=${GC_RUNS:-0} gc_errors=${GC_ERRORS:-0} blobs_deleted=${GC_BLOBS:-0}" >> "$GC_LOG"

  if [ "${GC_ERRORS:-0}" -gt 0 ]; then
    echo "GC errors detected: ${GC_ERRORS} errors in ${GC_RUNS:-0} runs"
    exit 1
  fi

  echo "GC OK: runs=${GC_RUNS:-0}, blobs_deleted=${GC_BLOBS:-0}"
fi

# Independent storage size check to detect GC stagnation
STORAGE_MB=$(du -sm "$ZOT_STORAGE" 2>/dev/null | awk '{print $1}')
if [ -n "$STORAGE_MB" ]; then
  echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) storage_mb=$STORAGE_MB" >> "$GC_LOG"
fi

curl -s "$HEARTBEAT_URL"

Set the Vigilmon heartbeat to 30 minutes. Review the GC log weekly — a storage size that only grows and never shrinks after deletions means GC isn't running or is failing silently. A storage size that fluctuates with deletions and GC runs means GC is healthy.

To trigger a manual GC run via the Zot API when needed:

# Trigger GC via the Zot management API (requires admin credentials)
curl -s -X POST "${ZOT_URL}/_zot/forcegc" \
  -H "Authorization: Bearer YOUR_TOKEN"

Step 6: Set Up Alert Channels

Configure Vigilmon to route Zot registry alerts appropriately:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #platform-alerts or #registry-ops.
  3. Add PagerDuty for the API health and push/pull monitors — registry downtime blocks all CI/CD pipelines immediately.
  4. Set Consecutive failures before alert to 1 for the API health monitor — there's no valid reason for the registry API to be down briefly.
  5. Set Consecutive failures before alert to 2 for throughput and GC monitors — transient I/O spikes are normal.

Add maintenance windows for Zot upgrades and GC-intensive operations:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "MONITOR_ID",
    "duration_minutes": 20,
    "reason": "zot registry upgrade and GC run"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor | /v2/ endpoint | Registry crash, port binding failure, TLS expiry | | Cron heartbeat (push/pull) | Timed ORAS push + pull cycle | Throughput degradation, storage backend I/O saturation | | Cron heartbeat (storage) | Disk + inode usage + write test | Disk full, inode exhaustion, read-only filesystem | | Cron heartbeat (replication) | Metrics sync error counters + replica HTTP | Replication lag, sync errors, replica downtime | | Cron heartbeat (GC) | GC metrics + storage size trend | GC stagnation, GC errors, unbounded disk growth |

Zot's OCI-native design makes it lean and auditable, but its storage backend, replication engine, and garbage collector each have failure modes that don't surface as API errors until they've already degraded your pipeline. Vigilmon's external checks give you visibility into the full operational surface — so you know about registry health problems before your CI/CD pipeline starts failing pushes.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles