How to Monitor Skopeo Container Image Sync and Registry Operations with Vigilmon

Skopeo is the Swiss Army knife of container image management — copying images between registries, inspecting manifests without pulling layers, and synchronizing entire repositories for air-gapped deployments. When Skopeo is at the center of your image supply chain (mirroring upstream images to an internal registry, syncing across regions, or managing multi-arch manifests), its health directly affects every team that pulls images for builds or deployments.

In this tutorial you'll set up monitoring for your Skopeo-powered image pipelines using Vigilmon — free tier, no credit card required.

Why Skopeo image operations need external monitoring

Skopeo operations are often run as scheduled jobs — cron tasks that mirror DockerHub to an internal registry, sync production images to DR sites, or rotate auth tokens before they expire. These silent background jobs are easy to forget about until a deployment fails because the image isn't in the mirror.

Key failure modes that CI dashboards miss:

Registry sync stall — a Skopeo sync job hangs on a large image layer and the cron job's next run doesn't fire because the previous one never exited
Auth token expiry — ECR tokens expire every 12 hours; when the rotation job fails, all subsequent Skopeo operations fail with 401s silently logged to a file no one reads
Mirror lag — upstream images are published but the mirror hasn't synced yet; deployments pull stale images without knowing
Cross-registry copy latency — image copies between geographically distant registries take too long and timeout; partial layers are left in the destination registry in an unusable state
Manifest list corruption — skopeo copy --all copies multi-arch manifests but the destination registry may not support manifest lists; images appear present but fail on ARM nodes
Rate limit from upstream registries — DockerHub and similar registries enforce pull limits; anonymous Skopeo syncs hit the limit and return 429s, silently breaking the mirror

External monitoring catches these by watching the mirror freshness, copy job health, and registry availability from outside your internal network.

What you'll need

Skopeo installed (skopeo ≥ 1.10)
At least one registry mirror or sync job running on a schedule
A free Vigilmon account (sign up in 30 seconds)

Step 1: Create a sync health wrapper script

Wrap your Skopeo sync jobs in a script that reports success or failure to an HTTP collector:

#!/bin/bash
# skopeo-sync-wrapper.sh
set -euo pipefail

SOURCE_REGISTRY="${1:?Usage: $0 <source> <dest> <report-url>}"
DEST_REGISTRY="${2:?}"
REPORT_URL="${3:?}"
SYNC_FILE="${4:-/etc/skopeo/sync.yaml}"

START=$(date +%s)

if skopeo sync \
    --src yaml \
    --dest docker \
    --dest-creds "$(cat /run/secrets/registry-creds)" \
    "$SYNC_FILE" \
    "$DEST_REGISTRY"; then

    END=$(date +%s)
    DURATION=$((END - START))
    curl -sf -X POST "$REPORT_URL/sync-result" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"success\", \"source\": \"$SOURCE_REGISTRY\", \"dest\": \"$DEST_REGISTRY\", \"duration_s\": $DURATION}" \
        || true  # don't fail the sync if reporting fails
else
    EXIT_CODE=$?
    curl -sf -X POST "$REPORT_URL/sync-result" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"error\", \"source\": \"$SOURCE_REGISTRY\", \"dest\": \"$DEST_REGISTRY\", \"exit_code\": $EXIT_CODE}" \
        || true
    exit $EXIT_CODE
fi

A Skopeo sync configuration file:

# /etc/skopeo/sync.yaml
registry.k8s.io:
  images:
    pause: ["3.9", "3.10"]
    coredns/coredns: ["v1.11.1"]

docker.io:
  images:
    library/nginx: ["1.25", "1.26", "alpine"]
    library/postgres: ["16", "17"]

Step 2: Build a sync health API

Create a small server that collects sync job results and exposes health for Vigilmon:

# sync-health-server/app.py
from flask import Flask, jsonify, request
from collections import defaultdict
import time
import threading

app = Flask(__name__)
lock = threading.Lock()

# Track last result per (source, dest) pair
sync_results = {}

# Config: how stale can a sync be before we alert?
MAX_SYNC_AGE_SECONDS = 3600  # 1 hour — adjust to match your cron schedule

@app.route("/sync-result", methods=["POST"])
def record_sync():
    data = request.get_json()
    key = f"{data.get('source', 'unknown')} -> {data.get('dest', 'unknown')}"
    with lock:
        sync_results[key] = {
            "status": data.get("status"),
            "timestamp": time.time(),
            "duration_s": data.get("duration_s"),
            "exit_code": data.get("exit_code"),
        }
    return jsonify({"ok": True})

@app.route("/health/sync")
def sync_health():
    now = time.time()
    with lock:
        results = dict(sync_results)

    if not results:
        return jsonify({"status": "unknown", "detail": "no sync results recorded yet"}), 200

    issues = []
    for key, result in results.items():
        age_s = now - result["timestamp"]
        if result["status"] != "success":
            issues.append(f"{key}: last sync failed (exit {result.get('exit_code')})")
        elif age_s > MAX_SYNC_AGE_SECONDS:
            issues.append(f"{key}: last sync {round(age_s / 60)}m ago (stale)")

    if issues:
        return jsonify({"status": "degraded", "issues": issues}), 503

    freshest_age = min((now - r["timestamp"]) for r in results.values())
    return jsonify({
        "status": "ok",
        "sync_pairs": len(results),
        "freshest_sync_minutes_ago": round(freshest_age / 60, 1),
    })

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9300)

Step 3: Monitor auth token validity

Skopeo auth failures are silent — the sync job exits non-zero and the mirror goes stale. Add an explicit token health check:

#!/bin/bash
# check-registry-auth.sh — run before each sync job
REGISTRY="${1:?}"
CREDS="${2:?}"  # user:password or token

# Try to inspect a small image — if auth is valid, this returns quickly
if skopeo inspect \
    --creds "$CREDS" \
    "docker://$REGISTRY/library/alpine:latest" \
    > /dev/null 2>&1; then
    echo "auth_valid=true"
    exit 0
else
    echo "auth_valid=false"
    exit 1
fi

Wrap this in a Python endpoint for Vigilmon:

# auth-check/app.py
import subprocess
import os
from flask import Flask, jsonify

app = Flask(__name__)

REGISTRY = os.environ.get("REGISTRY", "registry.example.com")
PROBE_IMAGE = os.environ.get("PROBE_IMAGE", "library/alpine:latest")
CREDS_FILE = os.environ.get("CREDS_FILE", "/run/secrets/registry-creds")

@app.route("/health/auth")
def auth_health():
    try:
        creds = open(CREDS_FILE).read().strip()
    except Exception as e:
        return jsonify({"status": "error", "detail": f"cannot read creds: {e}"}), 503

    try:
        result = subprocess.run(
            ["skopeo", "inspect", "--creds", creds,
             f"docker://{REGISTRY}/{PROBE_IMAGE}"],
            capture_output=True,
            text=True,
            timeout=20,
        )
        if result.returncode != 0:
            return jsonify({
                "status": "auth_failed",
                "detail": result.stderr.strip()[:200],
            }), 503

        return jsonify({
            "status": "ok",
            "registry": REGISTRY,
        })
    except subprocess.TimeoutExpired:
        return jsonify({"status": "timeout"}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9301)

Step 4: Monitor image copy latency

Long-running copies indicate registry network issues. Instrument copy operations:

#!/bin/bash
# timed-copy.sh — copy a probe image and report latency
SOURCE="$1"
DEST="$2"
REPORT_URL="$3"

START=$(date +%s%3N)  # milliseconds

if skopeo copy "docker://$SOURCE" "docker://$DEST"; then
    END=$(date +%s%3N)
    LATENCY_MS=$((END - START))
    curl -sf -X POST "$REPORT_URL/copy-latency" \
        -H "Content-Type: application/json" \
        -d "{\"source\": \"$SOURCE\", \"dest\": \"$DEST\", \"latency_ms\": $LATENCY_MS, \"status\": \"ok\"}" \
        || true
else
    curl -sf -X POST "$REPORT_URL/copy-latency" \
        -H "Content-Type: application/json" \
        -d "{\"source\": \"$SOURCE\", \"dest\": \"$DEST\", \"latency_ms\": -1, \"status\": \"error\"}" \
        || true
    exit 1
fi

Add a health endpoint that alerts when latency exceeds a threshold:

@app.route("/health/copy-latency")
def copy_latency_health():
    with lock:
        entries = list(copy_latency_log)

    if not entries:
        return jsonify({"status": "unknown"}), 200

    recent = sorted(entries, key=lambda x: x["timestamp"])[-5:]
    errors = [e for e in recent if e["status"] != "ok"]
    slow = [e for e in recent if e.get("latency_ms", 0) > 120_000]  # 2 minutes

    if errors:
        return jsonify({"status": "error", "failed_copies": len(errors)}), 503
    if slow:
        return jsonify({"status": "slow", "slow_copies": len(slow)}), 503

    avg_ms = sum(e["latency_ms"] for e in recent) / len(recent)
    return jsonify({"status": "ok", "avg_copy_latency_ms": round(avg_ms)})

Step 5: Add Vigilmon monitors

Sync health monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Skopeo — registry sync health | | URL | http://<your-sync-server>:9300/health/sync | | Check interval | 5 minutes | | Expected status | 200 | | Response must contain | "status":"ok" |

Auth token monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Skopeo — registry auth token | | URL | http://<your-server>:9301/health/auth | | Check interval | 15 minutes | | Expected status | 200 | | Response must contain | "status":"ok" |

Copy latency monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Skopeo — inter-registry copy latency | | URL | http://<your-server>:9300/health/copy-latency | | Check interval | 10 minutes | | Expected status | 200 | | Response must contain | "status":"ok" |

Step 6: Monitor the destination registry directly

Skopeo can only push to a registry that's up. Monitor the destination registry's v2 API:

# Test the registry API
curl -s https://registry.your-domain.com/v2/ -I
# HTTP/2 200 or 401 — both mean the API is responding

In Vigilmon, add a TCP or HTTP monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Internal registry — API availability | | URL | https://registry.your-domain.com/v2/ | | Check interval | 1 minute | | Expected status | 200 or 401 |

Step 7: Configure alerts

In Vigilmon, go to Alerts → Alert Channels and connect your notification channels.

Apply alert policies:

| Monitor | Alert after | Notes | |---------|------------|-------| | Sync health | 1 failure | Stale mirrors block deployments | | Auth token | 1 failure | Token expiry cascades to all sync jobs | | Copy latency | 2 failures | Allow one slow copy before alerting | | Registry API | 2 failures | Brief blips are common |

Enable recovery alerts on all monitors so you know when sync resumes after an outage.

What you're monitoring now

| Monitor | What it detects | |---------|-----------------| | Sync health HTTP | Failed syncs, stale mirrors, cron job hangs | | Auth token HTTP | ECR/registry token expiry before it blocks syncs | | Copy latency HTTP | Inter-registry network degradation, timeout failures | | Registry API HTTP | Destination registry outages blocking all pushes |

Conclusion

Skopeo is often invisible — it runs on a schedule in the background and nobody checks it until a deployment fails because the image isn't there. Vigilmon gives you the visibility to catch sync failures, stale mirrors, and expired auth tokens before they become 3am incidents. With health endpoints wrapping your sync jobs and Vigilmon polling them every few minutes, your image supply chain gets the same monitoring rigor as your production services.