tutorial

How to Monitor Harbor Satellite with Vigilmon (Sync Health, Replication Lag + Alerts)

Harbor Satellite extends the Harbor container registry to edge and remote sites — air-gapped factories, retail stores, remote offices, or offshore facilities...

Harbor Satellite extends the Harbor container registry to edge and remote sites — air-gapped factories, retail stores, remote offices, or offshore facilities — by running a lightweight registry node that replicates images from a central Harbor instance. Teams use it to eliminate dependency on WAN connectivity for container deployments: images are cached locally and served even when the link to headquarters is down.

But distributed registries introduce a new class of failure. Your satellite may be running and serving images while silently diverging from the central registry because a replication job stalled, a TLS certificate expired, or a network partition severed the sync link. In this tutorial you'll set up comprehensive monitoring for your Harbor Satellite infrastructure using Vigilmon — free tier, no credit card required.

Why Harbor Satellite needs external monitoring

Harbor's built-in health dashboard shows what the central registry sees. It cannot tell you what's happening at your satellite sites, and it won't alert you when a satellite stops syncing.

The failure modes that external monitoring catches:

  • Satellite unreachable — the satellite service crashes or the host goes offline; deployments at the edge site pull from the internet instead of the local mirror, hitting rate limits or WAN latency
  • Replication lag exceeding SLA — the sync job falls behind; the satellite serves stale images; edge deployments run old versions without knowing it
  • TLS certificate expiry — satellite certificates rotate independently from the central registry; an expired satellite cert causes all local container pulls to fail with x509 errors
  • Central registry connectivity lost — the satellite can't reach the central Harbor; it serves cached images but new images pushed to central will never arrive
  • Disk saturation — satellite storage fills up; replication jobs stall silently while the satellite continues serving previously cached images

External monitoring from Vigilmon provides a neutral view of both the satellite and the central registry, and alerts you when the gap between them becomes operationally significant.

What you'll need

  • Harbor Satellite deployed at one or more edge sites
  • Central Harbor instance accessible from the internet (or your monitoring network)
  • A free Vigilmon account — sign up takes 30 seconds

Step 1: Monitor satellite registry health

Harbor Satellite exposes a standard OCI registry API. Monitor the v2 root endpoint to verify the satellite is running and accepting connections.

  1. Go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://satellite.edge-site-1.example.com/v2/
  4. Interval: 1 minute
  5. Expected status: 200 or 401 (the OCI v2 root returns 401 for unauthenticated requests — this is normal and indicates the satellite is healthy)
  6. Name: Satellite edge-site-1 Health
  7. Save

Repeat for each satellite site. If you have many sites, use Vigilmon's tagging to group them:

  • Tag: satellite
  • Tag: edge-site-1

Step 2: Monitor central Harbor reachability from satellite perspective

The satellite's ability to sync depends on reaching the central Harbor. Monitor the central registry's API endpoint:

  1. New monitor → HTTP / HTTPS
  2. URL: https://harbor.central.example.com/api/v2.0/health
  3. Interval: 2 minutes
  4. Expected status: 200
  5. Name: Central Harbor API Health

Harbor's /api/v2.0/health returns a JSON object:

{
  "status": "healthy",
  "components": [
    {"name": "core", "status": "healthy"},
    {"name": "database", "status": "healthy"},
    {"name": "jobservice", "status": "healthy"},
    {"name": "registry", "status": "healthy"}
  ]
}

If jobservice becomes unhealthy, replication jobs stop running and satellite sync silently halts.

Step 3: Monitor replication lag with a heartbeat probe

Replication lag is the most operationally significant metric for satellite deployments. When lag exceeds your SLA, edge sites run stale images. Set up a probe that measures and reports replication lag to Vigilmon.

Create a probe script that runs on a host with access to both the central Harbor API and the satellite API:

#!/usr/bin/env bash
# harbor-satellite-lag-probe.sh

CENTRAL_URL="https://harbor.central.example.com"
SATELLITE_URL="https://satellite.edge-site-1.example.com"
HARBOR_USER="$HARBOR_PROBE_USER"
HARBOR_PASS="$HARBOR_PROBE_PASSWORD"
MAX_LAG_SECONDS=3600  # Alert if lag exceeds 1 hour
HEARTBEAT_URL="$VIGILMON_SATELLITE_HEARTBEAT_URL"

# Get the digest of a canary image from central
CENTRAL_DIGEST=$(curl -s \
  -u "$HARBOR_USER:$HARBOR_PASS" \
  "$CENTRAL_URL/v2/library/probe-image/manifests/latest" \
  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  -D - | grep -i "docker-content-digest" | awk '{print $2}' | tr -d '\r')

if [ -z "$CENTRAL_DIGEST" ]; then
  echo "Failed to fetch central digest"
  exit 1
fi

# Get the same image's digest from the satellite
SATELLITE_DIGEST=$(curl -s \
  -u "$HARBOR_USER:$HARBOR_PASS" \
  "$SATELLITE_URL/v2/library/probe-image/manifests/latest" \
  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  -D - | grep -i "docker-content-digest" | awk '{print $2}' | tr -d '\r')

if [ -z "$SATELLITE_DIGEST" ]; then
  echo "Failed to fetch satellite digest — satellite may be down"
  exit 1
fi

if [ "$CENTRAL_DIGEST" != "$SATELLITE_DIGEST" ]; then
  # Get when central image was last pushed
  PUSH_TIME=$(curl -s \
    -u "$HARBOR_USER:$HARBOR_PASS" \
    "$CENTRAL_URL/api/v2.0/projects/library/repositories/probe-image/artifacts?page_size=1" \
    | jq -r '.[0].push_time')

  PUSH_EPOCH=$(date -d "$PUSH_TIME" +%s 2>/dev/null || gdate -d "$PUSH_TIME" +%s)
  NOW_EPOCH=$(date +%s)
  LAG=$(( NOW_EPOCH - PUSH_EPOCH ))

  echo "Replication lag: ${LAG}s (central: $CENTRAL_DIGEST, satellite: $SATELLITE_DIGEST)"

  if [ "$LAG" -gt "$MAX_LAG_SECONDS" ]; then
    echo "LAG EXCEEDS SLA: ${LAG}s > ${MAX_LAG_SECONDS}s"
    exit 1
  fi
fi

# Satellite is in sync (or lag is within SLA)
curl -s "$HEARTBEAT_URL"
echo "Satellite sync healthy"

Schedule this every 15 minutes:

*/15 * * * * /opt/scripts/harbor-satellite-lag-probe.sh >> /var/log/satellite-probe.log 2>&1

In Vigilmon, create a Heartbeat monitor:

  • Name: Satellite edge-site-1 Replication Lag
  • Grace period: 30 minutes

Step 4: Monitor TLS certificate expiry

Satellite certificates often have independent rotation schedules from the central registry. A certificate expiry at an edge site breaks all container pulls silently — docker pull fails with an x509 error and the failure looks like a network problem.

  1. New monitor → HTTP / HTTPS
  2. URL: https://satellite.edge-site-1.example.com
  3. Enable SSL certificate expiry alert: 21 days before expiry
  4. Save as Satellite edge-site-1 TLS

Vigilmon checks the certificate on every poll and alerts you 21 days before expiry — enough time to rotate certificates before any disruption.

Repeat for each satellite site and for the central Harbor:

https://harbor.central.example.com  → 21-day TLS alert
https://satellite.edge-site-1.example.com  → 21-day TLS alert
https://satellite.edge-site-2.example.com  → 21-day TLS alert

Step 5: Monitor satellite disk capacity

A full disk silently stalls replication. Harbor Satellite doesn't emit alerts when storage is exhausted — it just stops writing new image layers and may serve incomplete manifests.

Add a disk capacity probe to your monitoring:

#!/usr/bin/env bash
# satellite-disk-probe.sh — run on the satellite host via SSH or as a local cron

SATELLITE_DATA_DIR="/var/lib/harbor-satellite"
THRESHOLD_PCT=85
HEARTBEAT_URL="$VIGILMON_DISK_HEARTBEAT_URL"

USED_PCT=$(df "$SATELLITE_DATA_DIR" | awk 'NR==2 {gsub(/%/,""); print $5}')

if [ -z "$USED_PCT" ]; then
  echo "Could not determine disk usage for $SATELLITE_DATA_DIR"
  exit 1
fi

if [ "$USED_PCT" -gt "$THRESHOLD_PCT" ]; then
  echo "Satellite disk usage critical: ${USED_PCT}% used (threshold: ${THRESHOLD_PCT}%)"
  exit 1
fi

echo "Satellite disk healthy: ${USED_PCT}% used"
curl -s "$HEARTBEAT_URL"

If the satellite host is accessible via SSH from your monitoring machine:

#!/usr/bin/env bash
# Check satellite disk remotely
ssh -i /opt/keys/satellite-key satellite@edge-site-1.example.com \
  'df /var/lib/harbor-satellite | awk "NR==2 {gsub(/%/,\"\"); if (\$5 > 85) exit 1; exit 0}"' \
  && curl -s "$VIGILMON_DISK_HEARTBEAT_URL"

Step 6: Monitor image availability at edge

The ultimate test of satellite health is whether specific images are actually pullable at the edge site. Create an image availability probe that attempts to pull a canary image from the satellite:

#!/usr/bin/env bash
# satellite-image-availability.sh

SATELLITE="satellite.edge-site-1.example.com"
CANARY_IMAGE="$SATELLITE/library/probe-image:latest"
HEARTBEAT_URL="$VIGILMON_AVAILABILITY_HEARTBEAT_URL"

# Pull the canary image — this exercises the full registry pull path
docker pull "$CANARY_IMAGE" > /tmp/satellite-pull.log 2>&1
PULL_STATUS=$?

if [ $PULL_STATUS -ne 0 ]; then
  echo "Failed to pull canary image from satellite:"
  cat /tmp/satellite-pull.log
  exit 1
fi

# Remove immediately to avoid disk accumulation
docker rmi "$CANARY_IMAGE" > /dev/null 2>&1

curl -s "$HEARTBEAT_URL"
echo "Canary image pullable from satellite"

Schedule every 10 minutes from a node co-located at the edge site. This probe runs the full pull path — DNS resolution, TLS handshake, authentication, and layer download — catching issues that a simple HTTP check of the API endpoint would miss.

Step 7: Monitor replication job success in Harbor

Harbor exposes replication job status via its API. Poll it to detect stuck or failed replication jobs before they cause lag:

#!/usr/bin/env bash
# harbor-replication-jobs.sh — check for recent replication failures

CENTRAL_URL="https://harbor.central.example.com"
HARBOR_USER="$HARBOR_PROBE_USER"
HARBOR_PASS="$HARBOR_PROBE_PASSWORD"
REPLICATION_POLICY_ID="1"  # Your satellite replication policy ID
HEARTBEAT_URL="$VIGILMON_REPLJOB_HEARTBEAT_URL"

# Get recent executions for this replication policy
EXECUTIONS=$(curl -s \
  -u "$HARBOR_USER:$HARBOR_PASS" \
  "$CENTRAL_URL/api/v2.0/replication/executions?policy_id=$REPLICATION_POLICY_ID&page_size=5")

FAILED=$(echo "$EXECUTIONS" | jq '[.[] | select(.status == "Failed")] | length')
IN_PROGRESS=$(echo "$EXECUTIONS" | jq '[.[] | select(.status == "InProgress")] | length')

echo "Replication jobs — Failed: $FAILED, In Progress: $IN_PROGRESS"

# Alert if any recent execution failed
if [ "$FAILED" -gt 0 ]; then
  echo "FAILED replication jobs detected"
  exit 1
fi

curl -s "$HEARTBEAT_URL"

Schedule every 5 minutes with a Vigilmon heartbeat grace period of 10 minutes.

Step 8: Set up alerts

Configure alerts in Vigilmon for all satellite monitors:

| Monitor | Alert threshold | Notification | |---|---|---| | Satellite HTTP health | 2 minutes down | Slack #edge-ops, email on-call | | Central Harbor API | 3 minutes down | Slack #infra-alerts | | Replication Lag heartbeat | 30-minute silence | Slack #edge-ops | | TLS certificate | 21 days before expiry | Email platform team | | Disk probe heartbeat | 20-minute silence | Slack #edge-ops | | Image availability heartbeat | 15-minute silence | PagerDuty | | Replication jobs heartbeat | 10-minute silence | Slack #edge-ops |

Group all satellite monitors under a Status Page in Vigilmon:

  1. Go to Status Pages → New
  2. Name: Edge Registry Satellite Status
  3. Add all satellite monitors
  4. Share the URL with your edge site operators so they can check status without needing Vigilmon access

Interpreting Vigilmon alerts for Harbor Satellite

| Alert | Likely cause | First action | |---|---|---| | Satellite HTTP down | Service crashed, host offline | SSH to satellite host; systemctl status harbor-satellite | | Central Harbor API down | Harbor service issue, database failure | Check Harbor admin panel; review jobservice component status | | Replication lag heartbeat silent | Probe script failed, replication stuck | Check /var/log/satellite-probe.log; review Harbor replication executions API | | TLS alert | Certificate approaching expiry | Rotate certificate; update satellite config with new cert | | Disk probe silent | Disk full, SSH key issues | Run df /var/lib/harbor-satellite; run GC in Harbor to free layers | | Image availability silent | Pull failure, auth error, layer corruption | Run docker pull manually from edge node; check satellite logs | | Replication jobs silent | Failed jobs, Harbor jobservice down | Check Harbor API for failed execution details |

What's next

With satellite monitoring in place:

  • Multi-site dashboard — use Vigilmon's status page to show replication health across all satellite sites in one view
  • Image signing verification — add Cosign verification to the image availability probe to catch cases where signed images arrive at the satellite with invalid signatures
  • Bandwidth alerting — monitor WAN link utilization alongside replication lag to correlate network congestion with sync delays
  • Automated GC scheduling — trigger Harbor garbage collection proactively when disk probes report usage above 75%, rather than waiting for saturation

Harbor Satellite solves the WAN dependency problem for container deployments at the edge. Vigilmon ensures that the satellite's own health — sync state, image availability, and connectivity — stays observable from a central vantage point, regardless of which edge site you're monitoring.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles