tutorial

harbor_health.py

## Prerequisites - Harbor 2.x deployed (Docker Compose or Helm) - Harbor admin credentials - A free account at [vigilmon.online](https://vigilmon.online) ---...

Prerequisites

  • Harbor 2.x deployed (Docker Compose or Helm)
  • Harbor admin credentials
  • A free account at vigilmon.online

Part 1: Understanding what to monitor in Harbor

Harbor has several failure surfaces:

  1. Core API (/api/v2.0/ping) — the registry API gateway; if this fails, all push/pull operations fail
  2. Portal — the web UI; less critical but user-facing
  3. Database — PostgreSQL backend; a locked or full database stops all write operations
  4. Redis — used for job queue and caching; job service fails without it
  5. Storage — the backend blob store (filesystem, S3, GCS); full storage silently breaks pushes
  6. Replication — if replication jobs fail, downstream registries diverge from the primary

Harbor exposes /api/v2.0/health which checks all subsystems. Use it as the primary monitor.

Part 2: Monitor Harbor's built-in health endpoint

Harbor ships a built-in health check at /api/v2.0/health that reports the status of all internal components.

Verify the endpoint manually

curl -u admin:Harbor12345 https://harbor.example.com/api/v2.0/health

Expected healthy response:

{
  "status": "healthy",
  "components": [
    {"name": "core",       "status": "healthy"},
    {"name": "database",   "status": "healthy"},
    {"name": "jobservice", "status": "healthy"},
    {"name": "portal",     "status": "healthy"},
    {"name": "redis",      "status": "healthy"},
    {"name": "registry",   "status": "healthy"},
    {"name": "registryctl","status": "healthy"},
    {"name": "trivy",      "status": "healthy"}
  ]
}

Any component that reports "status": "unhealthy" or "status": "unknown" will cause that component's status string to be absent from the "healthy" value — making a keyword check on "status":"healthy" in the top-level field sufficient.

Add a Vigilmon HTTP monitor for Harbor health

  1. Log in to vigilmon.online and click Add Monitor.
  2. Choose HTTP(S) monitor.
  3. Enter: https://harbor.example.com/api/v2.0/health
  4. Add a Basic Auth header: Authorization: Basic <base64(admin:password)>
  5. Set interval to 1 minute.
  6. Add a keyword check: must contain "status":"healthy".
  7. Add your alert channel.
  8. Click Save.

This single monitor covers core, database, job service, Redis, registry, and Trivy simultaneously.

Part 3: Build a Harbor extended health endpoint

The built-in endpoint does not surface storage capacity or replication lag. Build a sidecar for these.

Create harbor_health.py

import os
import requests
from fastapi import FastAPI, Response

app = FastAPI()

HARBOR_URL = os.environ.get("HARBOR_URL", "https://harbor.example.com")
HARBOR_USER = os.environ.get("HARBOR_USER", "admin")
HARBOR_PASS = os.environ.get("HARBOR_PASS", "")
REPLICATION_FAIL_THRESHOLD = int(os.environ.get("REPLICATION_FAIL_THRESHOLD", "3"))

AUTH = (HARBOR_USER, HARBOR_PASS)
API = f"{HARBOR_URL}/api/v2.0"


def get(path, **kwargs):
    return requests.get(f"{API}{path}", auth=AUTH, timeout=10, **kwargs)


@app.get("/health")
def health(response: Response):
    checks = {}

    # Built-in health
    try:
        resp = get("/health")
        data = resp.json()
        overall = data.get("status", "")
        if overall == "healthy":
            checks["harbor_core"] = "ok"
        else:
            unhealthy = [
                c["name"]
                for c in data.get("components", [])
                if c.get("status") != "healthy"
            ]
            checks["harbor_core"] = f"degraded: {', '.join(unhealthy)}"
    except Exception as e:
        checks["harbor_core"] = f"error: {e}"

    # Storage usage (requires system admin)
    try:
        resp = get("/systeminfo")
        info = resp.json()
        total = info.get("total_capacity", 0)
        free = info.get("free_space", 0)
        if total > 0:
            used_pct = int((total - free) / total * 100)
            checks["storage_used_pct"] = used_pct
            if used_pct >= 90:
                checks["storage"] = f"warning: {used_pct}% used"
            else:
                checks["storage"] = "ok"
        else:
            checks["storage"] = "ok (unknown capacity)"
    except Exception as e:
        checks["storage"] = f"error: {e}"

    # Replication job failures
    try:
        resp = get("/replication/executions", params={"status": "Failed", "page_size": 10})
        failed = resp.json()
        if isinstance(failed, list) and len(failed) >= REPLICATION_FAIL_THRESHOLD:
            checks["replication"] = f"warning: {len(failed)} recent failures"
        else:
            checks["replication"] = "ok"
    except Exception as e:
        checks["replication"] = f"error: {e}"

    core_ok = checks.get("harbor_core") == "ok"
    storage_ok = checks.get("storage") in ("ok", "ok (unknown capacity)")
    healthy = core_ok and storage_ok

    response.status_code = 200 if healthy else 503
    return {"status": "ok" if healthy else "degraded", "checks": checks}

Run the health service

pip install fastapi uvicorn requests
HARBOR_URL=https://harbor.example.com \
HARBOR_USER=admin \
HARBOR_PASS=Harbor12345 \
uvicorn harbor_health:app --host 0.0.0.0 --port 8097

Add a systemd unit

# /etc/systemd/system/harbor-health.service
[Unit]
Description=Harbor extended health endpoint
After=network.target

[Service]
User=harbor
EnvironmentFile=/etc/harbor-health.env
WorkingDirectory=/opt/harbor-health
ExecStart=/usr/local/bin/uvicorn harbor_health:app --host 0.0.0.0 --port 8097
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

# /etc/harbor-health.env
HARBOR_URL=https://harbor.example.com
HARBOR_USER=health-reader
HARBOR_PASS=<readonly-password>
REPLICATION_FAIL_THRESHOLD=3

sudo systemctl enable harbor-health
sudo systemctl start harbor-health

Verify the endpoint

curl http://localhost:8097/health

Expected healthy response:

{
  "status": "ok",
  "checks": {
    "harbor_core": "ok",
    "storage_used_pct": 47,
    "storage": "ok",
    "replication": "ok"
  }
}

Part 4: Set up monitors in Vigilmon

Monitor Harbor built-in health (primary)

Add this as your primary monitor — it reflects database, Redis, and component health:

| Field | Value | |---|---| | Type | HTTP(S) | | URL | https://harbor.example.com/api/v2.0/health | | Interval | 1 minute | | Keyword | "status":"healthy" | | Auth | Basic auth header |

Monitor Harbor extended health (storage + replication)

| Field | Value | |---|---| | Type | HTTP(S) | | URL | http://harbor-host:8097/health | | Interval | 5 minutes | | Keyword | "status":"ok" |

Monitor Harbor ping endpoint

The /api/v2.0/ping endpoint requires no authentication and is the fastest proxy for "is Harbor reachable at all":

  1. Add Monitor → HTTP(S).
  2. URL: https://harbor.example.com/api/v2.0/ping
  3. Interval: 1 minute.
  4. Expected response: "Pong" (keyword check).

Monitor per-project registries

If you host multiple projects with different criticality:

| Monitor name | URL | Notes | |---|---|---| | Harbor — production project | https://harbor.example.com/v2/production/manifests/latest | Tag pull test | | Harbor — CI project | https://harbor.example.com/v2/ci/manifests/latest | CI pipeline images |

Part 5: Docker Compose deployment

# docker-compose.yml (add alongside existing Harbor compose)
version: "3.8"

services:
  harbor-health:
    image: python:3.12-slim
    command: >
      sh -c "pip install -q fastapi uvicorn requests &&
             uvicorn harbor_health:app --host 0.0.0.0 --port 8097"
    ports:
      - "8097:8097"
    volumes:
      - ./harbor_health.py:/app/harbor_health.py
    working_dir: /app
    environment:
      HARBOR_URL: http://nginx
      HARBOR_USER: health-reader
      HARBOR_PASS: ${HARBOR_HEALTH_PASS}
    networks:
      - harbor

networks:
  harbor:
    external: true
    name: harbor_harbor

Part 6: Webhook alerts for registry outages

A Harbor outage causes imagePullBackOff across every namespace in the cluster. Alert the platform team immediately.

// webhook-receiver.ts
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhook/vigilmon', async (req, res) => {
  const { monitor_name, status, url, checked_at } = req.body;

  if (status === 'down') {
    console.error('[REGISTRY] Harbor is DOWN', {
      monitor: monitor_name,
      url,
      at: checked_at,
    });

    await alertPlatformTeam({
      title: `Harbor DOWN: ${monitor_name}`,
      severity: 'critical',
      impact: 'All image pulls from Harbor are failing — Kubernetes workloads cannot start',
      since: checked_at,
      runbook: 'https://wiki.example.com/runbooks/harbor-recovery',
    });

    await openIncident({
      service: 'container-registry',
      title: 'Harbor registry unreachable',
    });

  } else if (status === 'up') {
    console.info('[REGISTRY] Harbor recovered', { monitor: monitor_name });
    await resolveAlert(monitor_name);
  }

  res.sendStatus(204);
});

Vigilmon sends this payload on status change:

{
  "monitor_id": "mon_harbor",
  "monitor_name": "Harbor — Core API",
  "status": "down",
  "url": "https://harbor.example.com/api/v2.0/health",
  "checked_at": "2026-07-02T09:15:00Z",
  "response_code": 503,
  "response_time_ms": 30001
}

Part 7: SSL monitoring for Harbor

Harbor serves image pull traffic and the web UI over HTTPS. An expired certificate breaks all container image pulls cluster-wide.

  1. In Vigilmon, click Add Monitor.
  2. Choose SSL monitor.
  3. Enter: harbor.example.com
  4. Set alert threshold to 21 days before expiry (more lead time than usual because certificate rotation on a registry requires restarting Harbor and all cluster nodes that have the old cert cached).
  5. Add your alert channel.

If you use internal CAs, set a shorter threshold and test renewal automatically with cert-manager.

Summary

Your Harbor deployment now has layered monitoring:

  1. Built-in health API monitor — covers core, database, Redis, job service, registry, and Trivy in a single 1-minute check via Vigilmon.
  2. Extended health endpoint — surfaces storage capacity and replication failures not visible in the built-in API.
  3. Ping monitor — fast, unauthenticated check confirming Harbor is reachable at all.
  4. Per-project pull tests — validates that images in critical projects are actually pullable, not just that the API responds.
  5. Webhook escalation — triggers incident creation and platform team paging when Harbor goes down.
  6. SSL monitor with extended lead time — alerts 21 days before expiry to account for registry certificate rotation complexity.

Vigilmon handles check scheduling, multi-region polling, and alert routing. When Harbor goes down, your platform team knows within 60 seconds — before imagePullBackOff errors start filling Slack.

Monitor your Harbor container registry free at vigilmon.online

#harbor #containerregistry #kubernetes #devops #security #monitoring

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles