tutorial

Monitoring imgpkg Bundle Operations with Vigilmon: OCI Registry Health Checks, Bundle Relocation Pipelines, TLS Certificate Alerts & Carvel Supply Chain Heartbeats

How to use Vigilmon to monitor imgpkg OCI bundle operations — registry API health checks, TLS certificate expiry, and heartbeat monitoring for bundle push, pull, and relocation pipelines in Carvel-based Kubernetes supply chains.

imgpkg is a Carvel tool for packaging, distributing, and relocating Kubernetes configurations and OCI images as self-contained bundles. Used in VMware Tanzu, Cluster API, and GitOps supply chains, imgpkg wraps Kubernetes YAML manifests, Helm charts, and their dependent container images into a single OCI artifact that can be copied between air-gapped registries or promoted across environments. When the source or destination OCI registry becomes unavailable, when an air-gapped relocation pipeline silently stops running, or when a bundle push job fails without surfacing an alert, platform teams lose visibility into the state of their Kubernetes configuration packages. Vigilmon gives you external visibility into the OCI registries imgpkg depends on, TLS certificate health, and the bundle management pipelines that package and relocate your Kubernetes configurations.

What You'll Build

  • HTTP monitors on OCI registry API endpoints imgpkg pushes to and pulls from
  • SSL certificate monitors for each registry HTTPS endpoint
  • TCP monitors on registry ports for network-layer health
  • Heartbeat monitors for imgpkg bundle push, relocation, and copy pipelines
  • Alerting runbook mapping imgpkg failure modes to registry and pipeline health

Prerequisites

  • imgpkg installed (brew install vmware-tanzu/carvel/imgpkg or download from Carvel releases)
  • One or more OCI-compatible container registries (GHCR, Harbor, ECR, GCR, or self-hosted)
  • A free account at vigilmon.online

Step 1: Understand the imgpkg + Registry Health Surface

imgpkg communicates with OCI registries using the standard OCI Distribution Specification. Every imgpkg operation interacts with at least one registry:

# Push a bundle to an OCI registry
imgpkg push -b ghcr.io/myorg/my-bundle:v1.0.0 -f config/
# Expected: Pushed 'ghcr.io/myorg/my-bundle@sha256:...'

# Pull a bundle from a registry
imgpkg pull -b ghcr.io/myorg/my-bundle:v1.0.0 -o /tmp/bundle-output
# Expected: Pulling bundle 'ghcr.io/myorg/my-bundle@sha256:...'

# Copy a bundle from one registry to another (relocation)
imgpkg copy -b ghcr.io/myorg/my-bundle:v1.0.0 \
  --to-repo harbor.example.com/carvel/my-bundle
# Expected: copying image layers between registries

# Copy to a tarball for air-gapped environments
imgpkg copy -b ghcr.io/myorg/my-bundle:v1.0.0 \
  --to-tar /tmp/my-bundle.tar
# Expected: tar archive created

# Copy from tarball into air-gapped registry
imgpkg copy --tar /tmp/my-bundle.tar \
  --to-repo registry.airgap.local/carvel/my-bundle
# Expected: Pushed 'registry.airgap.local/carvel/my-bundle@sha256:...'

# Verify bundle contents after pull
ls /tmp/bundle-output/
# Expected: .imgpkg/ directory with images.yml and bundle config files

# Check OCI registry v2 API reachability
curl -I https://ghcr.io/v2/
# Expected: 401 (auth required — registry is up)

The key failure signals to monitor:

  1. OCI registry v2 API availability (for push, pull, and copy operations)
  2. TLS certificate validity on source and destination registries
  3. TCP network-layer reachability for each registry host
  4. Bundle pipeline heartbeats for push and relocation jobs

Step 2: Monitor OCI Registry API Endpoints

imgpkg operations fail immediately if the target registry's v2 API is unreachable. Monitor every registry that imgpkg bundles flow through:

# Check source registry (where bundles originate)
curl -I https://ghcr.io/v2/
# Expected: 401

# Check destination registry (Harbor, ECR, self-hosted)
curl -I https://harbor.example.com/v2/
# Expected: 200 or 401

# Check air-gapped/internal registry
curl -I https://registry.airgap.local/v2/
# Expected: 200 or 401

# Check GCR (used by Tanzu and Cluster API bundles)
curl -I https://gcr.io/v2/
# Expected: 401

Create Vigilmon monitors for each registry imgpkg interacts with:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://ghcr.io/v2/ (source registry).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status codes: 200, 401.
  6. Label: imgpkg source registry v2 API (GHCR).
  7. Click Save.

Repeat for each registry in the bundle supply chain:

  1. Add Monitor → HTTP.
  2. URL: https://harbor.example.com/v2/.
  3. Expected status codes: 200, 401.
  4. Label: imgpkg destination registry v2 API (Harbor).
  5. Click Save.

Step 3: Monitor Registry TCP Ports

Add TCP monitors to detect network-layer failures independently from the HTTP application layer:

# Test TCP connectivity to source registry
nc -zv ghcr.io 443
# Expected: succeeded

# Test TCP to Harbor destination registry
nc -zv harbor.example.com 443
# Expected: succeeded

# Test TCP to air-gapped registry (may use non-standard port)
nc -zv registry.airgap.local 5000
# Expected: succeeded
  1. Add Monitor → TCP.
  2. Host: ghcr.io.
  3. Port: 443.
  4. Check interval: 60 seconds.
  5. Label: imgpkg source registry TCP (443).
  6. Click Save.

For each additional registry:

  1. Add Monitor → TCP.
  2. Host: harbor.example.com.
  3. Port: 443.
  4. Label: imgpkg Harbor destination registry TCP (443).
  5. Click Save.

Step 4: Monitor TLS Certificates

imgpkg uses HTTPS for all registry communication. An expired certificate on any registry in the bundle supply chain breaks every push, pull, and copy operation for all pipelines simultaneously:

# Check TLS certificate on Harbor destination registry
echo | openssl s_client -connect harbor.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates
# notAfter=Nov 15 00:00:00 2026 GMT

# Verify TLS with imgpkg pull (TLS errors surface here)
imgpkg pull -b harbor.example.com/carvel/my-bundle:v1.0.0 -o /tmp/test 2>&1 | head -5
# If TLS expired: "x509: certificate has expired or is not yet valid"

# Find all registry hostnames imgpkg uses
grep -rE 'imgpkg (push|pull|copy)' .github/ Makefile scripts/ | \
  grep -oP '[a-z0-9.-]+\.(io|com|local)[/:]' | sed 's|[/:]||' | sort -u
# Output: ghcr.io gcr.io harbor.example.com registry.airgap.local
  1. Add Monitor → SSL Certificate.
  2. Domain: harbor.example.com (destination registry, most likely to have a self-managed cert).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Add SSL certificate monitors for every registry hostname in the imgpkg supply chain — including the source registry, destination registry, and any air-gapped internal registry that uses a self-managed TLS certificate.

Step 5: Heartbeat Monitoring for imgpkg Pipelines

imgpkg operations run inside CI/CD pipelines and automated GitOps reconciliation jobs that have no externally queryable health endpoint. Vigilmon heartbeats detect when these pipelines stop running.

Bundle Push Pipeline

A GitHub Actions workflow that packages Kubernetes configurations as an imgpkg bundle on every release:

# .github/workflows/bundle-push.yml
name: Package and push imgpkg bundle

on:
  push:
    tags: ['v*']

jobs:
  push-bundle:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install imgpkg
        run: |
          curl -sL https://github.com/vmware-tanzu/carvel-imgpkg/releases/latest/download/imgpkg-linux-amd64 \
            -o /usr/local/bin/imgpkg
          chmod +x /usr/local/bin/imgpkg

      - name: Login to GHCR
        run: |
          echo "${{ secrets.GITHUB_TOKEN }}" | \
            docker login ghcr.io -u ${{ github.actor }} --password-stdin

      - name: Push bundle
        run: |
          imgpkg push \
            -b ghcr.io/${{ github.repository_owner }}/my-bundle:${{ github.ref_name }} \
            -f config/

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/abc123

Set up the heartbeat monitor:

  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 10080 minutes (bundle pushed on every release tag, expected at least weekly).
  3. Copy the heartbeat URL and paste it into the workflow.
  4. Label: imgpkg bundle push — release pipeline.
  5. Click Save.

Bundle Relocation Pipeline (Cross-Registry Promotion)

A scheduled pipeline that copies bundles from the source registry to a Harbor instance for production deployment:

#!/bin/bash
# relocate-bundles.sh — run via cron or CI on a schedule

SOURCE_REGISTRY="ghcr.io/myorg"
DEST_REGISTRY="harbor.example.com/production"
VIGILMON_HB="https://vigilmon.online/heartbeat/def456"

BUNDLES=(
  "my-bundle"
  "platform-config"
  "app-manifests"
)

FAILED=0
for BUNDLE in "${BUNDLES[@]}"; do
  LATEST_TAG=$(crane ls "${SOURCE_REGISTRY}/${BUNDLE}" 2>/dev/null | tail -1)
  if [ -z "${LATEST_TAG}" ]; then
    echo "FAIL: Could not list tags for ${SOURCE_REGISTRY}/${BUNDLE}"
    FAILED=1
    continue
  fi

  echo "Relocating ${BUNDLE}:${LATEST_TAG} → ${DEST_REGISTRY}/${BUNDLE}"
  if ! imgpkg copy \
    -b "${SOURCE_REGISTRY}/${BUNDLE}:${LATEST_TAG}" \
    --to-repo "${DEST_REGISTRY}/${BUNDLE}" 2>&1; then
    echo "FAIL: Relocation failed for ${BUNDLE}"
    FAILED=1
  else
    echo "OK: ${BUNDLE}:${LATEST_TAG} relocated"
  fi
done

if [ "${FAILED}" -eq 0 ]; then
  curl -s "${VIGILMON_HB}"
fi
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 90 minutes (hourly relocation with a 30-minute grace period).
  3. Label: imgpkg bundle relocation — GHCR to Harbor.
  4. Click Save.

Air-Gapped Bundle Copy Pipeline

A pipeline that packages bundles to a tarball and imports them into an air-gapped environment:

#!/bin/bash
# airgap-bundle-import.sh — runs in the air-gapped network after tarball transfer

TARBALL_DIR="/transfer/bundles"
AIRGAP_REGISTRY="registry.airgap.local/carvel"
VIGILMON_HB="https://vigilmon.airgap.local/heartbeat/ghi789"  # internal Vigilmon

FAILED=0
for TARBALL in "${TARBALL_DIR}"/*.tar; do
  BUNDLE_NAME=$(basename "${TARBALL}" .tar)
  echo "Importing ${BUNDLE_NAME} from tarball..."

  if ! imgpkg copy \
    --tar "${TARBALL}" \
    --to-repo "${AIRGAP_REGISTRY}/${BUNDLE_NAME}" 2>&1; then
    echo "FAIL: Import failed for ${BUNDLE_NAME}"
    FAILED=1
  else
    echo "OK: ${BUNDLE_NAME} imported"
  fi
done

if [ "${FAILED}" -eq 0 ]; then
  curl -s "${VIGILMON_HB}"
fi
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes (daily air-gapped import with grace period).
  3. Label: imgpkg air-gapped bundle import.
  4. Click Save.

Step 6: Validate Bundle Availability After Push

A validation script that confirms imgpkg bundles remain pullable after push — useful as a smoke test after relocation:

#!/bin/bash
# validate-imgpkg-bundles.sh — run every 30 minutes via cron

REGISTRY="harbor.example.com/production"
VIGILMON_HB="https://vigilmon.online/heartbeat/jkl012"
TMPDIR="/tmp/bundle-validation-$$"
FAILED=0

BUNDLES=(
  "my-bundle"
  "platform-config"
)

mkdir -p "${TMPDIR}"
trap "rm -rf ${TMPDIR}" EXIT

for BUNDLE in "${BUNDLES[@]}"; do
  DIGEST=$(crane digest "${REGISTRY}/${BUNDLE}:latest" 2>/dev/null)
  if [ -z "${DIGEST}" ]; then
    echo "FAIL: Cannot get digest for ${REGISTRY}/${BUNDLE}:latest"
    FAILED=1
    continue
  fi

  # Attempt a pull to validate the bundle is intact
  PULL_DIR="${TMPDIR}/${BUNDLE}"
  mkdir -p "${PULL_DIR}"
  if ! imgpkg pull -b "${REGISTRY}/${BUNDLE}:latest" -o "${PULL_DIR}" 2>/dev/null; then
    echo "FAIL: Cannot pull ${REGISTRY}/${BUNDLE}:latest"
    FAILED=1
  elif [ ! -d "${PULL_DIR}/.imgpkg" ]; then
    echo "FAIL: Bundle structure missing .imgpkg directory in ${BUNDLE}"
    FAILED=1
  else
    echo "OK: ${REGISTRY}/${BUNDLE}:latest (${DIGEST})"
  fi
done

if [ "${FAILED}" -eq 0 ]; then
  curl -s "${VIGILMON_HB}"
fi

Schedule via cron:

# /etc/cron.d/imgpkg-validation
*/30 * * * * deploy /opt/scripts/validate-imgpkg-bundles.sh >> /var/log/imgpkg-validation.log 2>&1
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 35 minutes.
  3. Label: imgpkg bundle availability check.
  4. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | Source registry v2 API | Non-200/401 or timeout | Confirm with curl -I https://ghcr.io/v2/; all imgpkg push/pull operations will fail | | Destination registry v2 API | Non-200/401 | imgpkg copy and relocation pipelines will fail; check Harbor or ECR status | | Registry TCP | Connection refused | Network-layer failure; check firewall rules and DNS resolution for the registry host | | TLS certificate | < 30 days | Rotate certificate; verify with imgpkg pull after renewal to confirm no TLS errors | | Bundle push heartbeat | Missed ping | Check CI/CD pipeline logs; look for registry auth failures or imgpkg version mismatches | | Bundle relocation heartbeat | Missed ping | Check relocation cron job logs; verify both source and destination registries are reachable | | Air-gapped import heartbeat | Missed ping | Check tarball transfer completed; verify internal registry accepts uploads; check disk space | | Bundle availability heartbeat | Missed ping | Run imgpkg pull manually; check if bundle was deleted from registry |

Common imgpkg Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Source registry API down | Source registry monitor fires; all imgpkg push and copy operations fail | | Destination registry (Harbor) unavailable | Destination registry monitor fires; relocation pipelines stall | | TLS certificate expired on Harbor | SSL monitor fires; imgpkg copy returns x509 errors | | Bundle push pipeline disabled in CI | Push heartbeat goes silent after expected interval | | Relocation cron job misconfigured | Relocation heartbeat goes silent; production registry falls behind source | | Air-gapped tarball transfer stalled | Air-gapped import heartbeat goes silent; bundles in air-gapped environment go stale | | Registry disk full blocks layer uploads | Registry returns 5xx; v2 API monitor may stay green but imgpkg push fails | | Bundle deleted or GC'd from registry | Availability heartbeat fires; downstream systems cannot pull the bundle | | imgpkg version incompatibility after upgrade | Push heartbeat fires; check imgpkg version in CI vs registry OCI spec support | | Network partition between source and destination | Relocation heartbeat fires; TCP monitors confirm which side lost connectivity |

imgpkg makes Kubernetes configuration portable and relocatable across registries and air-gapped environments — but those operations are only as reliable as the OCI registries they flow through and the pipelines that automate them. Vigilmon gives you the external visibility layer that catches registry outages, TLS certificate expirations, and silent pipeline failures before they leave your Kubernetes configurations stale or your production registry out of sync with the source.

Start monitoring your imgpkg supply chain in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles