tutorial

Monitoring Minimal Container Builds with apko and Vigilmon: Registry Health, APK Repository Checks, TLS Certificate Alerts & Build Pipeline Heartbeats

How to use apko and Vigilmon together to monitor minimal OCI container image build pipelines — registry API health checks, Alpine/Wolfi APK repository availability, TLS certificate expiry, and heartbeat monitoring for apko-based image automation.

apko is Chainguard's declarative OCI image builder — it assembles minimal, distroless-style container images directly from APK packages using a YAML configuration file, without a Dockerfile, without a build daemon, and without any unnecessary runtime dependencies. Used in Chainguard's own Wolfi-based image supply chain and in security-conscious platform engineering teams, apko produces verifiably minimal images with complete SBOMs and signed provenance. When the APK repository apko pulls packages from is unreachable, when the OCI registry apko pushes images to goes down, or when a scheduled apko image rebuild pipeline silently stops running, security teams lose confidence that images remain patched and minimal. Vigilmon gives you external visibility into the APK package repositories and OCI registries apko depends on, TLS certificate health for those endpoints, and heartbeat monitoring for the apko-based image build pipelines that keep your container supply chain fresh.

What You'll Build

  • HTTP monitors on the OCI registry API endpoints apko pushes images to
  • HTTP monitors on the APK package repositories apko fetches packages from
  • SSL certificate monitors for registry and APK repository endpoints
  • TCP monitors on registry ports for network-layer health
  • Heartbeat monitors for apko-based scheduled rebuild pipelines
  • Alerting runbook mapping apko failure modes to registry and repository health issues

Prerequisites

  • apko installed (brew install apko or via the official release binary)
  • One or more OCI-compatible container registries (GHCR, Docker Hub, Chainguard registry, or self-hosted)
  • APK repositories configured in your apko YAML (Wolfi, Alpine, or private APK repositories)
  • A free account at vigilmon.online

Step 1: Understand the apko + Registry and APK Health Surface

apko reads a YAML configuration file and performs two network-dependent operations:

  1. Fetches APK packages from the configured APK repositories (Wolfi, Alpine, or custom repos)
  2. Pushes the assembled OCI image to the target container registry

Both of these can fail independently:

# Example apko YAML configuration (alpine-base.yaml)
# contents:
#   keyring:
#     - https://alpinelinux.org/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
#   repositories:
#     - https://dl-cdn.alpinelinux.org/alpine/edge/main
#   packages:
#     - alpine-baselayout
#     - busybox
#     - ca-certificates
# archs:
#   - x86_64
#   - aarch64

# Build and push an image with apko
apko publish alpine-base.yaml ghcr.io/myorg/alpine-base:edge

# Build to a local OCI tarball (no registry push)
apko build alpine-base.yaml alpine-base.tar

# Check the target registry is reachable
curl -I https://ghcr.io/v2/
# Expected: 401

# Test the APK repository is reachable
curl -I https://dl-cdn.alpinelinux.org/alpine/edge/main/
# Expected: 200

# Test the Wolfi APK repository
curl -I https://packages.wolfi.dev/os/
# Expected: 200

The key failure signals to monitor:

  1. Target OCI registry v2 API availability
  2. APK repository availability (Alpine CDN, Wolfi packages, custom repos)
  3. TLS certificate validity on registry and APK repository endpoints
  4. Build pipeline heartbeats for scheduled apko image rebuild jobs

Step 2: Monitor the OCI Registry v2 API

The /v2/ endpoint is the health signal for the OCI registry apko pushes assembled images to:

# GitHub Container Registry
curl -I https://ghcr.io/v2/
# Expected: 401

# Docker Hub
curl -I https://registry-1.docker.io/v2/
# Expected: 401

# Chainguard registry
curl -I https://cgr.dev/v2/
# Expected: 401

# Self-hosted registry
curl -I https://registry.example.com/v2/
# Expected: 200 or 401

Create a Vigilmon monitor for your primary apko target registry:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://ghcr.io/v2/ (or your registry's v2 endpoint).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status codes: 200, 401.
  6. Label: apko target registry v2 API.
  7. Click Save.

Step 3: Monitor APK Package Repositories

Unlike Dockerfile-based builders, apko directly fetches APK packages from configured repositories at build time. If the package repository is unreachable, every apko build and apko publish fails — even if the target registry is healthy:

# Test Alpine Linux CDN repository
curl -I https://dl-cdn.alpinelinux.org/alpine/edge/main/
# Expected: 200

# Test Alpine community repository
curl -I https://dl-cdn.alpinelinux.org/alpine/edge/community/
# Expected: 200

# Test Wolfi OS package repository
curl -I https://packages.wolfi.dev/os/
# Expected: 200

# Test a private APK repository
curl -I https://apk.example.com/stable/main/
# Expected: 200

# Verify APKINDEX is downloadable (the package index)
curl -s https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz | tar -tz | head -3
# Expected: DESCRIPTION APKINDEX

Add a Vigilmon monitor for each APK repository in your apko configuration:

Alpine Linux CDN:

  1. Add Monitor → HTTP.
  2. URL: https://dl-cdn.alpinelinux.org/alpine/edge/main/.
  3. Check interval: 120 seconds.
  4. Expected status: 200.
  5. Label: Alpine APK repository (edge/main).
  6. Click Save.

Wolfi OS repository:

  1. Add Monitor → HTTP.
  2. URL: https://packages.wolfi.dev/os/.
  3. Check interval: 120 seconds.
  4. Expected status: 200.
  5. Label: Wolfi APK repository.
  6. Click Save.

Repeat for each additional repository listed in your apko YAML files.

Step 4: Monitor the Registry TCP Port

Add a TCP monitor to detect network-layer failures independent of HTTP application failures:

# Test TCP connectivity to GHCR
nc -zv ghcr.io 443
# Expected: succeeded

# Test self-hosted registry TCP
nc -zv registry.example.com 443
# Expected: succeeded
  1. Add Monitor → TCP.
  2. Host: ghcr.io (or your registry host).
  3. Port: 443.
  4. Check interval: 60 seconds.
  5. Label: apko registry TCP (443).
  6. Click Save.

Step 5: Monitor TLS Certificates

apko uses HTTPS for all registry and APK repository communication. Expired TLS certificates on either the registry or the APK repository break all apko builds. The repository signing key infrastructure is separate from the TLS certificate, but both matter:

# Check TLS certificate on GHCR
echo | openssl s_client -connect ghcr.io:443 2>/dev/null | \
  openssl x509 -noout -dates
# notAfter=Dec 31 00:00:00 2026 GMT

# Check TLS certificate on Alpine CDN
echo | openssl s_client -connect dl-cdn.alpinelinux.org:443 2>/dev/null | \
  openssl x509 -noout -dates

# Check TLS certificate on Wolfi repository
echo | openssl s_client -connect packages.wolfi.dev:443 2>/dev/null | \
  openssl x509 -noout -dates

# Extract all unique HTTPS hostnames from apko YAML files
grep -rh 'https://' configs/ *.yaml 2>/dev/null \
  | grep -oP 'https://[a-z0-9.-]+' \
  | sed 's|https://||' | sort -u

Add SSL certificate monitors for each hostname:

  1. Add Monitor → SSL Certificate.
  2. Domain: ghcr.io (repeat for APK repositories and the registry).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Key domains to monitor: the OCI registry, Alpine CDN (dl-cdn.alpinelinux.org), Wolfi repository (packages.wolfi.dev), and any private APK repository endpoints.

Step 6: Heartbeat Monitoring for apko Rebuild Pipelines

The main value of apko in production is continuous, scheduled rebuilds — rebuilding images on a schedule ensures they always contain the latest patched APK versions. These rebuild pipelines run on cron schedules and have no HTTP endpoint. Heartbeat monitoring catches when they stop running.

Scheduled Image Rebuild in GitHub Actions

A weekly rebuild that picks up the latest APK security patches:

# .github/workflows/apko-rebuild.yml
name: Weekly image rebuild

on:
  schedule:
    - cron: '0 3 * * 1'  # Monday 03:00 UTC
  workflow_dispatch:

jobs:
  rebuild:
    runs-on: ubuntu-latest
    permissions:
      packages: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Install apko
        uses: chainguard-dev/actions/setup-chainctl@main
        # Or install apko binary directly:
        # run: |
        #   APKO_VERSION=$(curl -s https://api.github.com/repos/chainguard-dev/apko/releases/latest | jq -r .tag_name)
        #   curl -L "https://github.com/chainguard-dev/apko/releases/download/${APKO_VERSION}/apko_linux_amd64.tar.gz" | tar xz
        #   sudo mv apko /usr/local/bin/

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and publish with apko
        run: |
          apko publish configs/base-image.yaml ghcr.io/${{ github.repository_owner }}/base:latest
          apko publish configs/app-image.yaml ghcr.io/${{ github.repository_owner }}/app:latest

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/abc123

Set up the heartbeat monitor:

  1. In Vigilmon → Add Monitor → Cron Heartbeat.
  2. Expected interval: 10080 minutes (weekly rebuild — 7 × 1440 minutes; use 11520 minutes for a 7-day-plus-1-day grace period).
  3. Copy the heartbeat URL and paste it as the final workflow step.
  4. Label: apko weekly image rebuild.
  5. Click Save.

Daily Security Patch Rebuild

A daily rebuild to catch critical CVE patches in APK packages:

#!/bin/bash
# daily-apko-rebuild.sh

CONFIGS=(
  "configs/base-image.yaml:ghcr.io/myorg/base:latest"
  "configs/app-image.yaml:ghcr.io/myorg/app:latest"
  "configs/worker-image.yaml:ghcr.io/myorg/worker:latest"
)

FAILED=0
for CONFIG_PAIR in "${CONFIGS[@]}"; do
  CONFIG="${CONFIG_PAIR%%:*}"
  IMAGE="${CONFIG_PAIR##*:}"

  if ! apko publish "${CONFIG}" "${IMAGE}"; then
    echo "FAIL: ${CONFIG} → ${IMAGE}"
    FAILED=1
  else
    echo "OK: ${IMAGE}"
  fi
done

if [ "${FAILED}" -eq 0 ]; then
  curl -s https://vigilmon.online/heartbeat/def456
else
  echo "One or more builds failed — not pinging heartbeat"
  exit 1
fi
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes (daily rebuild with 60-minute grace).
  3. Label: Daily apko security patch rebuild.
  4. Click Save.

SBOM Generation Validation

apko generates SBOMs automatically. Validate that SBOM artifacts exist alongside pushed images:

#!/bin/bash
# validate-apko-sbom.sh

IMAGE="ghcr.io/myorg/app:latest"

# Check that the image exists
if ! crane digest "${IMAGE}" > /dev/null 2>&1; then
  echo "FAIL: ${IMAGE} not reachable"
  exit 1
fi

# Check that the SBOM attestation exists (apko attaches it as an OCI referrer)
if crane ls ghcr.io/myorg/app 2>/dev/null | grep -q "sbom"; then
  echo "OK: SBOM artifacts present"
  curl -s https://vigilmon.online/heartbeat/ghi789
else
  echo "WARN: No SBOM tag found for ${IMAGE}"
  # Still ping — SBOM presence varies by workflow configuration
  curl -s https://vigilmon.online/heartbeat/ghi789
fi

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | OCI registry v2 API | Non-200/401 or timeout | All apko publish runs will fail at the push step; check registry status page | | Alpine APK repository | Non-200 or timeout | apko cannot fetch packages; check https://status.alpinelinux.org/ | | Wolfi APK repository | Non-200 or timeout | Wolfi-based image builds fail; check https://packages.wolfi.dev/ | | Registry TCP | Connection refused | Network connectivity issue to registry; check firewall rules | | TLS certificate (registry) | < 30 days | Rotate registry certificate; apko returns TLS errors on expiry | | TLS certificate (APK repo) | < 30 days | APK repository TLS renewal needed; contact repo maintainers | | Weekly rebuild heartbeat | Missed ping | Check GitHub Actions workflow; review APK fetch errors or registry auth failures in logs | | Daily rebuild heartbeat | Missed ping | Check cron job; verify APK repositories are reachable; check for package signing key issues |

Common apko + Registry Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | OCI registry down during publish | v2 API monitor fires; apko publish fails at the image push step | | Alpine CDN outage | Alpine APK repository monitor fires; package fetch fails before image assembly | | Wolfi package repository unreachable | Wolfi APK repository monitor fires; Wolfi-based builds fail | | TLS certificate expired on registry | SSL monitor fires at 30-day threshold; apko returns TLS errors | | TLS certificate expired on APK repository | SSL monitor fires; apko cannot fetch APKINDEX or packages | | Scheduled weekly rebuild stopped running | Weekly heartbeat goes silent; images stop picking up CVE patches | | APK package signing key compromised or rotated | Rebuilds fail with keyring validation errors; heartbeat fires | | Private APK repository unreachable | Custom APK repo monitor fires; private package builds fail | | Registry auth token expired in CI | Rebuild heartbeat fires; apko publish fails with 401 | | APK package removed from repository | Rebuild fails with "package not found"; heartbeat fires |

apko makes minimal, SBOM-backed OCI images reproducible from declarative YAML — but that reproducibility depends on APK package repositories and OCI registries that can fail independently. Vigilmon gives you external visibility into the repositories apko pulls from, the registries it pushes to, and the scheduled rebuild pipelines that keep your images patched. When the Wolfi repository has an outage during your nightly rebuild, when an APK repository TLS certificate approaches expiry, or when a weekly scheduled rebuild silently stops running, Vigilmon alerts you before outdated images end up in production.

Start monitoring your apko build pipelines in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles