tutorial

Monitoring APK Package Builds with Melange and Vigilmon: Build Pipeline Heartbeats, Repository Health, TLS Certificate Alerts & Package Availability Checks

How to use Melange and Vigilmon together to monitor APK package build pipelines — build heartbeat monitoring, upstream source availability, signing key health, APK repository endpoint checks, and TLS certificate expiry alerts for Melange-based package automation.

Melange is Chainguard's declarative APK package builder — it builds Alpine-compatible APK packages from a YAML pipeline definition, with full reproducibility, built-in SBOM generation, and cryptographic signing built in. Designed to work alongside apko in the Chainguard / Wolfi supply chain ecosystem, Melange is how security-focused teams build their own minimal APK packages for inclusion in distroless container images. When the upstream source a Melange pipeline fetches is unreachable, when the signing key used to produce APK packages is misconfigured, or when a scheduled Melange build silently stops running, teams lose the ability to produce patched, signed packages — and downstream apko image builds either fail or produce images with outdated software. Vigilmon gives you external visibility into the upstream sources Melange fetches, the APK repository servers that distribute built packages, TLS certificate health for those endpoints, and heartbeat monitoring for the scheduled Melange build pipelines that keep your package supply chain current.

What You'll Build

  • HTTP monitors on upstream source endpoints Melange fetches tarballs and archives from
  • HTTP monitors on APK repository endpoints where built packages are served
  • SSL certificate monitors for upstream source and repository endpoints
  • TCP monitors on APK repository ports for network-layer health
  • Heartbeat monitors for Melange-based scheduled package build pipelines
  • Alerting runbook mapping Melange failure modes to build and repository health issues

Prerequisites

  • Melange installed (brew install melange or via the official release binary)
  • APK signing keys configured (melange keygen to generate melange.rsa and melange.rsa.pub)
  • An APK repository server or storage bucket where built packages are published
  • A free account at vigilmon.online

Step 1: Understand the Melange + Source and Repository Health Surface

Melange reads a YAML pipeline file and performs these network-dependent operations:

  1. Fetches the upstream source (a source tarball, Git repository, or release archive) from a specified URL
  2. Builds the APK package using the defined pipeline (configure, build, install steps)
  3. Signs the package using the configured RSA signing key
  4. Outputs the APK to a local directory for publishing to an APK repository

Publishing the built APK to a repository is usually a separate step — typically s3 sync, gsutil cp, or rsync to an APK repository server. Both the fetch and the publish can fail:

# Example Melange YAML (mypackage.yaml)
# package:
#   name: mypackage
#   version: 1.2.3
#   description: My custom package
#   target-architecture: [x86_64, aarch64]
# environment:
#   contents:
#     packages: [build-base, openssl-dev]
# pipeline:
#   - uses: fetch
#     with:
#       uri: https://github.com/example/mypackage/archive/refs/tags/v${{package.version}}.tar.gz
#       expected-sha256: abc123...
#   - uses: autoconf/configure
#   - uses: autoconf/make
#   - uses: autoconf/make-install

# Build a package
melange build mypackage.yaml \
  --signing-key melange.rsa \
  --arch x86_64,aarch64 \
  --out-dir ./packages

# Check if the upstream source URL is reachable
curl -I https://github.com/example/mypackage/archive/refs/tags/v1.2.3.tar.gz
# Expected: 200 or 302

# Verify the built APK exists
ls packages/x86_64/mypackage-1.2.3*.apk
# Expected: packages/x86_64/mypackage-1.2.3-r0.apk

The key failure signals to monitor:

  1. Upstream source URL availability (GitHub releases, tarballs, archive servers)
  2. APK repository server availability (where built packages are distributed)
  3. TLS certificate validity on source URLs and the repository server
  4. Build pipeline heartbeats for scheduled Melange builds

Step 2: Monitor Upstream Source Endpoints

Melange pipelines specify exact URLs for upstream source tarballs. If a GitHub release is deleted, a CDN serving a source archive goes down, or a tarball URL rotates after a version bump without updating the Melange YAML, the build fails at the fetch step:

# Check a GitHub release archive
curl -I https://github.com/example/mypackage/archive/refs/tags/v1.2.3.tar.gz
# Expected: 200 or 302 (GitHub CDN redirect)

# Check a direct tarball URL
curl -I https://download.example.com/mypackage-1.2.3.tar.gz
# Expected: 200

# Follow redirects to confirm the final URL is reachable
curl -IL https://github.com/example/mypackage/archive/refs/tags/v1.2.3.tar.gz 2>&1 | grep -E 'HTTP|Location'
# Expected: series of 302s ending in 200

# Extract all upstream source URLs from Melange YAML files
grep -rh 'uri:' pipelines/ *.yaml 2>/dev/null \
  | grep -oP 'https://[^\s"]+' | sort -u

Add HTTP monitors for each unique upstream source domain:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://github.com (the base domain — monitors if GitHub is reachable).
  3. Check interval: 120 seconds.
  4. Expected status: 200.
  5. Label: Upstream source — GitHub.
  6. Click Save.

For packages with non-GitHub source URLs:

  1. Add Monitor → HTTP.
  2. URL: the exact tarball or release URL used in the Melange YAML uri: field.
  3. Expected status codes: 200, 302.
  4. Label: Upstream source — mypackage tarball.
  5. Click Save.

Step 3: Monitor the APK Repository Server

After Melange builds the APK, it needs to be published to an APK repository so that apko and other consumers can install it. Monitor the repository endpoint where packages are served:

# Test the APK repository HTTP endpoint
curl -I https://apk.example.com/stable/main/
# Expected: 200

# Test the APKINDEX file (the package index)
curl -s https://apk.example.com/stable/main/x86_64/APKINDEX.tar.gz | tar -tz | head
# Expected: DESCRIPTION APKINDEX

# Test a self-hosted Nginx APK repository
curl -I https://packages.internal.example.com/
# Expected: 200

# Verify a specific package is present in the repository
curl -I https://apk.example.com/stable/main/x86_64/mypackage-1.2.3-r0.apk
# Expected: 200

Create a Vigilmon monitor for the APK repository:

  1. Add Monitor → HTTP.
  2. URL: https://apk.example.com/stable/main/.
  3. Check interval: 120 seconds.
  4. Expected status: 200.
  5. Label: APK repository (stable/main).
  6. Click Save.

If your APK repository is served from a cloud storage bucket with a public endpoint (S3, GCS):

  1. Add Monitor → HTTP.
  2. URL: https://my-apk-bucket.s3.amazonaws.com/ or https://storage.googleapis.com/my-apk-bucket/.
  3. Expected status codes: 200, 403 (403 on the bucket root is normal when directory listing is disabled; check a known package path for 200).
  4. Label: APK repository storage bucket.
  5. Click Save.

Step 4: Monitor APK Repository TCP Port

Add a TCP monitor on the APK repository server for network-layer health checks:

# Test TCP connectivity to the APK repository server
nc -zv apk.example.com 443
# Expected: succeeded

# For repositories on non-standard ports
nc -zv apk.example.com 8443
  1. Add Monitor → TCP.
  2. Host: apk.example.com.
  3. Port: 443.
  4. Check interval: 60 seconds.
  5. Label: APK repository TCP (443).
  6. Click Save.

Step 5: Monitor TLS Certificates

Melange fetches sources over HTTPS, and the APK repository serves packages over HTTPS. Expired TLS certificates on either endpoint break builds and package installations:

# Check TLS certificate on the APK repository
echo | openssl s_client -connect apk.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates
# notAfter=Dec 31 00:00:00 2026 GMT

# Check TLS on a self-hosted repository
echo | openssl s_client -connect packages.internal.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# Verify certificate on upstream source server
echo | openssl s_client -connect download.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# Find all unique HTTPS hosts referenced in Melange pipelines
grep -rh 'uri:\|https://' pipelines/ *.yaml 2>/dev/null \
  | grep -oP 'https://[a-z0-9.-]+' \
  | sed 's|https://||' | sort -u

Add SSL certificate monitors:

  1. Add Monitor → SSL Certificate.
  2. Domain: apk.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for each unique hostname that appears in your Melange pipeline uri: fields and the APK repository endpoint. An expired certificate on the APK repository breaks downstream apko build runs that try to install packages — the blast radius extends well beyond just the Melange build itself.

Step 6: Heartbeat Monitoring for Melange Build Pipelines

Melange builds run on schedules — daily or weekly rebuilds ensure packages pick up the latest upstream patches. These scheduled jobs have no HTTP endpoint. Heartbeat monitoring detects when they silently stop running.

Daily Package Rebuild in GitHub Actions

A daily rebuild workflow that produces fresh APK packages with the latest upstream source:

# .github/workflows/melange-build.yml
name: Daily package rebuild

on:
  schedule:
    - cron: '0 1 * * *'  # 01:00 UTC daily
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4

      - name: Install Melange
        run: |
          MELANGE_VERSION=$(curl -s https://api.github.com/repos/chainguard-dev/melange/releases/latest | jq -r .tag_name)
          curl -L "https://github.com/chainguard-dev/melange/releases/download/${MELANGE_VERSION}/melange_linux_amd64.tar.gz" | tar xz
          sudo mv melange /usr/local/bin/

      - name: Import signing key
        run: echo "${{ secrets.MELANGE_SIGNING_KEY }}" > melange.rsa

      - name: Build packages
        run: |
          for YAML in pipelines/*.yaml; do
            melange build "${YAML}" \
              --signing-key melange.rsa \
              --arch x86_64,aarch64 \
              --out-dir ./packages
          done

      - name: Publish to APK repository
        run: |
          # Sync to S3 (adjust for your repository backend)
          aws s3 sync ./packages s3://my-apk-bucket/stable/main/ \
            --delete --acl public-read

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/abc123

Set up the heartbeat monitor:

  1. In Vigilmon → Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes (daily rebuild at 01:00 UTC with 60-minute grace).
  3. Copy the heartbeat URL and paste it as the final workflow step.
  4. Label: Melange daily package rebuild.
  5. Click Save.

Multi-Package Build Script

For building multiple packages in a local or self-hosted pipeline:

#!/bin/bash
# melange-rebuild-all.sh

SIGNING_KEY="/etc/melange/melange.rsa"
PACKAGES_DIR="/var/www/apk/stable/main"
PIPELINES_DIR="/opt/melange/pipelines"

FAILED=0
for PIPELINE in "${PIPELINES_DIR}"/*.yaml; do
  PACKAGE_NAME=$(basename "${PIPELINE}" .yaml)
  echo "Building ${PACKAGE_NAME}..."

  if ! melange build "${PIPELINE}" \
    --signing-key "${SIGNING_KEY}" \
    --arch x86_64,aarch64 \
    --out-dir "${PACKAGES_DIR}"; then
    echo "FAIL: ${PACKAGE_NAME}"
    FAILED=1
  else
    echo "OK: ${PACKAGE_NAME}"
  fi
done

# Regenerate the APKINDEX after all builds
apk index -o "${PACKAGES_DIR}/x86_64/APKINDEX.tar.gz" "${PACKAGES_DIR}/x86_64"/*.apk
apk index -o "${PACKAGES_DIR}/aarch64/APKINDEX.tar.gz" "${PACKAGES_DIR}/aarch64"/*.apk

if [ "${FAILED}" -eq 0 ]; then
  curl -s https://vigilmon.online/heartbeat/def456
else
  echo "One or more builds failed — not pinging heartbeat"
  exit 1
fi

Schedule via cron:

# /etc/cron.d/melange-rebuild
0 2 * * * melange /opt/scripts/melange-rebuild-all.sh >> /var/log/melange-build.log 2>&1
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes.
  3. Label: Melange nightly package build.
  4. Click Save.

Package Availability Validation

After each build run, verify that newly built packages are accessible from the APK repository:

#!/bin/bash
# validate-apk-packages.sh — run every 30 minutes via cron

REPO_URL="https://apk.example.com/stable/main/x86_64"
PACKAGES=("mypackage" "mylib" "mytool")

FAILED=0
for PKG in "${PACKAGES[@]}"; do
  # Check if the package appears in the APKINDEX
  if curl -s "${REPO_URL}/APKINDEX.tar.gz" | tar xz -O APKINDEX 2>/dev/null | grep -q "^P:${PKG}$"; then
    echo "OK: ${PKG} found in APKINDEX"
  else
    echo "FAIL: ${PKG} not found in APKINDEX"
    FAILED=1
  fi
done

if [ "${FAILED}" -eq 0 ]; then
  curl -s https://vigilmon.online/heartbeat/ghi789
else
  echo "Package validation failed — not pinging heartbeat"
  exit 1
fi
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 35 minutes.
  3. Label: APK package availability check.
  4. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | Upstream source URL | Non-200/302 or timeout | Verify the source tarball URL in the Melange YAML; check if the upstream release was renamed or deleted | | GitHub source availability | Non-200 or timeout | GitHub outage; all Melange pipelines fetching from GitHub will fail | | APK repository HTTP | Non-200 or timeout | Downstream apko build runs and apk add commands will fail to install packages | | APK repository TCP | Connection refused | Check web server or reverse proxy process on the repository host | | TLS certificate (repository) | < 30 days | Renew repository TLS certificate; expired cert breaks all downstream consumers | | TLS certificate (upstream source) | < 30 days | Upstream server TLS renewal needed; Melange fetch step will fail on expiry | | Daily rebuild heartbeat | Missed ping | Check GitHub Actions or cron job logs; look for upstream fetch failures, signing key errors, or S3 sync failures | | Package availability heartbeat | Missed ping | Run APKINDEX check manually; verify packages were published to the repository after the last build |

Common Melange + Repository Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Upstream source tarball URL deleted or moved | Upstream source HTTP monitor fires; melange build fails at the fetch step | | GitHub release page unavailable | GitHub HTTP monitor fires; all pipelines fetching from GitHub fail | | APK repository server down | APK repository HTTP and TCP monitors fire; downstream apko builds and apk add calls fail | | TLS certificate expired on APK repository | SSL monitor fires at 30-day threshold; all HTTPS consumers of the repository fail | | Signing key corrupted or missing in CI | Build heartbeat fires; APK packages built without valid signatures are rejected by apk | | S3 bucket ACL change blocks public access | APK repository HTTP monitor fires with 403; consumers cannot fetch packages | | Scheduled rebuild cron disabled or broken | Daily heartbeat goes silent; packages stop being updated with upstream patches | | APKINDEX not regenerated after build | Package availability heartbeat fires; new package versions invisible to consumers | | Upstream release checksum mismatch in YAML | Build heartbeat fires; melange build fails at checksum verification step | | Build runner OOM during compilation | Build heartbeat fires; large C/C++ package builds exhausted memory before APK was produced |

Melange brings reproducibility, SBOM generation, and cryptographic signing to APK package builds — but those builds depend on upstream sources that can be deleted, APK repository servers that can go down, and scheduled CI jobs that can silently stop running. Vigilmon gives you external visibility into the upstream sources Melange fetches, the APK repositories that distribute built packages, and the scheduled pipelines that keep your package supply chain current and patched. When an upstream release tarball is deleted after a project restructuring, when an APK repository TLS certificate expires and breaks downstream apko builds, or when a nightly rebuild pipeline stops running and packages drift behind on CVE patches, Vigilmon alerts you before the impact reaches production.

Start monitoring your Melange build pipelines in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles