tutorial

Monitoring Go Container Builds with ko and Vigilmon: Registry Health, Build Pipeline Heartbeats, TLS Certificate Alerts & Image Availability Checks

How to use ko and Vigilmon together to monitor Go container image build pipelines — registry API health checks, OCI image push validation, TLS certificate expiry, and heartbeat monitoring for ko-based CI/CD automation.

ko is Google's container image builder for Go applications — it compiles Go binaries and packages them directly into OCI-compliant container images without Dockerfiles or a Docker daemon. Used in projects like Knative, Tekton, and Sigstore, ko is the standard for building distroless Go images in cloud-native CI/CD pipelines. When the container registry ko pushes to goes down, when the Go module proxy becomes unreachable, or when a nightly image build pipeline silently stops running, teams discover the failure only when a deployment references a stale or missing image. Vigilmon gives you external visibility into the OCI registry endpoints ko depends on, TLS certificate health for those registries, and heartbeat monitoring for the ko-based build pipelines that produce your container images.

What You'll Build

  • HTTP monitors on the OCI registry API endpoints ko pushes images to
  • SSL certificate monitors for each registry's HTTPS endpoint
  • TCP monitors on registry ports for network-layer health
  • Heartbeat monitors for ko-based CI/CD build and publish pipelines
  • Alerting runbook mapping ko failure modes to registry and build health issues

Prerequisites

  • ko installed (go install github.com/ko-build/ko/cmd/ko@latest or via Homebrew)
  • One or more OCI-compatible container registries (GHCR, GCR, ECR, Docker Hub, or self-hosted)
  • A free account at vigilmon.online

Step 1: Understand the ko + Registry Health Surface

ko communicates with OCI registries using the same HTTP API as Docker and crane. When you run ko build or ko apply, ko:

  1. Compiles the Go binary for the target platform
  2. Fetches a base image (usually gcr.io/distroless/static or a configured base) from a registry
  3. Pushes the resulting image to the target registry

Every one of these steps depends on registry availability:

# Set the target registry
export KO_DOCKER_REPO=ghcr.io/myorg

# Build and push a Go application image
ko build ./cmd/myapp
# Expected: ghcr.io/myorg/myapp@sha256:...

# Build without pushing (local Docker daemon)
ko build --local ./cmd/myapp

# Build for multiple platforms
ko build --platform linux/amd64,linux/arm64 ./cmd/myapp

# Check that the registry is reachable before running ko
curl -I https://ghcr.io/v2/
# Expected: 401 (registry is up, auth required)

# Validate a pushed image exists
crane digest ghcr.io/myorg/myapp:latest
# Expected: sha256:...

The key failure signals to monitor:

  1. Registry v2 API availability (for both base image pulls and image pushes)
  2. TLS certificate validity on the target and base image registries
  3. Go module proxy availability (for go build inside ko)
  4. Build pipeline heartbeats for ko-based CI/CD jobs

Step 2: Monitor the Registry v2 API

The /v2/ endpoint is the primary health signal for any OCI-compliant registry. ko checks this endpoint when pushing images and when pulling base images:

# GitHub Container Registry
curl -I https://ghcr.io/v2/
# Expected: 401 (auth required — registry is up)

# Google Container Registry
curl -I https://gcr.io/v2/
# Expected: 401

# AWS ECR public
curl -I https://public.ecr.aws/v2/
# Expected: 200 or 401

# Docker Hub
curl -I https://registry-1.docker.io/v2/
# Expected: 401

# Self-hosted registry
curl -I https://registry.example.com/v2/
# Expected: 200 or 401

Create a Vigilmon monitor for your primary ko registry:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://ghcr.io/v2/ (or your registry's v2 endpoint).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status codes: 200, 401.
  6. Label: ko target registry v2 API.
  7. Click Save.

If ko uses a custom base image registry, add a separate monitor for the base image source. For example, if you're using gcr.io/distroless:

  1. Add Monitor → HTTP.
  2. URL: https://gcr.io/v2/.
  3. Expected status codes: 200, 401.
  4. Label: Distroless base image registry.
  5. Click Save.

Step 3: Monitor the Registry TCP Port

Add a TCP monitor to detect network-layer failures independent of the HTTP application layer:

# Test TCP connectivity to GHCR
nc -zv ghcr.io 443
# Expected: succeeded

# Test self-hosted registry
nc -zv registry.example.com 443
# Expected: succeeded
  1. Add Monitor → TCP.
  2. Host: ghcr.io (or your registry host).
  3. Port: 443.
  4. Check interval: 60 seconds.
  5. Label: ko registry TCP (443).
  6. Click Save.

A TCP success paired with an HTTP failure isolates the issue to the application layer rather than the network.

Step 4: Monitor TLS Certificates

ko uses HTTPS for all registry communication. An expired TLS certificate on the target registry or the base image registry breaks every ko build run, across all developers and all CI pipelines simultaneously:

# Check TLS certificate on GHCR
echo | openssl s_client -connect ghcr.io:443 2>/dev/null | \
  openssl x509 -noout -dates
# notAfter=Dec 31 00:00:00 2026 GMT

# Verify TLS with crane
crane ls ghcr.io/myorg/myapp 2>&1 | head -3
# If TLS is broken: "x509: certificate has expired or is not yet valid"

# Find all unique registry hosts ko pushes to
grep -rE 'KO_DOCKER_REPO|ko build|ko apply' .github/ Makefile \
  | grep -oP '[a-z0-9.-]+\.(io|com|example\.com)[/:]' \
  | sed 's|[/:]||' | sort -u
# Output: ghcr.io gcr.io registry.example.com
  1. Add Monitor → SSL Certificate.
  2. Domain: ghcr.io (repeat for each registry domain ko uses).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Add an SSL certificate monitor for every registry hostname that appears in your ko configuration and .github/workflows/ files.

Step 5: Heartbeat Monitoring for ko Build Pipelines

ko is most commonly run inside CI/CD pipelines — ko build on every commit, or ko apply to deploy updated Kubernetes manifests. These pipelines have no external HTTP endpoint. Vigilmon's heartbeat monitor detects when they stop running at the expected frequency.

Release Image Build Pipeline

A GitHub Actions workflow that builds and pushes images on every merge to main:

# .github/workflows/ko-build.yml
name: Build and push image

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      packages: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod

      - name: Install ko
        uses: ko-build/setup-ko@v0.7

      - name: Build and push
        env:
          KO_DOCKER_REPO: ghcr.io/${{ github.repository_owner }}
        run: ko build --bare ./cmd/myapp

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/abc123

Set up the heartbeat monitor in Vigilmon:

  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 1440 minutes (build runs on each main branch push, expected at least daily).
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.
  4. Paste it as the last step in your workflow.
  5. Label: ko image build — main branch.
  6. Click Save.

Scheduled Multi-Platform Build

A cron-triggered workflow that builds images for all target platforms nightly:

# .github/workflows/nightly-multiplatform.yml
name: Nightly multiplatform build

on:
  schedule:
    - cron: '0 2 * * *'

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      packages: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
      - uses: ko-build/setup-ko@v0.7

      - name: Build multiplatform images
        env:
          KO_DOCKER_REPO: ghcr.io/${{ github.repository_owner }}
        run: |
          ko build --platform linux/amd64,linux/arm64 ./cmd/myapp
          ko build --platform linux/amd64,linux/arm64 ./cmd/worker

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/def456
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes (runs at 02:00 UTC daily — 1440 minutes plus 60-minute grace period).
  3. Label: Nightly multiplatform ko build.
  4. Click Save.

ko apply Deployment Pipeline

When using ko apply to apply Kubernetes manifests with freshly built images:

#!/bin/bash
# deploy.sh

# Build and apply Kubernetes manifests with ko-resolved image references
KO_DOCKER_REPO=ghcr.io/myorg ko apply -f config/

# Signal successful deployment to Vigilmon
curl -s https://vigilmon.online/heartbeat/ghi789
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: matches your deployment cadence (e.g., 60 minutes for hourly deploys).
  3. Label: ko apply deployment.
  4. Click Save.

Step 6: Validate Image Availability After Build

Use a validation script to confirm that ko-pushed images remain pullable. Run this on a schedule as a synthetic smoke test:

#!/bin/bash
# validate-ko-images.sh — run every 30 minutes via cron

REGISTRY="ghcr.io/myorg"
IMAGES=(
  "myapp"
  "worker"
  "scheduler"
)

FAILED=0
for IMAGE in "${IMAGES[@]}"; do
  if ! crane digest "${REGISTRY}/${IMAGE}:latest" > /dev/null 2>&1; then
    echo "FAIL: ${REGISTRY}/${IMAGE}:latest is unreachable"
    FAILED=1
  else
    echo "OK: ${REGISTRY}/${IMAGE}:latest"
  fi
done

if [ "${FAILED}" -eq 0 ]; then
  curl -s https://vigilmon.online/heartbeat/jkl012
else
  echo "Image validation failed — not pinging heartbeat"
  exit 1
fi

Schedule via cron:

# /etc/cron.d/ko-image-validation
*/30 * * * * deploy /opt/scripts/validate-ko-images.sh >> /var/log/ko-validation.log 2>&1
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 35 minutes.
  3. Label: ko image availability check.
  4. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | Registry v2 API | Non-200/401 or timeout | Confirm with curl -I https://ghcr.io/v2/; all ko builds will fail; check registry status page | | Base image registry | Non-200/401 | ko cannot pull the distroless base image; ko build fails at the layer fetch step | | Registry TCP | Connection refused | Check network connectivity and firewall rules to the registry host | | TLS certificate | < 30 days | Rotate certificate immediately; verify with crane ls registry/myapp after renewal | | Build pipeline heartbeat | Missed ping | Check GitHub Actions workflow logs; look for go build errors, registry auth failures, or module proxy timeouts | | Nightly build heartbeat | Missed ping | Check scheduled workflow run; review platform-specific build failures in logs | | Image availability heartbeat | Missed ping | Run crane digest manually for each image; investigate if images were overwritten or registry GC ran |

Common ko + Registry Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Target registry API down | v2 API monitor fires; all ko build pushes fail | | Base image registry (GCR) unavailable | Base image registry monitor fires; ko cannot assemble the image layer | | TLS certificate expired on target registry | SSL monitor fires at 30-day threshold; ko returns TLS errors | | Build pipeline workflow disabled or broken | Build heartbeat goes silent after expected interval | | Go module proxy unreachable during go build | Build heartbeat fires; builds fail before the image push step | | GHCR rate limit or outage | v2 API returns 429 or 5xx; ko push step fails | | Image deleted or overwritten in registry | Availability validation heartbeat fires | | Multi-platform build cron misconfigured | Nightly heartbeat goes silent; arm64 images stop being updated | | Registry auth token expired in long CI job | Build heartbeat fires; registry push step fails with 401 | | ko base image digest pinned to deleted tag | Base image fetch fails; SSL and TCP monitors stay green, isolating the issue to the image reference |

ko makes Go container builds reproducible, minimal, and fast — but those builds are only as reliable as the OCI registry they push to and pull from. Vigilmon gives you external visibility into registry availability, TLS health, and the CI/CD pipelines that run ko on every commit. When GHCR has an outage, when a base image registry certificate expires, or when a nightly multi-platform build silently stops running, Vigilmon alerts you before stale images reach production.

Start monitoring your ko build pipelines in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles