How to Monitor Shipwright Build Services with Vigilmon

Shipwright is a Kubernetes-native framework for building container images — it abstracts over Kaniko, BuildKit, Buildah, and other build strategies behind a unified API. When Shipwright is healthy your development pipeline runs smoothly; when it breaks, every CI/CD job that depends on image builds stalls silently.

In this tutorial you'll set up external uptime monitoring for your Shipwright build infrastructure using Vigilmon — free tier, no credit card required.

Why Shipwright needs external monitoring

Shipwright runs as a Kubernetes controller and a set of webhook admission controllers. Both can fail in ways that internal Kubernetes probes won't catch:

• Controller crash loop — the Shipwright build controller pod restarts repeatedly; new Build resources are accepted but never reconciled, and build queues silently stall
• Webhook admission failure — the mutating or validating webhook becomes unresponsive; Kubernetes starts rejecting all Build and BuildRun resource submissions with 500 errors
• API server connectivity loss — the Shipwright controller loses contact with the Kubernetes API, stops processing events, but continues to report as Running
• BuildRun executor saturation — all build pods are consumed by stuck runs, and new builds queue indefinitely with no timeout alert
• Node pressure killing build pods — OOM events or node evictions silently terminate long-running image builds mid-flight

External monitoring gives you a ground-truth signal that your Shipwright infrastructure is actually processing builds — not just showing green in kubectl get pods.

What you'll need

• A Kubernetes cluster with Shipwright installed
• A simple HTTP health endpoint exposed from your build infrastructure (or a build-trigger service)
• A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose a health endpoint from your build infrastructure

Shipwright's controller exposes metrics and a health endpoint on port 8383 by default. You can forward this to a NodePort or create a dedicated monitoring service.

Check the controller health endpoint directly:

# Port-forward the controller metrics port
kubectl port-forward -n shipwright-build \
  deployment/shipwright-build-controller 8383:8383

# Verify health
curl http://localhost:8383/healthz
# ok

For production monitoring, expose it via a ClusterIP service and use an Ingress or a lightweight monitoring proxy:

# shipwright-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: shipwright-health
  namespace: shipwright-build
spec:
  type: NodePort
  selector:
    app: shipwright-build-controller
  ports:
    - name: metrics
      port: 8383
      targetPort: 8383
      nodePort: 30383
      protocol: TCP

Apply it:

kubectl apply -f shipwright-health-service.yaml

Verify the health endpoint is reachable externally:

curl http://<node-ip>:30383/healthz
# ok

Alternatively, if you have an ingress controller in front of Shipwright, create a dedicated health-check route in your BuildStrategy namespace that calls the controller's healthz endpoint and proxies it through HTTPS.

Step 2: Set up HTTP monitoring in Vigilmon

With the health endpoint reachable, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS as the type
3. Set the URL to your Shipwright health endpoint, e.g. http://your-node.example.com:30383/healthz
4. Set the check interval to 1 minute
5. Under Expected response, set:
   • Status code: 200
   • Response body contains: ok
6. Save the monitor

Vigilmon probes from multiple geographic regions. If the Shipwright controller becomes unresponsive — even while its pod shows Running in Kubernetes — the monitor will fire an alert.

Why multi-region consensus matters for build infrastructure

Shipwright failures often look like slow degradation, not instant crashes. Multi-region consensus means Vigilmon won't alert on a single slow probe response — it waits for multiple regions to confirm the failure before triggering an incident. This dramatically reduces false alarms from transient network latency while still catching real controller outages within seconds.

Step 3: Monitor the webhook admission port

Shipwright registers a validating webhook for Build and BuildRun resources. If the webhook becomes unresponsive, Kubernetes will reject all build submissions. Monitor the webhook port directly:

# shipwright-webhook-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: shipwright-webhook-monitor
  namespace: shipwright-build
spec:
  type: NodePort
  selector:
    app: shipwright-build-webhook
  ports:
    - name: webhook
      port: 8443
      targetPort: 8443
      nodePort: 30443
      protocol: TCP

Then in Vigilmon:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your node hostname and port: your-node.example.com / 30443
4. Save the monitor

A TCP check on the webhook port catches admission controller failures before your build pipeline ever sees a rejection error.

Step 4: Create a build-probe heartbeat

Beyond controller health, you want to know that Shipwright can actually complete a build. Set up a lightweight heartbeat build that runs on a schedule:

# probe-build.sh — run this as a Kubernetes CronJob
#!/bin/bash
set -euo pipefail

# Submit a minimal probe build
kubectl apply -f - <<EOF
apiVersion: shipwright.io/v1beta1
kind: BuildRun
metadata:
  generateName: probe-build-
  namespace: build-monitoring
spec:
  build:
    name: probe-build
  serviceAccountName: build-bot
EOF

# Wait up to 3 minutes for completion
kubectl wait buildrun -n build-monitoring \
  -l shipwright.io/build=probe-build \
  --for=condition=Succeeded \
  --timeout=180s

# Ping Vigilmon heartbeat URL on success
curl -fsS "https://hb.vigilmon.online/your-heartbeat-token"

In Vigilmon, create a Heartbeat monitor with a 10-minute interval. If the probe build stops completing within 10 minutes, you'll get an alert.

Step 5: Configure alert channels

1. In Vigilmon, go to Alert Channels → Add Channel
2. Choose Email or Webhook
3. For webhooks, enter your Slack, PagerDuty, or Opsgenie URL
4. Assign all Shipwright monitors to the channel

Example Vigilmon webhook payload on a Shipwright controller outage:

{
  "monitor_name": "Shipwright Controller Health",
  "status": "down",
  "url": "http://your-node.example.com:30383/healthz",
  "started_at": "2024-03-10T08:15:00Z",
  "duration_seconds": 90
}

Use this payload in your on-call handler to immediately run triage commands:

# Quick Shipwright triage
kubectl get pods -n shipwright-build
kubectl describe deployment shipwright-build-controller -n shipwright-build
kubectl get buildruns -A --sort-by=.metadata.creationTimestamp | tail -20
kubectl get events -n shipwright-build --sort-by=.metadata.creationTimestamp | tail -30

Step 6: Create a public status page

If your Shipwright cluster serves multiple teams or external CI pipelines, a status page keeps stakeholders informed without requiring Kubernetes access.

1. In Vigilmon, go to Status Pages → New Status Page
2. Name it "Build Infrastructure Status"
3. Add your monitors: Shipwright Controller, Webhook, Heartbeat
4. Publish the page

You'll get a URL like https://status.vigilmon.online/your-page to share with development teams.

Putting it all together

| Monitor | Type | What it catches | |---------|------|-----------------| | http://node:30383/healthz | HTTP | Controller crashes, reconciliation failures | | node:30443 | TCP | Webhook admission controller failures | | Probe build heartbeat | Heartbeat | End-to-end build pipeline stalls |

Key diagnostic commands when Vigilmon alerts:

# Controller status
kubectl get pods -n shipwright-build -l app=shipwright-build-controller

# Recent build activity
kubectl get buildruns -A --sort-by=.metadata.creationTimestamp

# Stuck builds
kubectl get buildruns -A | grep -v Succeeded | grep -v Failed

# Webhook registration
kubectl get validatingwebhookconfigurations | grep shipwright
kubectl get mutatingwebhookconfigurations | grep shipwright

What's next

• SSL certificate monitoring — if your Shipwright webhook uses TLS (it does by default), Vigilmon can alert you before the certificate expires and causes build rejections
• Build registry monitoring — add HTTP monitors for your container registry endpoints; Shipwright can't push images if the registry is down
• Multi-cluster coverage — if you run Shipwright across staging and production clusters, create separate monitors for each and group them on your status page

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.