tutorial

Monitoring Shipyard with Vigilmon

Shipyard spins up ephemeral preview environments for every pull request — but if the orchestration API or dynamic DNS routing breaks, no one gets their preview. Here's how to monitor Shipyard end-to-end with Vigilmon.

Shipyard makes pull request review faster by giving every PR its own live environment — a fully functional, dynamically routed preview that developers, QA teams, and stakeholders can click through before merge. But that automation depends on Shipyard's orchestration API staying up, Docker or Kubernetes being reachable, dynamic DNS routing working correctly, and GitHub webhooks arriving reliably. Vigilmon monitors each of those layers so you know the moment ephemeral environments stop being created — or stop being accessible.

What You'll Set Up

  • HTTP uptime monitor for the Shipyard orchestration API (port 3000)
  • Wildcard TLS certificate expiry alert for dynamic subdomains
  • TCP monitor for PostgreSQL connectivity
  • HTTP monitor for GitHub/GitLab webhook endpoint
  • Docker/Kubernetes cluster health check
  • Environment provisioning heartbeat

Prerequisites

  • Self-hosted Shipyard running Go + PostgreSQL
  • Shipyard orchestration API accessible on port 3000
  • Dynamic subdomain routing configured (e.g. *.shipyard.yourdomain.com)
  • A free Vigilmon account

Step 1: Monitor the Shipyard Orchestration API

The Shipyard API on port 3000 handles all environment lifecycle operations: provisioning new environments when PRs open, routing traffic to existing environments, and tearing down environments when PRs close. If this service is unavailable, no new environments are created and existing subdomain routing may break.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your Shipyard API URL: https://shipyard.yourdomain.com (or http://your-server:3000).
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

Shipyard exposes a health endpoint at /health that reports internal state:

https://shipyard.yourdomain.com/health

Use this endpoint for the monitor rather than the root URL — it checks database connectivity and cluster reachability in a single probe.

Step 2: Wildcard TLS Certificate Expiry Alert

Shipyard routes each preview environment to a unique subdomain (feature-branch.shipyard.yourdomain.com), and all those subdomains share a single wildcard certificate (*.shipyard.yourdomain.com). If this wildcard certificate expires, every preview environment becomes inaccessible with a TLS error simultaneously.

  1. Open the HTTP monitor for your Shipyard API.
  2. Enable Monitor SSL certificate under the SSL section.
  3. Set Alert when certificate expires in less than 30 days.

Use a 30-day window for the wildcard certificate (longer than the standard 21 days) because wildcard certificate renewal is more complex — it typically requires DNS-01 ACME challenges rather than HTTP-01, and propagation time must be factored in.

Also add a second dedicated SSL monitor targeting a test subdomain that exercises the wildcard:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter https://health.shipyard.yourdomain.com (create a static health subdomain for this purpose).
  3. Enable Monitor SSL certificate with a 30 day alert window.
  4. Click Save.

This confirms the wildcard is actually being served for subdomains, not just the apex domain.

Step 3: Monitor PostgreSQL Connectivity

Shipyard stores environment state, PR metadata, and subdomain assignments in PostgreSQL. A database outage causes Shipyard to lose track of which environments exist, leading to orphaned containers and broken routing.

Add a TCP monitor for PostgreSQL:

  1. Click Add Monitor → TCP Port.
  2. Enter your database host and port: your-server:5432.
  3. Set Check interval to 1 minute.
  4. Click Save.

Step 4: Monitor GitHub/GitLab Webhook Endpoint

Shipyard's environment lifecycle is event-driven: PR opened → environment created; PR updated → environment rebuilt; PR closed → environment torn down. These events arrive as webhooks from GitHub or GitLab. If Shipyard's webhook receiver is unhealthy, PRs can be merged without their environments ever being cleaned up — leading to resource exhaustion from orphaned environments.

Add an HTTP monitor for the webhook endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter https://shipyard.yourdomain.com/webhooks/github (adjust path for your deployment).
  3. Set Expected HTTP status to 200 or 405 (webhook endpoints often return 405 on GET).
  4. Set Check interval to 1 minute.
  5. Click Save.

You can also verify webhook delivery health from the GitHub side by checking your repository's webhook delivery log under Settings → Webhooks — look for any failed deliveries and investigate connectivity.

Step 5: Monitor Docker/Kubernetes Cluster Connectivity

Shipyard provisions environments by orchestrating Docker Compose or Kubernetes — if the container runtime is unreachable, environment creation fails with errors that don't surface in the Shipyard dashboard.

For Docker-based Shipyard deployments

Add a TCP monitor for the Docker daemon socket exposed over TCP (if configured):

  1. Click Add Monitor → TCP Port.
  2. Enter your-server:2376 (Docker TLS port) or 2375 (unencrypted — not recommended for production).
  3. Set Check interval to 1 minute.

Alternatively, expose a Docker health probe via a small sidecar service:

# Minimal Docker health check service
#!/bin/bash
docker info > /dev/null 2>&1 && echo "ok" || echo "error"

For Kubernetes-based Shipyard deployments

Add an HTTP monitor for the Kubernetes API server:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter https://your-k8s-api:6443/healthz.
  3. Set Check interval to 2 minutes.
  4. Click Save.

Step 6: Environment Provisioning Heartbeat

The most important signal for Shipyard isn't whether the API responds to health checks — it's whether environments are actually being created successfully when PRs open. Use a Vigilmon cron heartbeat to confirm provisioning is completing.

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to your team's typical PR frequency (e.g. 120 minutes if your team opens at least one PR every two hours during business hours).
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_ID.
  4. Add a ping to Shipyard's post-provisioning callback:
package hooks

import (
    "net/http"
    "log"
)

func OnEnvironmentReady(env *Environment) {
    // Notify team that environment is ready
    notifyTeam(env)

    // Ping Vigilmon heartbeat
    resp, err := http.Get("https://vigilmon.online/heartbeat/YOUR_ID")
    if err != nil {
        log.Printf("heartbeat ping failed: %v", err)
    } else {
        resp.Body.Close()
    }
}

If no environments are successfully provisioned within the expected window during active working hours, Vigilmon alerts — catching infrastructure issues that don't manifest as API errors.

Step 7: Environment Cleanup Health Check

Orphaned environments — preview environments whose PRs have been closed but whose containers were never torn down — waste cluster resources and can exhaust node capacity. Monitor cleanup job health with another heartbeat:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to your cleanup job frequency (e.g. 30 minutes).
  3. Copy the heartbeat URL for the cleanup job.
  4. Add a ping to your cleanup job completion handler:
#!/bin/bash
# Shipyard cleanup job
shipyard cleanup --older-than 1h

# Ping heartbeat on success
curl -s https://vigilmon.online/heartbeat/YOUR_CLEANUP_ID

Also add a scheduled probe that counts currently running environments and alerts if the count exceeds your expected maximum — a spike may indicate cleanup jobs are failing silently.

Step 8: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
  2. For the orchestration API monitor, set Consecutive failures before alert to 2.
  3. For PostgreSQL and the Docker/Kubernetes cluster, alert on 1 failure — infrastructure outages require immediate action.
  4. For wildcard TLS certificate, alert at 30 days to give time for DNS-01 renewal.

Recommended alert routing for a development team:

| Monitor | Channel | Threshold | |---|---|---| | Orchestration API | #dev-alerts | 2 failures | | Wildcard TLS certificate | #dev-infra | 30 days | | PostgreSQL TCP | #on-call | 1 failure | | GitHub webhook endpoint | #dev-alerts | 2 failures | | Docker/Kubernetes cluster | #on-call | 1 failure | | Provisioning heartbeat | #dev-alerts | 1 missed beat | | Cleanup heartbeat | #dev-infra | 1 missed beat |

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP API | https://shipyard.yourdomain.com/health | Orchestration service outage | | SSL wildcard cert | *.shipyard.yourdomain.com | All preview environments becoming inaccessible | | TCP | PostgreSQL port 5432 | Environment state database outage | | HTTP | Webhook endpoint | PR events not triggering environments | | TCP/HTTP | Docker daemon or Kubernetes API | Container runtime unavailable | | Cron heartbeat | Provisioning heartbeat URL | Environments failing to provision | | Cron heartbeat | Cleanup heartbeat URL | Orphaned environment resource leak |

Shipyard's value is making every PR reviewable in a live environment — the moment that flow breaks, developers lose their feedback loop and reviewers lose context. With Vigilmon watching every layer from the orchestration API to the container runtime, you'll catch issues before they silently block the entire team's preview workflow.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles