tutorial

How to Monitor Tilt Dev Environments with Vigilmon

Tilt is a Kubernetes-native development tool that manages your entire local dev environment — watching files, rebuilding containers, and syncing changes to y...

Tilt is a Kubernetes-native development tool that manages your entire local dev environment — watching files, rebuilding containers, and syncing changes to your cluster automatically. It's the inner development loop for teams running microservices. But Tilt also introduces a class of problems unique to managed dev environments: services that look healthy in the Tilt UI but are silently broken, builds that complete without errors but deploy stale code, or dependent services that restart in the wrong order.

In this tutorial you'll set up external monitoring for your Tilt-managed development and staging services using Vigilmon — free tier, no credit card required.

Why Tilt environments need external monitoring

Tilt's built-in UI shows resource status, logs, and build state. It's excellent for active development. But it doesn't tell you what a client outside your cluster actually sees:

  • Port-forward drop — Tilt's local_resource or k8s_resource port-forward dies silently after a cluster restart; the Tilt UI shows the resource as healthy, but localhost:8080 returns connection refused
  • Hot reload broken — a file sync succeeds but the in-process hot reload fails; old code serves requests while Tilt reports everything green
  • Service dependency ordering — a microservice starts before its database is ready and enters a crash loop backoff; Tilt shows it as running after the 3rd restart, but the service never successfully initialized
  • Ingress misconfiguration — a Tilt-managed Kubernetes resource update changes an Ingress rule; other services in the mesh stop receiving traffic
  • Staging drift — your Tiltfile manages a staging cluster, not just local; a broken deploy goes unnoticed until someone manually tests it

For staging environments especially, external monitoring is essential — Tilt manages the deploy, but it can't tell you whether what deployed actually works.

What you'll need

  • Tilt v0.30+ installed and a running Tiltfile
  • Services exposed externally (via Ingress, LoadBalancer, or Tilt's serve_dir / local_resource)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose services and add health endpoints

Tilt manages your services, but for external monitoring you need each service to expose a /health endpoint and be reachable from outside the cluster.

Here's a minimal Tiltfile with health-check-aware resource configuration:

# Tiltfile

# Build and deploy the API service
docker_build('your-registry/api', './api',
  live_update=[
    sync('./api/src', '/app/src'),
    run('cd /app && npm run build', trigger=['./api/src']),
  ]
)

k8s_yaml(['./k8s/api-deployment.yaml', './k8s/api-service.yaml'])
k8s_resource('api',
  port_forwards=['8080:3000'],
  readiness_probe=probe(
    http_get=http_get_action(port=3000, path='/health'),
    initial_delay_secs=5,
    period_secs=10,
  ),
  labels=['backend'],
)

# Build and deploy the worker service
docker_build('your-registry/worker', './worker')
k8s_yaml('./k8s/worker-deployment.yaml')
k8s_resource('worker',
  resource_deps=['api'],  # worker waits for api to be ready
  labels=['backend'],
)

# Local resource: run db migrations before services start
local_resource('db-migrations',
  cmd='kubectl exec -n dev deploy/postgres -- \
    psql -U appuser -d appdb -f /migrations/latest.sql',
  resource_deps=['postgres'],
  labels=['infra'],
)

Ensure your API service has a /health route:

// api/src/routes/health.js
app.get('/health', async (req, res) => {
  try {
    await db.query('SELECT 1');  // verify db connectivity
    res.json({ status: 'ok', version: process.env.APP_VERSION });
  } catch (err) {
    res.status(503).json({ status: 'error', message: err.message });
  }
});

Verify external reachability:

# If using port-forward
curl http://localhost:8080/health

# If using an Ingress in staging
curl https://api-staging.example.com/health

Step 2: Set up HTTP monitoring in Vigilmon

For staging environments managed by Tilt, add each externally accessible service to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the monitor type
  3. Set the URL to your staging service: https://api-staging.example.com/health
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

Monitoring local port-forwards

For local development, port-forwards run on localhost and aren't reachable from Vigilmon's external probes. Use Vigilmon's heartbeat monitoring instead: run a local script that probes localhost and pings Vigilmon on success.

#!/bin/bash
# local-monitor.sh — run this on your dev machine on a cron

HEALTH_URL="http://localhost:8080/health"
VIGILMON_HEARTBEAT="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

response=$(curl -sf --max-time 5 "$HEALTH_URL")
if echo "$response" | grep -q '"status":"ok"'; then
  curl -fsS "$VIGILMON_HEARTBEAT"
fi

# Add to crontab
*/2 * * * * /path/to/local-monitor.sh

Create a Heartbeat monitor in Vigilmon with a 5-minute interval. If your local port-forward dies, the heartbeat stops arriving and you get an alert within 5 minutes.

Step 3: Monitor Tilt builds with a heartbeat

Tilt rebuilds happen continuously. A build that appears to succeed but deploys broken code is a real failure mode — especially with live_update. Monitor your CI/staging builds with a heartbeat that fires only on verified deployments:

#!/bin/bash
# deploy-staging.sh — called after Tilt deploys to staging

tilt ci --timeout=5m

# Verify the health endpoint
for i in $(seq 1 12); do
  STATUS=$(curl -sf https://api-staging.example.com/health \
    | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))")
  if [ "$STATUS" = "ok" ]; then
    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
    echo "Deploy verified, heartbeat sent"
    exit 0
  fi
  echo "Waiting for service... ($i/12)"
  sleep 10
done

echo "Deploy failed health check — heartbeat NOT sent"
exit 1

In Vigilmon, create a Heartbeat monitor with an interval slightly longer than your typical deploy time (e.g., 30 minutes for a deploy that usually takes 5). If a deploy stalls or fails the health check, you'll know within one interval.

Step 4: Monitor multiple microservices

Tilt often manages a fleet of microservices. Add a monitor for each service's health endpoint and group them in Vigilmon:

| Service | URL | Check interval | |---------|-----|----------------| | API | https://api-staging.example.com/health | 1 minute | | Worker | https://worker-staging.example.com/health | 1 minute | | Gateway | https://gateway-staging.example.com/health | 1 minute | | Scheduler | https://scheduler-staging.example.com/health | 2 minutes |

Group them on a Vigilmon status page labeled "Staging" so your team can see at a glance which microservice is broken after a Tilt deploy.

Step 5: Configure alert channels

Configure Vigilmon to alert your team when a staging service breaks:

Slack integration

  1. In Vigilmon, go to Alert Channels → Add Channel → Webhook
  2. Enter your team's Slack incoming webhook URL
  3. Assign to all staging monitors

When a staging service fails, your Slack channel gets an immediate notification with the service name and failure time — without anyone needing to check the Tilt UI or run kubectl.

Email for on-call

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your engineering team's email
  3. Assign to production-adjacent staging monitors

Vigilmon sends this payload when a service goes down:

{
  "monitor_name": "api-staging /health",
  "status": "down",
  "url": "https://api-staging.example.com/health",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 300
}

Use this in a webhook handler to automatically trigger a Tilt rebuild:

# Auto-recover webhook handler
SERVICE=$(echo "$PAYLOAD" | python3 -c "import sys,json; print(json.load(sys.stdin)['monitor_name'].split()[0])")
tilt trigger "$SERVICE"

Step 6: Integrate monitoring into your Tiltfile

Add monitoring status as a local resource in Tilt so developers can see it alongside their services:

# Tiltfile — add Vigilmon status check as a local resource
local_resource(
  'vigilmon-status',
  serve_cmd='''bash -c "
    while true; do
      curl -sf https://api.vigilmon.online/v1/monitors \
        -H 'Authorization: Bearer $VIGILMON_API_KEY' \
        | python3 -c \\"
import sys, json
monitors = json.load(sys.stdin)
for m in monitors:
    icon = '✓' if m['status'] == 'up' else '✗'
    print(f\\"{icon} {m[\\\"name\\\"]} — {m[\\\"status\\\"]}\\")
      \\"
      sleep 60
    done
  "''',
  labels=['monitoring'],
)

This displays Vigilmon monitor statuses directly in the Tilt UI — developers see monitoring health alongside build and deploy health in one view.

Step 7: Create a staging status page

Create a Vigilmon status page for your staging environment:

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name it "Staging Environment"
  3. Add monitors for all Tilt-managed services
  4. Group by team or functional area (e.g., "API Layer", "Background Workers")
  5. Publish the URL and pin it in your team's Slack channel

When a staging deploy breaks, anyone can check the status page without needing Tilt access or Kubernetes credentials.

Monitoring reference

| Monitor | Type | What it catches | |---------|------|-----------------| | https://api-staging.example.com/health | HTTP | Broken deploys, crash loops, routing issues | | https://worker-staging.example.com/health | HTTP | Background worker failures | | Deploy pipeline | Heartbeat | Stalled Tilt builds, failed deploys | | Local port-forward | Heartbeat | Port-forward drops on dev machine |

Useful Tilt and Kubernetes commands when alerts fire:

# Check Tilt resource status
tilt get uiresource

# View Tilt session logs for a specific resource
tilt logs -f api

# Force a rebuild of a specific resource
tilt trigger api

# Check pod status in the cluster
kubectl get pods -n dev -l app=api

# View recent pod events
kubectl describe pod -n dev -l app=api | tail -30

# Check readiness probe failures
kubectl get events -n dev \
  --field-selector reason=Unhealthy \
  --sort-by='.lastTimestamp'

# Run Tilt in CI mode (blocks until all resources ready or fails)
tilt ci --timeout=10m

What's next

  • Production parity — if your Tiltfile also manages production deploys via tilt up in CI, extend the same monitors to production endpoints with tighter alert thresholds
  • SSL certificate monitoring — Vigilmon alerts you before TLS certificates on your staging ingress expire, preventing debugging sessions caused by silent renewal failures
  • Heartbeat for nightly staging resets — if your staging environment resets nightly (database refresh, cluster recreation), add a heartbeat that confirms the reset completed and all services came back up

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles