tutorial

Monitoring Tilt with Vigilmon

Tilt is your local Kubernetes dev loop — but when the dashboard goes dark, builds stall, or pods crash-loop, you lose visibility instantly. Here's how to monitor every critical Tilt signal with Vigilmon.

Tilt is an open source local Kubernetes development tool that dramatically accelerates the inner-loop development cycle for microservices. It automatically syncs code changes to running pods (without full image rebuilds), displays live logs from every service in a terminal UI, and provides a web dashboard at port 10350 for viewing build status and service health. But Tilt is orchestrating a complex system — a local Kubernetes cluster, Docker image builds, port-forwards, and file sync processes — and any layer can fail silently. Vigilmon gives you continuous visibility into Tilt's web dashboard, build pipeline, pod health, and port-forward stability so you know the moment your local dev environment starts degrading.

What You'll Set Up

  • HTTP uptime monitor for the Tilt web dashboard (port 10350)
  • Cron heartbeats for image build health and live update sync
  • TCP monitors for port-forward connections
  • Kubernetes cluster connectivity monitoring

Prerequisites

  • Tilt running locally with a Tiltfile configured
  • Tilt web dashboard accessible at http://localhost:10350
  • A free Vigilmon account

Why Monitoring Tilt Matters

Tilt orchestrates a dozen moving parts simultaneously. When something goes wrong, the failure mode is often silent:

  • Dashboard down: If the Tilt web server on port 10350 crashes, you lose all visibility into build status, service logs, and alerts — you may not notice until a deployment silently fails.
  • Image builds hanging: A Docker build that gets stuck (waiting for network, hitting a cache layer bug, or running a slow RUN step) blocks deployment without surfacing an obvious alert.
  • Live update sync failures: Tilt's file sync sidesteps full image rebuilds for hot reloading — when sync fails, changes stop appearing in running pods and require a full rebuild to recover.
  • Pod crash loops: A service stuck in CrashLoopBackOff stops serving traffic and fills Tilt's log panel with noise, but doesn't page anyone.
  • Port-forward instability: Tilt maintains kubectl port-forward tunnels for local service access. These reconnect silently but can drop requests during reconnection.
  • Kubernetes cluster loss: If the local cluster (kind, k3d, minikube) goes down, all Tilt operations fail — but Tilt continues running, showing stale state.

Step 1: Monitor the Tilt Web Dashboard

The Tilt web dashboard at port 10350 is your primary visibility surface for build status, service health, and logs. When it's down, your dev loop is essentially flying blind.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Set URL to http://localhost:10350 (or the LAN IP of your dev machine if monitoring from a separate host).
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

If Vigilmon's external probes can't reach your localhost, run a local monitoring agent on the dev machine that posts to Vigilmon's inbound heartbeat API instead — see Step 3 for the pattern.

For Tilt's JSON API health endpoint (more reliable than the UI root):

http://localhost:10350/api/view

This returns a JSON view of all Tilt resources — a 200 response confirms Tilt's API engine is healthy, not just the UI assets.

Step 2: Monitor Kubernetes Cluster Connectivity

Tilt's core dependency is a running Kubernetes cluster. If kind, k3d, or minikube goes down, Tilt loses the ability to deploy, sync, or port-forward — but the Tilt process itself keeps running, showing stale resource states.

Add a TCP monitor for the Kubernetes API server:

  1. Click Add Monitor → TCP Port.
  2. Set Host to localhost (or 127.0.0.1).
  3. Set Port to 6443 (default Kubernetes API port for kind/k3d) or 8443 for minikube.
  4. Set Check interval to 1 minute.
  5. Click Save.

Alternatively, use an HTTP monitor for the Kubernetes healthz endpoint:

https://localhost:6443/healthz

Note: this endpoint requires a client certificate or skipping TLS verification (--insecure). Use the TCP monitor if you prefer not to configure cert-based Vigilmon probes.

Step 3: Heartbeat for Image Build Health

Tilt rebuilds Docker images when relevant files change. A build that hangs for over a threshold (say, 10 minutes when a typical build takes 2 minutes) indicates a stuck layer pull, a network issue, or a Dockerfile problem.

Add a heartbeat that Tilt (or a build wrapper) emits after each successful build:

#!/bin/bash
# tilt-build-probe.sh — run every 2 minutes, check Tilt's API for build status

BUILD_STATUS=$(curl -sf http://localhost:10350/api/view | \
  jq '[.uiResources[] | select(.status.buildHistory[0].error != null)] | length')

if [ "$BUILD_STATUS" = "0" ]; then
  # No build errors — all resources built successfully
  curl -sf https://vigilmon.online/heartbeat/tilt-build-xyz
fi
  1. Add a Cron Heartbeat in Vigilmon with interval 10 minutes.
  2. Schedule the probe script every 2 minutes: */2 * * * * /path/to/tilt-build-probe.sh

If a build hangs or errors persist for more than 10 minutes, the heartbeat stops arriving and Vigilmon alerts you.

Step 4: Heartbeat for Live Update Sync Health

Tilt's live update feature syncs file changes directly into running pods, bypassing a full image rebuild. When sync fails (file permission issues, pod restarts, sync configuration errors), developers think their changes are deployed when they're not.

Poll Tilt's API to check for sync errors:

#!/bin/bash
# tilt-sync-probe.sh — run every 2 minutes

SYNC_ERRORS=$(curl -sf http://localhost:10350/api/view | \
  jq '[.uiResources[] | select(.status.runtimeStatus == "error" or .status.buildHistory[0].error != null)] | length')

if [ "$SYNC_ERRORS" = "0" ]; then
  curl -sf https://vigilmon.online/heartbeat/tilt-sync-xyz
fi
  1. Add a Cron Heartbeat in Vigilmon with interval 5 minutes.
  2. Run the script every minute.

Step 5: Monitor Port-Forward Connections

Tilt maintains kubectl port-forward tunnels for each service you expose locally. These reconnect automatically, but during reconnection, requests to the local port get connection-refused errors.

Monitor each critical port-forward with a TCP probe:

API gateway / backend service (e.g. port 8080):

  1. Click Add Monitor → TCP Port.
  2. Set Host to localhost, Port to 8080.
  3. Set Check interval to 1 minute.
  4. Click Save.

Database (e.g. PostgreSQL at port 5432 via port-forward):

Repeat for port 5432.

Frontend dev server (e.g. port 3000):

Repeat for port 3000.

For services that serve HTTP, use an HTTP monitor instead of TCP — this verifies the service itself is healthy, not just that the port-forward tunnel is connected:

http://localhost:8080/health

Step 6: Pod Health Monitoring via Tilt API

Tilt detects CrashLoopBackOff pods and surfaces them in the UI — but only if you're watching the dashboard. Add a heartbeat that fires only when no pods are in a crash loop:

#!/bin/bash
# tilt-pod-health.sh — run every 2 minutes

# Check for any CrashLoopBackOff pods via kubectl
CRASH_COUNT=$(kubectl get pods --all-namespaces \
  --field-selector=status.phase!=Running 2>/dev/null | \
  grep -c CrashLoop || echo 0)

if [ "$CRASH_COUNT" = "0" ]; then
  curl -sf https://vigilmon.online/heartbeat/tilt-pods-xyz
fi
  1. Add a Cron Heartbeat in Vigilmon with interval 5 minutes.
  2. Schedule the script every minute.

This gives you paged alerting for crash-looping pods without requiring a separate Kubernetes monitoring stack.

Step 7: Tiltfile Parse Health

The Tiltfile is re-evaluated every time it changes. A syntax error or invalid resource definition in the Tiltfile blocks Tilt from updating any service until it's fixed.

Detect Tiltfile errors via the Tilt API:

#!/bin/bash
# tilt-tiltfile-health.sh — run every minute

TILTFILE_ERROR=$(curl -sf http://localhost:10350/api/view | \
  jq '.uiSession.status.tiltfileKey // "" | length')

if [ "$TILTFILE_ERROR" = "0" ]; then
  curl -sf https://vigilmon.online/heartbeat/tiltfile-health-xyz
fi

Add a Cron Heartbeat with interval 3 minutes and run the script every minute.

Step 8: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon.
  2. Add Slack for team-visible development environment alerts.
  3. For the Tilt dashboard monitor, set Consecutive failures before alert to 2 — the Tilt process can briefly restart when the Tiltfile is reloaded.
  4. For TCP port-forward monitors, set Consecutive failures to 3 — port-forwards reconnect within seconds and transient drops are normal.
  5. For pod health and sync heartbeats, set 1 missed ping — these always warrant immediate attention.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP uptime | http://localhost:10350 | Tilt dashboard down | | HTTP uptime | http://localhost:10350/api/view | Tilt API engine failure | | TCP port | localhost:6443 | Kubernetes cluster down | | Build health heartbeat | Heartbeat URL (10m interval) | Stuck or failing builds | | Sync health heartbeat | Heartbeat URL (5m interval) | Live update sync failure | | TCP port | localhost:8080 | Port-forward connection lost | | Pod health heartbeat | Heartbeat URL (5m interval) | CrashLoopBackOff pods | | Tiltfile heartbeat | Heartbeat URL (3m interval) | Tiltfile parse error |

Tilt eliminates the manual deploy-wait-test cycle for Kubernetes development — but only when its entire stack is healthy. With Vigilmon watching the dashboard, build pipeline, pod health, and port-forwards, you'll catch the moment your local dev environment starts degrading instead of losing an hour debugging stale state.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles