tutorial

Monitoring Hatchet with Vigilmon: API Health, Dashboard Uptime, Worker Connectivity & Workflow Queue Depth

How to monitor Hatchet distributed task queue and workflow orchestration with Vigilmon — API health checks, dashboard uptime, worker TCP connectivity, and SSL certificate monitoring.

Hatchet is the self-hostable distributed task queue and workflow orchestration engine built for teams who need Temporal-style durable execution without managed-service lock-in. It powers DAG-based workflows, fan-out/fan-in patterns, priority queues, and rate-limited job execution across Go, Python, and TypeScript workers. When Hatchet's API server goes down, workers can no longer poll for tasks, DAG edges stop triggering downstream steps, and new workflow runs are rejected. When the dashboard is unavailable, engineers lose visibility into running workflows, stuck steps, and failed runs. Vigilmon gives you external visibility into Hatchet's availability before your workers start timing out.

What You'll Build

  • An HTTP monitor on the Hatchet API health endpoint to catch server failures
  • An HTTP monitor on the Hatchet dashboard to detect UI-level outages
  • A TCP monitor on the gRPC port used by SDK workers
  • An SSL certificate monitor for your Hatchet domain
  • An alerting setup that separates API failures from worker connectivity issues

Prerequisites

  • A self-hosted Hatchet instance with a network-reachable domain (e.g., https://hatchet.example.com)
  • The Hatchet API and dashboard exposed via HTTPS
  • The gRPC engine port accessible (default 7070)
  • A free account at vigilmon.online

Step 1: Understand Hatchet's Service Architecture

A self-hosted Hatchet deployment exposes three key network surfaces:

| Surface | Default port | Role | |---|---|---| | REST API | 8080 (HTTP/HTTPS) | Used by the dashboard, CLI, and management plane | | gRPC engine | 7070 | Used by Go, Python, and TypeScript SDK workers for task polling | | Dashboard (Vite/Next.js) | 8080 (same origin) | Browser UI for workflow inspection and management |

External monitoring focuses on the API health endpoint and the dashboard — both are HTTP/HTTPS reachable. The gRPC engine port gets a TCP check to confirm worker connectivity at the network layer.

Step 2: Create a Vigilmon HTTP Monitor for the API Health Endpoint

Hatchet exposes a liveness endpoint at /api/v1/health. A 200 response confirms the API server is up and connected to its database:

curl https://hatchet.example.com/api/v1/health
# Returns: {"status":"ok"}
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://hatchet.example.com/api/v1/health.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok (present in the health response body).
  7. Label: Hatchet API health.
  8. Click Save.

This monitor catches:

  • Hatchet API server crashes or OOM kills
  • Database connectivity failures (Postgres down → API returns non-200)
  • Kubernetes pod restarts for the Hatchet engine container
  • Misconfigured ingress blocking API traffic after upgrades

Alert sensitivity: Set to trigger after 1 consecutive failure. When the API is down, no new workflows can be started and no worker can register or poll tasks.

Step 3: Monitor the Hatchet Dashboard

The Hatchet dashboard provides the browser interface for inspecting workflow runs, step statuses, and worker connections. Monitoring it separately from the API catches frontend-specific failures:

curl https://hatchet.example.com
# Returns HTML with "Hatchet" in the page title
  1. Add Monitor → HTTP.
  2. URL: https://hatchet.example.com.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Hatchet.
  7. Label: Hatchet dashboard.
  8. Click Save.

Why monitor the dashboard separately? The API and dashboard share the same origin in most Hatchet deployments, but they can fail independently. A broken asset build, a CDN misconfiguration, or a static file server crash can take down the dashboard while the API health endpoint continues to return 200. Separate monitors make root cause analysis faster.

Step 4: Monitor the gRPC Engine Port via TCP Check

Hatchet workers connect to the gRPC engine on port 7070 to register, poll for tasks, and report step completion. A TCP check confirms the port is accepting connections at the network level, which is the first thing to verify when workers report connectivity failures:

# Test gRPC port connectivity
nc -zv hatchet.example.com 7070
  1. Add Monitor → TCP.
  2. Host: hatchet.example.com.
  3. Port: 7070.
  4. Check interval: 60 seconds.
  5. Label: Hatchet gRPC engine port.
  6. Click Save.

Why TCP monitoring for the gRPC port? Hatchet SDK workers — whether written in Go (github.com/hatchet-dev/hatchet/pkg/client), Python (hatchet_sdk), or TypeScript (@hatchet-dev/typescript-sdk) — all dial this gRPC port. If it's unreachable, workers stall without executing any tasks. A TCP check detects port-level failures (firewall rules, load balancer misconfiguration, Hatchet engine process crash) without requiring gRPC tooling.

Step 5: Monitor the REST API Workflows Endpoint

Beyond the health check, monitoring a lightweight authenticated endpoint validates that the API is functional end-to-end and not just returning a cached ok response:

curl -H "Authorization: Bearer <token>" \
  https://hatchet.example.com/api/v1/workflows
# Returns JSON list of registered workflows
  1. Add Monitor → HTTP.
  2. URL: https://hatchet.example.com/api/v1/workflows.
  3. Method: GET.
  4. Headers: Authorization: Bearer <your-api-token>.
  5. Check interval: 2 minutes.
  6. Response timeout: 15 seconds.
  7. Expected status: 200.
  8. Keyword: rows (present in paginated list responses).
  9. Label: Hatchet workflow list API.
  10. Click Save.

Store the API token as a Vigilmon monitor secret rather than a plain header value to keep credentials out of your monitor configuration UI.

Step 6: Monitor SSL Certificates

An expired TLS certificate breaks HTTPS access to the dashboard, the API, and gRPC-TLS connections from workers:

openssl s_client -connect hatchet.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: hatchet.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

If your Hatchet deployment uses a separate domain for the gRPC engine (e.g., grpc.hatchet.example.com), add a second SSL monitor for that domain.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | API health | Non-200 or ok missing | Check Hatchet engine pod logs; verify Postgres connectivity; inspect ingress | | Dashboard | Non-200 or Hatchet missing | Check static file serving; verify frontend build; inspect CDN or ingress config | | gRPC port TCP | Connection refused | Engine process down; workers stalled; check pod restarts and firewall rules | | Workflow list API | Non-200 or keyword missing | API degraded beyond health check; database query failure; auth middleware broken | | SSL certificate | < 30 days to expiry | Renew certificate; test dashboard HTTPS and worker gRPC-TLS after renewal |

Alert after: 1 consecutive failure for the API health and gRPC port monitors. 2 consecutive failures for the dashboard and workflow list monitors.

Common Hatchet Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Hatchet engine pod OOM killed | API health monitor fires; gRPC TCP monitor fires; alert within 60 s | | Postgres database down | API health returns non-200; API health monitor fires | | gRPC port blocked by firewall rule | TCP monitor fires immediately; API health may still pass | | Dashboard asset build broken | Dashboard monitor fires; API health still returns ok | | SSL certificate expires | SSL monitor alerts at 30-day threshold; browser and worker TLS connections fail | | Hatchet upgrade migration fails | API health returns non-200 or 500 during failed migration | | Worker container crash | Workers stop polling; not directly caught — monitor separately with worker heartbeat | | DNS misconfiguration | All monitors fire simultaneously | | Rate limiter misconfiguration | Workflow runs rejected; not caught by basic health check | | Queue depth spike | Not directly catchable via external HTTP monitoring |

Monitoring Worker Health

Vigilmon monitors the Hatchet service endpoints — the orchestration plane that workers connect to. Whether your Go, Python, or TypeScript workers are actually polling and executing tasks is a separate concern:

  • Hatchet's built-in dashboard: The worker tab in the Hatchet UI shows connected workers, their last heartbeat time, and active slot counts.
  • Prometheus metrics: Hatchet exposes Prometheus-compatible metrics. Monitor hatchet_worker_slots_available and hatchet_step_run_failed_total for worker-level alerting.
  • Worker heartbeat pattern: Have each worker emit a custom HTTP request to a Vigilmon heartbeat URL on every successful task completion — silence triggers an alert when workers stop processing.

Vigilmon catches infrastructure-level failures (API down, port unreachable, certificate expired). Application-level worker health requires SDK-level instrumentation or Hatchet's built-in metrics.

Hatchet is the orchestration backbone for durable distributed workflows — when the engine goes down, every worker stops executing tasks, DAG steps stop triggering, and in-flight business processes stall silently. Vigilmon gives you external visibility into Hatchet's availability that doesn't depend on your workers reporting status: API health, dashboard uptime, gRPC port connectivity, and SSL certificate expiry, so you know the moment the cluster becomes unreachable and can restore workflow execution before retry budgets are exhausted.

Start monitoring Hatchet in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles