Monitoring Kestra with Vigilmon

Kestra is a modern open-source event-driven workflow orchestration platform — YAML-defined pipelines, parallel task execution, 200+ plugins (Kafka, dbt, BigQuery, Spark), and webhook/cron/event triggers, all packaged as a scalable alternative to Apache Airflow. When Kestra's executor stalls or the scheduler misses a cron trigger, your data pipelines fail silently. Vigilmon catches it: web UI availability, the health API, executor service status, SSL certificate expiry, and scheduler heartbeats — all in one dashboard.

What You'll Set Up

• HTTP uptime monitor for the Kestra web UI (port 8080)
• REST API health check for /api/v1/health (API + database connectivity)
• Executor service health via /api/v1/health
• SSL certificate expiry alerts for HTTPS-proxied deployments
• Heartbeat monitoring for the Kestra scheduler (cron triggers and execution queue)

Prerequisites

• Kestra running via Docker Compose or Kubernetes (default port 8080)
• HTTPS proxy configured (Nginx or Traefik) for production deployments
• A free Vigilmon account

Step 1: Monitor the Kestra Web UI

The Kestra web interface is the quickest signal that something's wrong — a crashed JVM process, a failed container, or a misconfigured reverse proxy will all prevent the dashboard from loading. Add an HTTP monitor to catch it immediately:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Kestra URL: https://kestra.yourdomain.com (or http://your-server:8080 for local deployments).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

For a more precise signal, target the Kestra login page or dashboard route directly, since these routes are served only when the Vue.js frontend is loaded and the backend is responsive:

https://kestra.yourdomain.com/ui/login

Step 2: Monitor the /api/v1/health Endpoint

Kestra exposes a REST health endpoint that verifies both the API server and its database connection. This is the most authoritative health signal because it validates the full stack — Java API, PostgreSQL/MySQL/H2 backend, and internal service wiring:

1. Add a new HTTP / HTTPS monitor in Vigilmon.
2. Set the URL to https://kestra.yourdomain.com/api/v1/health.
3. Set Check interval to 1 minute.
4. Set Expected HTTP status to 200.
5. Under Response body contains, enter "status" to assert a valid JSON health response.
6. Click Save.

Verify the endpoint before adding the monitor:

curl -s https://kestra.yourdomain.com/api/v1/health | jq .

A healthy Kestra instance responds with:

{
  "status": "UP",
  "components": {
    "db": { "status": "UP" },
    "queue": { "status": "UP" }
  }
}

If the database goes down or the internal queue becomes unreachable, this endpoint will report a degraded or DOWN status — and Vigilmon will alert you before your scheduled flows start piling up missed executions.

Step 3: Monitor the Kestra Executor Service

Kestra separates the executor (the component that actually runs workflow tasks) from the API server. In a multi-component deployment, the executor can crash or stall while the web UI and API continue to respond normally — flows trigger, appear as "running" in the dashboard, but tasks never actually execute.

The /api/v1/health endpoint includes executor status in its response. Add a dedicated monitor for it with a response body assertion on the executor component:

1. Add another HTTP / HTTPS monitor in Vigilmon.
2. Set the URL to https://kestra.yourdomain.com/api/v1/health.
3. Name it Kestra Executor Health to distinguish it from the API health check.
4. Set Check interval to 1 minute.
5. Under Response body contains, enter "executor" to assert the executor component is reported in the health response.
6. Click Save.

For Kubernetes deployments, you can also monitor the executor pod directly if it exposes its own health port. Check your Kestra Helm chart or deployment YAML for the executor service configuration.

Step 4: SSL Certificate Alerts

Kestra is typically deployed behind Nginx or Traefik for TLS termination. A lapsed certificate blocks access to the Kestra API and web UI — not just browser access, but also any external services that call Kestra webhooks or API endpoints.

Enable SSL monitoring on your existing Kestra web UI monitor:

1. Open the Kestra web UI monitor created in Step 1.
2. Enable Monitor SSL certificate in the SSL section.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

For Nginx-proxied Kestra deployments:

# Test renewal before it's needed
certbot renew --dry-run

# Check expiry dates for all managed certificates
certbot certificates

# Force renewal if auto-renewal has fallen behind
certbot renew --force-renewal -d kestra.yourdomain.com
nginx -s reload

The 21-day alert window is particularly important for Kestra because a TLS failure can silently break webhook triggers — external services posting to Kestra webhook URLs will start receiving TLS errors, halting event-driven workflows without any visible error in the Kestra dashboard.

Step 5: Heartbeat Monitoring for the Kestra Scheduler

Kestra's internal scheduler is responsible for firing cron-triggered flows on schedule. If the scheduler component stalls — due to a database lock, a queue backlog, or a JVM memory issue — scheduled flows simply stop executing without any alert. You'll only notice when someone asks why the nightly ETL didn't run.

Monitor the executions API availability:

Add a heartbeat check against the /api/v1/executions endpoint, which Kestra uses internally to track flow runs. If this endpoint goes dark, the scheduler can't record execution state:

1. Add an HTTP / HTTPS monitor in Vigilmon.
2. Set the URL to https://kestra.yourdomain.com/api/v1/executions?pageSize=1.
3. Set Check interval to 1 minute.
4. Set Expected HTTP status to 200.
5. Click Save.

Add a canary flow heartbeat:

For definitive scheduler health verification, create a Kestra flow that runs on a cron schedule and pings a Vigilmon heartbeat URL on successful completion:

id: vigilmon-heartbeat
namespace: monitoring
description: Canary flow — pings Vigilmon every 10 minutes to confirm the scheduler is running

triggers:
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/10 * * * *"

tasks:
  - id: ping-vigilmon
    type: io.kestra.plugin.core.http.Request
    uri: https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID
    method: GET

Deploy this flow via the Kestra UI or API:

curl -X POST https://kestra.yourdomain.com/api/v1/flows \
  -H "Content-Type: application/x-yaml" \
  --data-binary @vigilmon-heartbeat.yml

If the Kestra scheduler stops, this flow stops executing, the heartbeat stops pinging — and Vigilmon alerts after 10 minutes of silence.

Monitor the Kestra health endpoint for queue status:

The /api/v1/health endpoint also surfaces the state of Kestra's internal execution queue (backed by Kafka in enterprise deployments, or an in-memory queue in standalone mode). Add a response body assertion on "queue" in the health check monitor from Step 2 to catch stalled queue conditions:

# Check queue health directly
curl -s https://kestra.yourdomain.com/api/v1/health | jq '.components.queue'

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. Set Consecutive failures before alert to 2 on the web UI monitor — Kestra's JVM startup takes a few seconds and can cause one probe to fail during restarts.
3. Add a maintenance window during Kestra version upgrades:

# Pause monitors during rolling upgrade
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 20}'

# Upgrade Kestra (Docker Compose example)
docker compose pull kestra && docker compose up -d kestra

# Window expires automatically after 20 minutes

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web UI | https://kestra.domain.com | JVM crash, container failure, nginx down | | Health API | /api/v1/health | Database disconnection, API errors | | Executor health | /api/v1/health (executor component) | Executor crash, stalled task execution | | SSL certificate | Main domain | Let's Encrypt renewal failure, TLS webhook break | | Executions API | /api/v1/executions | Scheduler state recording failure | | Canary flow heartbeat | Vigilmon heartbeat URL | Scheduler stopped, cron flows not running |

Kestra's YAML-first orchestration and plugin ecosystem make it a compelling Airflow replacement for data engineering teams — but self-hosted orchestration means you own the reliability story. With Vigilmon watching the health API, executor, scheduler heartbeat, and SSL certificate, you'll catch stalled pipelines in minutes rather than finding out during a post-mortem the next morning.