tutorial

How to Monitor Trino: Coordinator Health, Query Queue Depth, Memory Pressure, and External Checks with Vigilmon

Trino coordinator failures and memory pool exhaustion silently kill query workloads. Learn how to monitor Trino coordinator health, worker availability, query queue depth, and memory pressure with Vigilmon external checks.

Trino (formerly PrestoSQL) is the query engine that lets your data platform answer ad-hoc questions at scale — but when the coordinator goes down, every query fails with a connection error. Worker nodes run out of memory and start throwing EXCEEDED_GLOBAL_MEMORY_LIMIT. The query queue fills up and new queries time out waiting for a slot. None of this generates an HTTP alert by default; users just see failed queries and assume the data is wrong.

Vigilmon gives you external visibility into Trino health through HTTP probe monitoring of the Trino REST API and heartbeat monitoring for query SLAs. This tutorial covers coordinator monitoring, worker health, query queue depth, and memory pressure.

Why Trino Needs External Monitoring

Trino's built-in UI (at :8080/ui) and JMX metrics provide detailed internal telemetry — but external monitoring with Vigilmon adds:

  • Coordinator availability probing from outside your cluster network
  • Worker node count alerting so you catch departing workers before query capacity degrades
  • Query queue depth monitoring to detect scheduling bottlenecks before users notice slow queries
  • Memory pool pressure alerting before EXCEEDED_GLOBAL_MEMORY_LIMIT errors start hitting query workloads
  • Heartbeat SLA checks for scheduled query workloads that must complete within a window

These layers work together: the coordinator can be healthy while memory pressure causes widespread query failures, and query failures can spike while the coordinator's own health endpoint returns 200.

Step 1: Build a Trino Health Endpoint

Trino exposes REST endpoints on the coordinator. Wrap the key checks in a sidecar:

Node.js Health Sidecar

// health/trino.js
const express = require('express');
const axios = require('axios');

const app = express();
const TRINO_URL = process.env.TRINO_URL || 'http://localhost:8080';
const TRINO_USER = process.env.TRINO_USER || 'monitoring';

const headers = {
  'X-Trino-User': TRINO_USER,
  'X-Trino-Source': 'vigilmon-health-check',
};

// Coordinator health — is Trino up and accepting connections?
app.get('/health/trino', async (req, res) => {
  try {
    const infoRes = await axios.get(`${TRINO_URL}/v1/info`, { headers, timeout: 5000 });
    const clusterRes = await axios.get(`${TRINO_URL}/v1/cluster`, { headers, timeout: 5000 });

    const info = infoRes.data;
    const cluster = clusterRes.data;

    if (!info.starting === false && info.starting) {
      return res.status(503).json({ status: 'degraded', reason: 'coordinator_starting' });
    }

    const minWorkers = parseInt(process.env.TRINO_MIN_WORKERS || '2');
    if (cluster.activeWorkers < minWorkers) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'insufficient_workers',
        active_workers: cluster.activeWorkers,
        required: minWorkers,
      });
    }

    return res.status(200).json({
      status: 'ok',
      version: info.version,
      active_workers: cluster.activeWorkers,
      running_queries: cluster.runningQueries,
      queued_queries: cluster.queuedQueries,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Query queue depth — alert when too many queries are queued
app.get('/health/trino/queue', async (req, res) => {
  try {
    const clusterRes = await axios.get(`${TRINO_URL}/v1/cluster`, { headers, timeout: 5000 });
    const cluster = clusterRes.data;

    const maxQueued = parseInt(process.env.TRINO_MAX_QUEUED || '50');
    if (cluster.queuedQueries > maxQueued) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'high_queue_depth',
        queued_queries: cluster.queuedQueries,
        threshold: maxQueued,
      });
    }

    return res.status(200).json({
      status: 'ok',
      queued_queries: cluster.queuedQueries,
      running_queries: cluster.runningQueries,
      blocked_queries: cluster.blockedQueries,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Memory pressure — check pool utilization via JMX REST endpoint
app.get('/health/trino/memory', async (req, res) => {
  try {
    const memRes = await axios.get(`${TRINO_URL}/v1/cluster/memory`, { headers, timeout: 5000 });
    const pools = memRes.data;

    const generalPool = pools.pools?.general;
    if (!generalPool) {
      return res.status(200).json({ status: 'ok', detail: 'no_pool_data' });
    }

    const usedBytes = generalPool.reservedBytes || 0;
    const maxBytes = generalPool.maxBytes || 1;
    const utilizationPct = (usedBytes / maxBytes) * 100;

    const threshold = parseInt(process.env.TRINO_MEMORY_THRESHOLD_PCT || '85');
    if (utilizationPct > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'memory_pressure',
        utilization_pct: Math.round(utilizationPct),
        threshold_pct: threshold,
      });
    }

    return res.status(200).json({
      status: 'ok',
      memory_utilization_pct: Math.round(utilizationPct),
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3010);

Python Health Sidecar

# health/trino_health.py
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)
TRINO_URL = os.environ.get("TRINO_URL", "http://localhost:8080")
HEADERS = {
    "X-Trino-User": os.environ.get("TRINO_USER", "monitoring"),
    "X-Trino-Source": "vigilmon-health-check",
}
MIN_WORKERS = int(os.environ.get("TRINO_MIN_WORKERS", "2"))
MAX_QUEUED = int(os.environ.get("TRINO_MAX_QUEUED", "50"))
MEMORY_THRESHOLD = int(os.environ.get("TRINO_MEMORY_THRESHOLD_PCT", "85"))

@app.route("/health/trino")
def trino_health():
    try:
        info = requests.get(f"{TRINO_URL}/v1/info", headers=HEADERS, timeout=5).json()
        cluster = requests.get(f"{TRINO_URL}/v1/cluster", headers=HEADERS, timeout=5).json()
        if cluster.get("activeWorkers", 0) < MIN_WORKERS:
            return jsonify({"status": "degraded", "active_workers": cluster.get("activeWorkers")}), 503
        return jsonify({
            "status": "ok",
            "active_workers": cluster.get("activeWorkers"),
            "running_queries": cluster.get("runningQueries"),
        }), 200
    except Exception as e:
        return jsonify({"status": "down", "error": str(e)}), 503

@app.route("/health/trino/queue")
def queue_health():
    try:
        cluster = requests.get(f"{TRINO_URL}/v1/cluster", headers=HEADERS, timeout=5).json()
        queued = cluster.get("queuedQueries", 0)
        if queued > MAX_QUEUED:
            return jsonify({"status": "degraded", "queued_queries": queued, "threshold": MAX_QUEUED}), 503
        return jsonify({"status": "ok", "queued_queries": queued}), 200
    except Exception as e:
        return jsonify({"status": "down", "error": str(e)}), 503

if __name__ == "__main__":
    app.run(port=3010)

Step 2: Configure Vigilmon HTTP Monitors for Trino

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Trino health endpoint: https://your-host.example.com/health/trino
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your on-call Slack or PagerDuty channel
  7. Save the monitor

Add monitors for queue depth and memory pressure:

| Monitor | URL | Interval | Priority | |---|---|---|---| | Coordinator | /health/trino | 1 minute | P1 | | Query queue | /health/trino/queue | 2 minutes | P2 | | Memory pressure | /health/trino/memory | 2 minutes | P1 |

Vigilmon's multi-region probing prevents transient coordinator GC pauses from triggering false-positive alerts.

Step 3: Heartbeat Monitoring for Scheduled Query Workloads

Many Trino use cases include scheduled queries — nightly aggregations, hourly refreshes of materialized views, or SLA-bound reporting workloads. Vigilmon heartbeat monitors detect when these workloads stop completing on time.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: trino-nightly-aggregation
  3. Set the expected interval: 24 hours
  4. Set the grace period: 30 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire Heartbeats Into Your Query Scripts

# run_trino_job.py
import os, trino, requests

conn = trino.dbapi.connect(
    host=os.environ["TRINO_HOST"],
    port=int(os.environ.get("TRINO_PORT", "8080")),
    user=os.environ["TRINO_USER"],
    catalog="hive",
    schema="analytics",
)

cursor = conn.cursor()

try:
    # Run the transformation query
    cursor.execute("""
        INSERT INTO hive.analytics.daily_revenue
        SELECT
            DATE(created_at) AS report_date,
            SUM(amount) AS revenue,
            COUNT(*) AS transaction_count
        FROM hive.raw.transactions
        WHERE DATE(created_at) = CURRENT_DATE - INTERVAL '1' DAY
    """)
    cursor.fetchall()  # Wait for completion

    # Query succeeded — ping Vigilmon
    requests.get(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=10)
    print("Heartbeat sent — query completed successfully")

except Exception as e:
    print(f"Query failed: {e}")
    raise  # Re-raise to signal failure to the orchestrator
finally:
    cursor.close()
    conn.close()

For Airflow-orchestrated Trino workloads, add the heartbeat call as a downstream task:

# airflow_dag.py
from airflow import DAG
from airflow.providers.trino.operators.trino import TrinoOperator
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
import requests, os

def ping_vigilmon():
    requests.get(os.environ["VIGILMON_TRINO_HEARTBEAT_URL"], timeout=10)

with DAG("trino_daily_transform", schedule_interval="0 4 * * *", start_date=days_ago(1)) as dag:
    run_query = TrinoOperator(
        task_id="daily_revenue_aggregation",
        trino_conn_id="trino_default",
        sql="sql/daily_revenue.sql",
    )

    notify_vigilmon = PythonOperator(
        task_id="notify_vigilmon",
        python_callable=ping_vigilmon,
    )

    run_query >> notify_vigilmon

Step 4: Worker Node Monitoring via Direct Probe

Trino coordinator exposes active worker info. Monitor worker node count directly:

#!/bin/bash
# check_trino_workers.sh

TRINO_URL="${TRINO_URL:-http://localhost:8080}"
MIN_WORKERS="${TRINO_MIN_WORKERS:-3}"
HEARTBEAT_URL="${VIGILMON_WORKER_HEARTBEAT_URL}"

ACTIVE=$(curl -s -H "X-Trino-User: monitoring" "${TRINO_URL}/v1/cluster" \
  | jq '.activeWorkers')

echo "Active workers: ${ACTIVE} (min: ${MIN_WORKERS})"

if [ "${ACTIVE}" -ge "${MIN_WORKERS}" ]; then
  curl -fsS "${HEARTBEAT_URL}"
  echo "Worker count OK — heartbeat sent"
else
  echo "Worker count below threshold — no heartbeat"
  exit 1
fi

Schedule via cron every 5 minutes:

*/5 * * * * /opt/scripts/check_trino_workers.sh >> /var/log/trino-health.log 2>&1

Summary

Trino failures cascade: coordinator downtime blocks all queries, memory pressure causes random query failures, and queue buildup means new queries time out before they even start. External monitoring with Vigilmon catches all three before users escalate:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/trino | Coordinator availability, worker node count | | HTTP monitor on /health/trino/queue | Query queue depth, blocked query detection | | HTTP monitor on /health/trino/memory | Memory pool utilization, pressure alerts | | Heartbeat monitor | Scheduled query workload SLA compliance |

Get started free at vigilmon.online — your first Trino monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles