tutorial

How to Monitor Apache Kyuubi SQL Gateway Availability and Engine Session Health with Vigilmon

Apache Kyuubi engine process crashes, stalled SQL sessions, and unresponsive Thrift endpoints block data analysts from querying Spark and Hive without any built-in alerting. Learn how to monitor Kyuubi availability, engine health, and query heartbeats with Vigilmon.

Apache Kyuubi is the distributed SQL query gateway that sits in front of Spark, Hive, Trino, and other engines — providing a unified JDBC/ODBC interface for analytics tools — but when the Kyuubi server process hangs from a JVM GC pause, an engine session terminates and leaves an analyst's BI tool with a frozen query, or the Thrift binary transport becomes unresponsive due to thread pool exhaustion, your SQL access layer disappears with no outbound alert. Kyuubi exposes REST APIs and metrics endpoints, but they require active polling and provide no proactive notifications.

Vigilmon gives you external visibility into Kyuubi availability through HTTP monitors on its REST and metrics endpoints and a health sidecar that validates engine session state, plus heartbeat monitors that confirm scheduled SQL workloads are completing on time. This tutorial covers both.

Why Kyuubi Needs External Monitoring

Kyuubi's built-in tooling (the web UI, JMX metrics, and Prometheus metrics endpoint) requires active dashboard watching. External monitoring with Vigilmon adds:

  • Service liveness alerting when the Kyuubi gateway becomes unreachable before BI tools and scheduled jobs attempt connections
  • Engine session health checks detecting engines stuck in PENDING or ERROR state before they block queries
  • Thrift endpoint monitoring confirming the JDBC/ODBC connection surface is responsive, not just the HTTP API
  • Query completion heartbeats verifying that scheduled analytical SQL workloads run and complete end-to-end
  • Response time monitoring catching Kyuubi thread pool saturation before query timeouts cascade to clients

These layers matter because Kyuubi sits between dozens of clients (BI tools, notebooks, ETL pipelines) and multiple engine backends — a single Kyuubi server issue can silently break the SQL access of your entire data platform at once.

Step 1: Probe Kyuubi's Built-in REST API and Metrics

Kyuubi exposes a REST API for server info, engines, and sessions. The server info endpoint is a lightweight liveness check:

# Check Kyuubi REST API liveness
curl -s http://kyuubi.example.com:10099/api/v1/server/info | python3 -m json.tool

# Expected response:
# {"version":"...", "java": "...", ...}

# List active engine sessions
curl -s http://kyuubi.example.com:10099/api/v1/sessions

# List registered engines
curl -s http://kyuubi.example.com:10099/api/v1/engines

Check Prometheus metrics (enabled by default in Kyuubi):

# Prometheus metrics endpoint
curl -s http://kyuubi.example.com:10099/metrics

# Key metrics to watch:
# kyuubi_backend_service_open_session_count — active engine sessions
# kyuubi_operation_total — total operations executed
# jvm_memory_heap_used_bytes — JVM heap usage

Point a Vigilmon monitor at http://kyuubi.example.com:10099/api/v1/server/info for immediate availability alerting.

Step 2: Build a Kyuubi Health Sidecar

For comprehensive health analysis — engine session state, pending engine counts, and Prometheus metric thresholds — build a sidecar that interprets Kyuubi's REST API and metrics.

Python Health Sidecar

# kyuubi_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

KYUUBI_URL           = os.environ.get('KYUUBI_URL',          'http://localhost:10099')
MAX_PENDING_ENGINES  = int(os.environ.get('MAX_PENDING_ENGINES', '3'))
MAX_ERROR_SESSIONS   = int(os.environ.get('MAX_ERROR_SESSIONS',  '5'))
HEAP_WARN_BYTES      = int(os.environ.get('HEAP_WARN_BYTES',     str(6 * 1024**3)))  # 6 GiB
HEAP_CRIT_BYTES      = int(os.environ.get('HEAP_CRIT_BYTES',     str(8 * 1024**3)))  # 8 GiB

@app.route('/health/kyuubi')
def kyuubi_health():
    try:
        r = requests.get(f'{KYUUBI_URL}/api/v1/server/info', timeout=5)
        if r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'http_{r.status_code}'}), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    return jsonify({'status': 'ok', 'version': r.json().get('version', 'unknown')}), 200

@app.route('/health/kyuubi/engines')
def kyuubi_engines():
    errors   = []
    warnings = []
    try:
        r = requests.get(f'{KYUUBI_URL}/api/v1/engines', timeout=10)
        r.raise_for_status()
        engines = r.json()
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    if not isinstance(engines, list):
        engines = []

    pending_count = sum(1 for e in engines if e.get('engineStatus', '').upper() in ('PENDING', 'STARTING'))
    error_count   = sum(1 for e in engines if e.get('engineStatus', '').upper() in ('ERROR', 'DEAD'))

    if error_count > 0:
        errors.append(f'{error_count}_engines_in_error_state')
    if pending_count >= MAX_PENDING_ENGINES:
        warnings.append(f'{pending_count}_engines_pending')

    if errors:
        return jsonify({'status': 'down', 'errors': errors, 'warnings': warnings, 'total_engines': len(engines)}), 503
    if warnings:
        return jsonify({'status': 'degraded', 'warnings': warnings, 'total_engines': len(engines)}), 200
    return jsonify({'status': 'ok', 'total_engines': len(engines), 'pending_engines': pending_count}), 200

@app.route('/health/kyuubi/sessions')
def kyuubi_sessions():
    try:
        r = requests.get(f'{KYUUBI_URL}/api/v1/sessions', timeout=10)
        r.raise_for_status()
        sessions = r.json()
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    if not isinstance(sessions, list):
        sessions = []

    error_count = sum(1 for s in sessions if s.get('sessionStatus', '').upper() in ('ERROR',))

    if error_count >= MAX_ERROR_SESSIONS:
        return jsonify({'status': 'down', 'error_sessions': error_count, 'total_sessions': len(sessions)}), 503
    return jsonify({'status': 'ok', 'total_sessions': len(sessions), 'error_sessions': error_count}), 200

@app.route('/health/kyuubi/metrics')
def kyuubi_metrics():
    """Parse Prometheus metrics for heap and session count thresholds."""
    warnings = []
    errors   = []
    try:
        r = requests.get(f'{KYUUBI_URL}/metrics', timeout=10)
        r.raise_for_status()
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    heap_used = None
    for line in r.text.splitlines():
        if line.startswith('jvm_memory_heap_used_bytes') and not line.startswith('#'):
            try:
                heap_used = float(line.split()[-1])
            except ValueError:
                pass

    if heap_used is not None:
        if heap_used >= HEAP_CRIT_BYTES:
            errors.append(f'heap_critical_{round(heap_used / 1024**3, 1)}GiB')
        elif heap_used >= HEAP_WARN_BYTES:
            warnings.append(f'heap_warning_{round(heap_used / 1024**3, 1)}GiB')

    if errors:
        return jsonify({'status': 'down', 'errors': errors, 'warnings': warnings}), 503
    if warnings:
        return jsonify({'status': 'degraded', 'warnings': warnings}), 200
    return jsonify({'status': 'ok', 'heap_used_gib': round((heap_used or 0) / 1024**3, 2)}), 200

if __name__ == '__main__':
    app.run(port=8100)

Node.js Health Sidecar

// kyuubi-health.js
const express = require('express');
const axios   = require('axios');

const app = express();

const KYUUBI_URL          = process.env.KYUUBI_URL          || 'http://localhost:10099';
const MAX_PENDING_ENGINES = parseInt(process.env.MAX_PENDING_ENGINES || '3');
const MAX_ERROR_SESSIONS  = parseInt(process.env.MAX_ERROR_SESSIONS  || '5');

app.get('/health/kyuubi', async (req, res) => {
  try {
    const r = await axios.get(`${KYUUBI_URL}/api/v1/server/info`, { timeout: 5000 });
    return res.status(200).json({ status: 'ok', version: r.data?.version });
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }
});

app.get('/health/kyuubi/engines', async (req, res) => {
  let engines;
  try {
    const r = await axios.get(`${KYUUBI_URL}/api/v1/engines`, { timeout: 10000 });
    engines = Array.isArray(r.data) ? r.data : [];
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }

  const pendingCount = engines.filter(e => ['PENDING', 'STARTING'].includes((e.engineStatus || '').toUpperCase())).length;
  const errorCount   = engines.filter(e => ['ERROR', 'DEAD'].includes((e.engineStatus || '').toUpperCase())).length;

  if (errorCount > 0) return res.status(503).json({ status: 'down', error_engines: errorCount, total_engines: engines.length });
  if (pendingCount >= MAX_PENDING_ENGINES) return res.status(200).json({ status: 'degraded', pending_engines: pendingCount });
  return res.status(200).json({ status: 'ok', total_engines: engines.length });
});

app.listen(3016, () => console.log('kyuubi-health listening on 3016'));

Step 3: Configure Vigilmon HTTP Monitor for Kyuubi

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to http://kyuubi.example.com:10099/api/v1/server/info
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "version"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your data platform Slack channel
  7. Save the monitor

Add a second monitor for the engine health sidecar:

  • URL: https://your-app.example.com/health/kyuubi/engines
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert channel: data platform on-call channel

Add a Prometheus metrics monitor:

  • URL: https://your-app.example.com/health/kyuubi/metrics
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert channel: infrastructure team Slack channel

Add a session health monitor:

  • URL: https://your-app.example.com/health/kyuubi/sessions
  • Expected: 200
  • Interval: 2 minutes
  • Alert channel: data engineering on-call channel

For high-availability Kyuubi deployments (multiple server instances behind a load balancer or ZooKeeper-discovered), create monitors for both the load balancer endpoint and individual Kyuubi node health endpoints.

Step 4: Heartbeat Monitoring for Scheduled Kyuubi SQL Workloads

Service liveness and engine health checks tell you Kyuubi and its engine backends are operational — but they don't confirm that your scheduled SQL jobs (ETL transformations, daily aggregations, report precomputations) are successfully executing and completing on time. A scheduled query could hang on a lock, time out at the engine level without Kyuubi surfacing a client error, or be skipped by an upstream scheduler when Kyuubi was briefly unavailable.

Vigilmon heartbeat monitors detect these silent failures: the SQL job wrapper pings Vigilmon after each successful query execution.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: kyuubi-daily-etl-query
  3. Set the expected interval: 24 hours (or your query schedule)
  4. Set the grace period: 2 hours
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Python JDBC Query with Heartbeat (using PyHive)

# kyuubi_query_runner.py — run a SQL query via Kyuubi JDBC and send heartbeat on success
from pyhive import hive
import requests
import os

KYUUBI_HOST  = os.environ['KYUUBI_HOST']
KYUUBI_PORT  = int(os.environ.get('KYUUBI_PORT',  '10009'))
VIGILMON_HB  = os.environ['VIGILMON_HEARTBEAT_URL']
TARGET_DB    = os.environ.get('TARGET_DB', 'default')
TARGET_TABLE = os.environ.get('TARGET_TABLE', 'daily_summary')

conn = hive.Connection(
    host=KYUUBI_HOST,
    port=KYUUBI_PORT,
    database=TARGET_DB,
    auth='NONE'
)

cursor = conn.cursor()

try:
    # Example: refresh a daily summary partition
    cursor.execute(f"""
        INSERT OVERWRITE TABLE {TARGET_TABLE}
        PARTITION (dt = '{__import__('datetime').date.today()}')
        SELECT user_id, COUNT(*) AS event_count
        FROM events
        WHERE dt = '{__import__('datetime').date.today()}'
        GROUP BY user_id
    """)

    # Verify the partition was written
    cursor.execute(f"SELECT COUNT(*) FROM {TARGET_TABLE} WHERE dt = '{__import__('datetime').date.today()}'")
    row_count = cursor.fetchone()[0]
    if row_count == 0:
        raise ValueError(f"Partition for today written but returned 0 rows — possible silent failure.")

    print(f"OK: {row_count} rows written to {TARGET_TABLE}.")

    # Send heartbeat
    requests.get(VIGILMON_HB, timeout=10)
    print("Vigilmon heartbeat sent.")

finally:
    cursor.close()
    conn.close()

Bash SQL Runner via Beeline (JDBC CLI)

#!/bin/bash
# kyuubi_beeline_runner.sh — run SQL via Beeline against Kyuubi with heartbeat
set -e

KYUUBI_HOST="${KYUUBI_HOST:-kyuubi.example.com}"
KYUUBI_PORT="${KYUUBI_PORT:-10009}"
VIGILMON_HB="${VIGILMON_HEARTBEAT_URL}"
SQL_FILE="${SQL_FILE:-/opt/etl/daily_refresh.sql}"

echo "[$(date)] Running Beeline query against Kyuubi ${KYUUBI_HOST}:${KYUUBI_PORT}..."

beeline \
  -u "jdbc:hive2://${KYUUBI_HOST}:${KYUUBI_PORT}/default" \
  -f "${SQL_FILE}" \
  --silent=false \
  --fastConnect=false \
  2>&1 | tee /tmp/kyuubi_beeline_output.log

EXIT_CODE=${PIPESTATUS[0]}

if [ $EXIT_CODE -ne 0 ]; then
  echo "[$(date)] ERROR: Beeline query failed with exit code ${EXIT_CODE}."
  exit 1
fi

echo "[$(date)] Beeline query succeeded. Sending Vigilmon heartbeat..."
curl -s --max-time 10 "${VIGILMON_HB}" > /dev/null
echo "[$(date)] Heartbeat sent."

Airflow Operator Pattern (DAG heartbeat)

# kyuubi_airflow_dag.py — Airflow DAG that runs Kyuubi SQL and pings Vigilmon on completion
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.apache.hive.operators.hive import HiveOperator
import requests
import os

VIGILMON_HB = os.environ.get('VIGILMON_HEARTBEAT_URL_KYUUBI_ETL')

def ping_vigilmon(**context):
    if VIGILMON_HB:
        requests.get(VIGILMON_HB, timeout=10)
        print("Vigilmon heartbeat sent.")

with DAG('kyuubi_daily_etl', schedule_interval='0 6 * * *', catchup=False) as dag:
    run_etl = HiveOperator(
        task_id='run_kyuubi_etl',
        hql='/opt/etl/daily_refresh.sql',
        hive_cli_conn_id='kyuubi_jdbc',
    )
    heartbeat = PythonOperator(
        task_id='send_vigilmon_heartbeat',
        python_callable=ping_vigilmon,
    )
    run_etl >> heartbeat

Step 5: Alert Routing for Kyuubi Failures

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Kyuubi server info API | Slack + PagerDuty | P1 | | Engine health sidecar | Slack + PagerDuty | P1 | | Session health sidecar | Slack | P2 | | JVM heap metrics | Slack | P2 | | Heartbeat: scheduled ETL query | Slack + email | P1 | | Heartbeat: daily report aggregation | Email | P2 |

Set response time thresholds for early warning:

  • Alert at 3000ms for the Kyuubi server info endpoint (slow REST responses predict thread pool saturation)
  • Alert at 10000ms for the engine health sidecar (engine enumeration latency spikes before Kyuubi runs out of available engine slots)

For analytical workloads where freshness SLAs exist (dashboards must reflect data no older than N hours), configure heartbeat grace periods tightly aligned to those SLAs. A daily ETL that runs at midnight with a 2-hour SLA should alert by 02:30, not at the next business check-in.

Summary

Kyuubi failures span gateway availability, engine session state, JVM resource pressure, and end-to-end SQL workload completion. External monitoring catches all four:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on server info API | Kyuubi gateway liveness | | HTTP monitor on engine health sidecar | Engine session state (pending/error) | | HTTP monitor on session sidecar | Active query session health | | HTTP monitor on metrics sidecar | JVM heap pressure thresholds | | Heartbeat monitor on SQL job completion | Scheduled ETL/report query success |

Get started free at vigilmon.online — your first Kyuubi SQL gateway monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles