How to Monitor Apache Kylin Query Engine Health and Job Status with Vigilmon

Apache Kylin is the OLAP engine that makes sub-second analytical queries possible over petabyte-scale Hadoop datasets — but when the Kylin server goes down, a cube build job fails halfway through, or the query engine loses its HBase connection, your dashboards go dark and your BI tools start returning stale or empty results. Unlike simple web services, Kylin failures can be partial: the REST API stays up while cube segments become unavailable, or new data stops flowing into cubes while historical queries continue working.

Vigilmon gives you external visibility into Kylin health through HTTP probe monitoring of the Kylin REST API and heartbeat monitoring for your cube-building pipelines. This tutorial covers both.

Why Kylin Needs External Monitoring

Kylin's built-in tooling (the Kylin web UI on port 7070, the Kylin REST API, and job monitoring in the UI) requires someone to actively check it. External monitoring with Vigilmon adds:

• Proactive alerting when the Kylin server becomes unreachable or returns errors
• Cube build job failure detection via the Kylin REST API job status endpoint
• Query health verification by probing a lightweight test query endpoint
• Heartbeat monitoring so you know immediately when your ETL/cube-build pipelines stop running
• Multi-region availability checking from outside your Hadoop cluster

These layers work together: the Kylin REST API can return 200 while a cube build job is in ERROR state, and a cube can be queryable while new segments fail to build — meaning your dashboards show data that is hours or days stale without any visible error.

Step 1: Build a Kylin Health Endpoint

Kylin's REST API exposes several health-checkable endpoints you can probe directly.

Direct REST API Probes

# Check Kylin server is up and returning a valid response
curl -i http://kylin.example.com:7070/kylin/api/user/authentication \
  -H "Authorization: Basic $(echo -n 'ADMIN:KYLIN' | base64)"

# List projects — confirms HBase/metadata connectivity
curl -i http://kylin.example.com:7070/kylin/api/projects \
  -H "Authorization: Basic $(echo -n 'ADMIN:KYLIN' | base64)"

# Check running/error jobs — use this to detect stuck cube builds
curl -i "http://kylin.example.com:7070/kylin/api/jobs?status=ERROR&limit=10" \
  -H "Authorization: Basic $(echo -n 'ADMIN:KYLIN' | base64)"

Custom Health Sidecar

For a clean 200/503 health endpoint that Vigilmon can probe without authentication complexity, build a thin health sidecar:

// kylin-health.js
const express = require('express');
const axios = require('axios');

const app = express();

const KYLIN_URL = process.env.KYLIN_URL || 'http://localhost:7070';
const KYLIN_AUTH = Buffer.from(
  `${process.env.KYLIN_USER || 'ADMIN'}:${process.env.KYLIN_PASSWORD}`
).toString('base64');

app.get('/health/kylin', async (req, res) => {
  try {
    // Check server is up
    const authRes = await axios.get(`${KYLIN_URL}/kylin/api/user/authentication`, {
      headers: { Authorization: `Basic ${KYLIN_AUTH}` },
      timeout: 8000,
    });

    if (authRes.status !== 200) {
      return res.status(503).json({ status: 'down', reason: 'auth_failed' });
    }

    // Check for ERROR-state jobs in the last hour
    const jobsRes = await axios.get(
      `${KYLIN_URL}/kylin/api/jobs?status=ERROR&limit=5&timeFilter=1`,
      { headers: { Authorization: `Basic ${KYLIN_AUTH}` }, timeout: 8000 }
    );

    const errorJobs = jobsRes.data || [];
    if (errorJobs.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'cube_build_errors',
        error_job_count: errorJobs.length,
        latest_error: errorJobs[0]?.uuid,
      });
    }

    return res.status(200).json({
      status: 'ok',
      server: KYLIN_URL,
    });

  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Probe endpoint: run a lightweight test query on a known cube
app.get('/health/kylin/query', async (req, res) => {
  const cubeName = process.env.KYLIN_TEST_CUBE;
  const testQuery = process.env.KYLIN_TEST_QUERY ||
    `SELECT count(*) FROM ${cubeName} LIMIT 1`;

  try {
    const queryRes = await axios.post(
      `${KYLIN_URL}/kylin/api/query`,
      { sql: testQuery, project: process.env.KYLIN_PROJECT, limit: 1 },
      {
        headers: {
          Authorization: `Basic ${KYLIN_AUTH}`,
          'Content-Type': 'application/json',
        },
        timeout: 15000,
      }
    );

    if (queryRes.data?.isException) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'query_exception',
        exception: queryRes.data.exceptionMessage,
      });
    }

    return res.status(200).json({
      status: 'ok',
      query_duration_ms: queryRes.data?.duration,
    });

  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3007);

Python Health Sidecar

# kylin_health.py
from flask import Flask, jsonify
import requests, os, base64

app = Flask(__name__)

KYLIN_URL = os.environ.get('KYLIN_URL', 'http://localhost:7070')
auth = base64.b64encode(
    f"{os.environ.get('KYLIN_USER', 'ADMIN')}:{os.environ['KYLIN_PASSWORD']}".encode()
).decode()
HEADERS = {'Authorization': f'Basic {auth}'}

@app.route('/health/kylin')
def kylin_health():
    try:
        r = requests.get(
            f'{KYLIN_URL}/kylin/api/user/authentication',
            headers=HEADERS, timeout=8
        )
        if r.status_code != 200:
            return jsonify({'status': 'down', 'code': r.status_code}), 503
        return jsonify({'status': 'ok'}), 200
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8091)

Step 2: Configure Vigilmon HTTP Monitor for Kylin

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your Kylin health endpoint: https://your-app.example.com/health/kylin
4. Set the check interval to 2 minutes
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 15000ms (Kylin admin API calls involve HBase metadata reads)
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Add a second monitor for query health:

• URL: https://your-app.example.com/health/kylin/query
• Expected: 200, body contains "status":"ok"
• Interval: 5 minutes (test queries against real cubes take longer)
• Alert channel: separate analytics-on-call channel

Vigilmon's multi-region probe consensus prevents false alerts from transient HDFS metadata latency spikes.

Step 3: Heartbeat Monitoring for Kylin Cube Build Pipelines

Kylin server health and query availability are necessary — but not sufficient. Your cube build pipeline (the ETL job that feeds new data into Kylin cubes) can be stuck, failed silently, or skipped due to a scheduler misconfiguration, while the Kylin server looks healthy serving queries against stale cube segments.

Vigilmon heartbeat monitors detect silent pipeline stalls: your cube-build job pings Vigilmon after each successful build. If pings stop, Vigilmon alerts before your data goes stale.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: kylin-cube-build-pipeline
3. Set the expected interval: 1 hour (or your cube refresh cadence)
4. Set the grace period: 30 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Cube Build Script

#!/bin/bash
# build_cube.sh — triggered by Airflow, cron, or Oozie

set -e

KYLIN_URL="${KYLIN_URL:-http://localhost:7070}"
CUBE_NAME="${KYLIN_CUBE_NAME}"
PROJECT="${KYLIN_PROJECT}"
VIGILMON_HB="${VIGILMON_HEARTBEAT_URL}"

echo "Triggering cube build for $CUBE_NAME..."

# Submit build job via Kylin REST API
RESPONSE=$(curl -s -X PUT \
  "${KYLIN_URL}/kylin/api/cubes/${CUBE_NAME}/build" \
  -H "Authorization: Basic $(echo -n "${KYLIN_USER}:${KYLIN_PASSWORD}" | base64)" \
  -H "Content-Type: application/json" \
  -d '{"buildType":"BUILD","startTime":0,"endTime":'"$(date +%s%3N)"'}')

JOB_ID=$(echo "$RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin)['uuid'])")
echo "Job submitted: $JOB_ID"

# Poll for job completion
for i in $(seq 1 120); do
  STATUS=$(curl -s "${KYLIN_URL}/kylin/api/jobs/${JOB_ID}" \
    -H "Authorization: Basic $(echo -n "${KYLIN_USER}:${KYLIN_PASSWORD}" | base64)" \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['job_status'])")

  echo "  Status: $STATUS"
  if [ "$STATUS" = "FINISHED" ]; then
    echo "Build complete. Pinging Vigilmon..."
    curl -s "$VIGILMON_HB" > /dev/null
    exit 0
  elif [ "$STATUS" = "ERROR" ] || [ "$STATUS" = "DISCARDED" ]; then
    echo "Build failed with status: $STATUS"
    exit 1
  fi
  sleep 30
done

echo "Build timed out after 1 hour"
exit 1

Python Pipeline with Heartbeat

# kylin_pipeline.py
import requests, os, time, base64

KYLIN_URL = os.environ['KYLIN_URL']
auth = base64.b64encode(
    f"{os.environ['KYLIN_USER']}:{os.environ['KYLIN_PASSWORD']}".encode()
).decode()
headers = {'Authorization': f'Basic {auth}', 'Content-Type': 'application/json'}

def build_and_monitor(cube_name, project, start_ms, end_ms):
    # Trigger build
    resp = requests.put(
        f'{KYLIN_URL}/kylin/api/cubes/{cube_name}/build',
        headers=headers,
        json={'buildType': 'BUILD', 'startTime': start_ms, 'endTime': end_ms}
    )
    resp.raise_for_status()
    job_id = resp.json()['uuid']
    print(f'Build job {job_id} submitted')

    # Poll for completion
    for _ in range(120):
        time.sleep(30)
        job = requests.get(
            f'{KYLIN_URL}/kylin/api/jobs/{job_id}', headers=headers
        ).json()
        status = job['job_status']
        print(f'  {status}')
        if status == 'FINISHED':
            # Ping Vigilmon on successful build
            requests.get(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)
            return
        if status in ('ERROR', 'DISCARDED'):
            raise RuntimeError(f'Cube build failed: {status}')

    raise TimeoutError('Cube build did not complete within 1 hour')

build_and_monitor(
    cube_name=os.environ['KYLIN_CUBE_NAME'],
    project=os.environ['KYLIN_PROJECT'],
    start_ms=0,
    end_ms=int(time.time() * 1000)
)

Step 4: Alert Routing for Kylin Failures

Kylin failures have two distinct blast radii: server failures affect all queries immediately, while cube build failures affect data freshness silently over time. Route alerts accordingly.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Kylin server health /health/kylin | Slack + PagerDuty | P1 | | Kylin query health /health/kylin/query | Slack | P1 | | Heartbeat: cube build pipeline | Slack + email | P2 | | Heartbeat: incremental refresh job | Email | P3 |

Set response time thresholds for early warning:

• Alert at 10000ms for the server health endpoint (slow HBase metadata reads signal cluster pressure)
• Alert at 30000ms for the query health endpoint (Kylin query latency degrades before full failure)

For SLA-bound analytics, configure a stale data alert: if the cube build heartbeat misses two consecutive expected intervals, escalate immediately — by the third missed interval, your dashboards are showing data that is 2+ refresh cycles out of date.

Summary

Kylin failures split into two categories: server downtime (immediate, visible) and cube build failures (silent, data-staleness). You need monitoring at both layers:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/kylin | Server availability, build job error detection | | HTTP monitor on /health/kylin/query | Query engine liveness, cube accessibility | | Heartbeat monitor | Cube build pipeline health, data freshness |

Get started free at vigilmon.online — your first Kylin monitor is running in under two minutes.