tutorial

How to Monitor Apache Hive HiveServer2 and Query Health with Vigilmon

HiveServer2 failures and long-running query hangs silently break your data warehouse workloads. Learn how to monitor Hive HiveServer2 availability, Metastore connectivity, and query pipeline liveness with Vigilmon HTTP probes and heartbeat monitors.

Apache Hive is the SQL-on-Hadoop data warehouse that gives your analysts and BI tools familiar query semantics over petabytes of HDFS data — but when HiveServer2 crashes, the Hive Metastore loses its database connection, or a query gets stuck in a YARN queue with no available resources, your data pipelines silently stall. Unlike HTTP API failures, Hive failures often manifest as hung JDBC connections, infinite query waits, or Beeline sessions that accept statements but never return results.

Vigilmon gives you external visibility into Hive health through HTTP probe monitoring of HiveServer2's web UI and a custom health sidecar, plus heartbeat monitoring for your query pipelines. This tutorial covers both.

Why Hive Needs External Monitoring

Hive's built-in tooling (the HiveServer2 web UI on port 10002, YARN application tracking, and Cloudera Manager/Ambari metrics) requires active watching. External monitoring with Vigilmon adds:

  • Proactive alerting when HiveServer2 becomes unreachable (failed process, OOM kill, GC stall)
  • Metastore connectivity detection via a lightweight health query to the Thrift Metastore
  • Query pipeline heartbeats so you know immediately when an ETL job stops submitting or completing queries
  • Multi-region availability checking from outside your Hadoop cluster network perimeter

These layers work together: HiveServer2 can accept JDBC connections while the Metastore is unreachable (queries fail at parse time), and the Metastore can be healthy while YARN has no containers available for MapReduce/Tez execution (queries queue indefinitely without error).

Step 1: Build a Hive Health Endpoint

HiveServer2 exposes a web UI on port 10002 with a /status path you can probe, and the Thrift interface on port 10000 for JDBC — but for HTTP monitoring you need either a direct web UI probe or a custom health sidecar.

Direct HiveServer2 Web UI Probe

# HiveServer2 web UI status page — returns 200 when the server is running
curl -i http://hiveserver2.example.com:10002/

# Prometheus metrics endpoint (if enabled)
curl -i http://hiveserver2.example.com:10002/metrics

Point a Vigilmon monitor at http://hiveserver2.example.com:10002/ for a fast reachability check — if the web UI returns non-200, HiveServer2 has crashed or is unresponsive.

Custom Python Health Sidecar (PyHive / impyla)

For deeper health checks — Metastore connectivity, a test query, JDBC liveness — build a thin sidecar:

# hive_health.py
from flask import Flask, jsonify
from pyhive import hive
import os

app = Flask(__name__)

HIVE_HOST = os.environ.get('HIVE_HOST', 'localhost')
HIVE_PORT = int(os.environ.get('HIVE_PORT', '10000'))
HIVE_DATABASE = os.environ.get('HIVE_DATABASE', 'default')
HIVE_USERNAME = os.environ.get('HIVE_USERNAME', 'hive')

@app.route('/health/hive')
def hive_health():
    try:
        conn = hive.Connection(
            host=HIVE_HOST,
            port=HIVE_PORT,
            database=HIVE_DATABASE,
            username=HIVE_USERNAME,
            timeout=10,
        )
        cursor = conn.cursor()
        # Lightweight query: list databases confirms HiveServer2 + Metastore are both up
        cursor.execute('SHOW DATABASES')
        dbs = [row[0] for row in cursor.fetchall()]
        cursor.close()
        conn.close()
        return jsonify({'status': 'ok', 'databases': len(dbs)}), 200
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

@app.route('/health/hive/query')
def hive_query_health():
    test_table = os.environ.get('HIVE_TEST_TABLE')
    if not test_table:
        return jsonify({'status': 'skipped', 'reason': 'no_test_table_configured'}), 200
    try:
        conn = hive.Connection(
            host=HIVE_HOST,
            port=HIVE_PORT,
            database=HIVE_DATABASE,
            username=HIVE_USERNAME,
            timeout=30,
        )
        cursor = conn.cursor()
        cursor.execute(f'SELECT count(*) FROM {test_table} LIMIT 1')
        result = cursor.fetchone()
        cursor.close()
        conn.close()
        return jsonify({'status': 'ok', 'row_count': result[0]}), 200
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8092)

Node.js Health Sidecar (node-hive / JDBC via rest)

// hive-health.js
const express = require('express');
const axios = require('axios');

const app = express();

// If you expose HiveServer2 via Apache Knox or a REST gateway:
const HIVE_REST_URL = process.env.HIVE_REST_URL;
const HIVE_AUTH = Buffer.from(
  `${process.env.HIVE_USER}:${process.env.HIVE_PASSWORD}`
).toString('base64');

app.get('/health/hive', async (req, res) => {
  try {
    // Knox Gateway health endpoint
    const r = await axios.get(`${HIVE_REST_URL}/v1/cluster`, {
      headers: { Authorization: `Basic ${HIVE_AUTH}` },
      timeout: 8000,
    });
    return res.status(200).json({ status: 'ok', cluster: r.data?.name });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3008);

Step 2: Configure Vigilmon HTTP Monitor for Hive

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Hive health endpoint: https://your-app.example.com/health/hive
  4. Set the check interval to 2 minutes
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 15000ms (JDBC connections to HiveServer2 involve Metastore Thrift calls)
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the HiveServer2 web UI directly:

  • URL: http://hiveserver2.example.com:10002/
  • Expected: 200
  • Interval: 1 minute
  • Alert channel: data-platform on-call channel

And optionally a third monitor for the Hive Metastore service:

  • URL: https://your-app.example.com/health/hive/query
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes (test queries involve YARN resource allocation)
  • Alert channel: analytics-team Slack channel

Step 3: Heartbeat Monitoring for Hive Query Pipelines

HiveServer2 availability and Metastore connectivity are necessary — but not sufficient. Your ETL jobs that use Hive for data transformation (nightly batch loads, incremental processing jobs, dbt runs over Hive) can be silently failing to complete, stuck waiting for YARN containers, or running queries that finish with empty result sets due to partition filtering bugs — while the Hive health endpoint returns 200.

Vigilmon heartbeat monitors detect silent pipeline stalls: your ETL job pings Vigilmon after each successful query batch or processing cycle. If pings stop, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: hive-etl-pipeline
  3. Set the expected interval: 6 hours (or your ETL cadence)
  4. Set the grace period: 1 hour
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your ETL Script

#!/bin/bash
# hive_etl.sh — Beeline-based ETL with Vigilmon heartbeat

set -e

HIVE_URL="jdbc:hive2://${HIVE_HOST}:10000/${HIVE_DATABASE}"
VIGILMON_HB="${VIGILMON_HEARTBEAT_URL}"

echo "Running Hive ETL..."

beeline -u "$HIVE_URL" \
  -n "${HIVE_USERNAME}" \
  -p "${HIVE_PASSWORD}" \
  -f /opt/etl/transform.hql

if [ $? -eq 0 ]; then
  echo "ETL complete. Pinging Vigilmon heartbeat..."
  curl -s "$VIGILMON_HB" > /dev/null
  echo "Done."
else
  echo "ETL failed — Vigilmon heartbeat NOT sent."
  exit 1
fi

Python ETL with Heartbeat (PyHive)

# hive_etl.py
from pyhive import hive
import requests, os

HIVE_HOST = os.environ['HIVE_HOST']
HIVE_DATABASE = os.environ.get('HIVE_DATABASE', 'default')

conn = hive.Connection(
    host=HIVE_HOST,
    port=int(os.environ.get('HIVE_PORT', '10000')),
    database=HIVE_DATABASE,
    username=os.environ.get('HIVE_USERNAME', 'hive'),
)

cursor = conn.cursor()

# Run transformation queries
etl_queries = [
    "INSERT OVERWRITE TABLE daily_summary SELECT ...",
    "MSCK REPAIR TABLE partitioned_events",
    "ANALYZE TABLE daily_summary COMPUTE STATISTICS",
]

for query in etl_queries:
    print(f"Running: {query[:60]}...")
    cursor.execute(query)
    print("  OK")

cursor.close()
conn.close()

# Ping Vigilmon only after all queries complete successfully
requests.get(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)
print("Vigilmon heartbeat sent.")

dbt on Hive with Heartbeat

If you use dbt with the Hive adapter, add a post-hook to your dbt_project.yml:

# dbt_project.yml
on-run-end:
  - "{{ log('dbt run complete', info=True) }}"

# After the run, ping Vigilmon from your CI/CD pipeline:
# dbt run --profiles-dir . && curl "$VIGILMON_HEARTBEAT_URL"

Or wrap the dbt run in a shell script:

#!/bin/bash
dbt run --profiles-dir /opt/dbt --project-dir /opt/dbt/project
dbt test --profiles-dir /opt/dbt --project-dir /opt/dbt/project
curl -s "$VIGILMON_HEARTBEAT_URL" > /dev/null

Step 4: Alert Routing for Hive Failures

Hive failures have distinct urgency levels: HiveServer2 down affects all analytical queries immediately, while ETL pipeline failures cause data freshness degradation that becomes critical over hours.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | HiveServer2 web UI (port 10002) | Slack + PagerDuty | P1 | | Hive JDBC health /health/hive | Slack + PagerDuty | P1 | | Hive query probe /health/hive/query | Slack | P2 | | Heartbeat: nightly ETL pipeline | Slack + email | P2 | | Heartbeat: incremental refresh job | Email | P3 |

Set response time thresholds for early warning:

  • Alert at 8000ms for the JDBC health endpoint (slow connections signal Metastore pressure or connection pool exhaustion)
  • Alert at 45000ms for the query probe endpoint (slow test queries signal YARN queue congestion or Tez session startup delays)

For SLA-bound data warehouses, use a 24-hour data freshness budget: if the ETL heartbeat misses beyond 1.5× its expected interval without recovery, escalate to the data platform team immediately — analysts will notice stale dashboards before the next alert cycle.

Summary

Hive failures span two dimensions: service availability (immediate) and data freshness (gradual). External monitoring catches both:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on HiveServer2 web UI | Process availability, HiveServer2 crash detection | | HTTP monitor on JDBC health endpoint | Metastore connectivity, JDBC liveness | | HTTP monitor on query probe | YARN availability, end-to-end query health | | Heartbeat monitor | ETL pipeline health, data freshness assurance |

Get started free at vigilmon.online — your first Hive monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles