tutorial

How to Monitor Apache Livy REST Service Availability and Spark Session Health with Vigilmon

Apache Livy session timeouts, Spark context failures, and silent REST service crashes break your programmatic Spark access without any notification. Learn how to monitor Livy availability, session state, and batch job heartbeats with Vigilmon.

Apache Livy is the REST service that lets applications interact with Apache Spark without needing to maintain a long-running Spark connection — but when the Livy server process crashes from a heap overflow, a Spark session times out and leaves a client's submitted code hanging mid-execution, or an interactive session silently enters a dead state, your data platform's programmatic Spark access disappears with no outbound alert. Livy exposes a REST API for session and batch status, but it has no built-in health monitoring or alerting.

Vigilmon gives you external visibility into Livy availability through HTTP monitors on its REST endpoints and a health sidecar that checks session and batch state, plus heartbeat monitors that confirm batch jobs are completing on schedule. This tutorial covers both.

Why Livy Needs External Monitoring

Livy's built-in tooling (the web UI at /ui, YARN application tracking for submitted Spark jobs, and server logs) requires active inspection. External monitoring with Vigilmon adds:

  • Service availability alerting when the Livy REST service is unreachable before clients attempt to create sessions
  • Session health checks detecting sessions in dead, error, or not_started states before they cause client-side failures
  • Batch job tracking alerting when submitted batch jobs fail or time out
  • Response time monitoring catching Livy server performance degradation before it causes client timeouts
  • Proactive dead-session detection finding stale dead sessions that accumulate and mask active session counts

These layers matter because Livy failures are often invisible to the clients that use it: a client SDK may retry session creation silently, accumulating dead sessions and consuming YARN resources, while your application appears to be running normally until it exhausts its retry budget and crashes.

Step 1: Probe Livy's Built-in REST API

Livy exposes a REST API for session, batch, and version management. The root version endpoint is a lightweight liveness check:

# Check Livy server liveness
curl -s http://livy.example.com:8998/ | python3 -m json.tool

# Check sessions list (confirms REST API is operational)
curl -s http://livy.example.com:8998/sessions

# Expected: {"from":0,"total":N,"sessions":[...]}

Check batch job status:

# List all batch jobs
curl -s http://livy.example.com:8998/batches

# Check a specific batch job by ID
curl -s http://livy.example.com:8998/batches/<batch-id>
# State values: not_started | starting | recovering | idle | running | busy | shutting_down | error | dead | success

Point a Vigilmon monitor at http://livy.example.com:8998/sessions for a fast REST API reachability check.

Step 2: Build a Livy Health Sidecar

For deeper health analysis — dead session counts, batch job failure rates, and session state distribution — build a sidecar that interprets Livy's REST API.

Python Health Sidecar

# livy_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

LIVY_URL             = os.environ.get('LIVY_URL',           'http://localhost:8998')
MAX_DEAD_SESSIONS    = int(os.environ.get('MAX_DEAD_SESSIONS',   '5'))
MAX_ERROR_SESSIONS   = int(os.environ.get('MAX_ERROR_SESSIONS',  '3'))
LIVY_USER            = os.environ.get('LIVY_USER',           '')
LIVY_PASS            = os.environ.get('LIVY_PASS',           '')

def livy_headers():
    if LIVY_USER:
        import base64
        token = base64.b64encode(f'{LIVY_USER}:{LIVY_PASS}'.encode()).decode()
        return {'Authorization': f'Basic {token}'}
    return {}

@app.route('/health/livy')
def livy_health():
    try:
        r = requests.get(f'{LIVY_URL}/sessions', headers=livy_headers(), timeout=5)
        if r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'http_{r.status_code}'}), 503
        data = r.json()
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    sessions = data.get('sessions', [])
    dead_count  = sum(1 for s in sessions if s.get('state') in ('dead', 'error'))
    total_count = len(sessions)

    if dead_count >= MAX_DEAD_SESSIONS:
        return jsonify({
            'status': 'down',
            'error': f'{dead_count}_dead_or_error_sessions',
            'total_sessions': total_count
        }), 503

    return jsonify({
        'status': 'ok',
        'total_sessions': total_count,
        'dead_or_error_sessions': dead_count
    }), 200

@app.route('/health/livy/batches')
def livy_batches():
    try:
        r = requests.get(f'{LIVY_URL}/batches', headers=livy_headers(), timeout=5)
        r.raise_for_status()
        batches = r.json().get('sessions', [])  # Livy uses "sessions" key for batches too
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    running_count = sum(1 for b in batches if b.get('state') in ('running', 'starting'))
    error_count   = sum(1 for b in batches if b.get('state') in ('dead', 'error'))

    if error_count >= MAX_ERROR_SESSIONS:
        return jsonify({
            'status': 'down',
            'error': f'{error_count}_failed_batches',
            'running': running_count
        }), 503

    return jsonify({
        'status': 'ok',
        'running_batches': running_count,
        'failed_batches': error_count
    }), 200

@app.route('/health/livy/sessions/<session_id>')
def livy_session_detail(session_id):
    try:
        r = requests.get(f'{LIVY_URL}/sessions/{session_id}', headers=livy_headers(), timeout=5)
        r.raise_for_status()
        session = r.json()
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    state = session.get('state', 'unknown')
    if state in ('dead', 'error', 'shutting_down'):
        return jsonify({'status': 'down', 'state': state, 'session_id': session_id}), 503
    if state not in ('idle', 'busy', 'running'):
        return jsonify({'status': 'degraded', 'state': state, 'session_id': session_id}), 200

    return jsonify({'status': 'ok', 'state': state, 'session_id': session_id}), 200

if __name__ == '__main__':
    app.run(port=8099)

Node.js Health Sidecar

// livy-health.js
const express = require('express');
const axios   = require('axios');

const app = express();

const LIVY_URL           = process.env.LIVY_URL           || 'http://localhost:8998';
const MAX_DEAD_SESSIONS  = parseInt(process.env.MAX_DEAD_SESSIONS  || '5');
const MAX_ERROR_BATCHES  = parseInt(process.env.MAX_ERROR_BATCHES  || '3');

app.get('/health/livy', async (req, res) => {
  let data;
  try {
    const r = await axios.get(`${LIVY_URL}/sessions`, { timeout: 5000 });
    data = r.data;
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }

  const sessions   = data.sessions || [];
  const deadCount  = sessions.filter(s => ['dead', 'error'].includes(s.state)).length;

  if (deadCount >= MAX_DEAD_SESSIONS) {
    return res.status(503).json({ status: 'down', dead_or_error_sessions: deadCount });
  }
  return res.status(200).json({ status: 'ok', total_sessions: sessions.length, dead_or_error_sessions: deadCount });
});

app.get('/health/livy/batches', async (req, res) => {
  let data;
  try {
    const r = await axios.get(`${LIVY_URL}/batches`, { timeout: 5000 });
    data = r.data;
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }

  const batches    = data.sessions || [];
  const errorCount = batches.filter(b => ['dead', 'error'].includes(b.state)).length;

  if (errorCount >= MAX_ERROR_BATCHES) {
    return res.status(503).json({ status: 'down', failed_batches: errorCount });
  }
  return res.status(200).json({ status: 'ok', running_batches: batches.filter(b => b.state === 'running').length, failed_batches: errorCount });
});

app.listen(3015, () => console.log('livy-health listening on 3015'));

Step 3: Configure Vigilmon HTTP Monitor for Livy

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to http://livy.example.com:8998/sessions
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "sessions"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your data engineering Slack channel
  7. Save the monitor

Add a second monitor for the session health sidecar:

  • URL: https://your-app.example.com/health/livy
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert channel: data platform on-call channel

Add a batch health monitor:

  • URL: https://your-app.example.com/health/livy/batches
  • Expected: 200
  • Interval: 5 minutes
  • Alert channel: data engineering team Slack channel

If you run Livy in high-availability mode (multiple Livy server instances behind a load balancer), create a monitor for the load balancer endpoint and individual health monitors for each Livy node.

Step 4: Heartbeat Monitoring for Livy Batch Jobs

Session and batch health checks tell you the Livy service is operational — but they don't confirm that your scheduled batch Spark jobs are submitting, executing, and completing successfully on schedule. A batch job could be stuck in starting state (YARN queue depth), timing out at the Spark driver level without Livy reporting it as failed, or skipped entirely by an upstream scheduler.

Vigilmon heartbeat monitors detect these failures: the job submission wrapper pings Vigilmon after each successful batch completion.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: livy-etl-batch-job
  3. Set the expected interval: 1 hour (or your job schedule)
  4. Set the grace period: 30 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Python Batch Submission with Heartbeat

# livy_batch_runner.py — submit a Livy batch job and ping heartbeat on success
import requests
import time
import os

LIVY_URL       = os.environ['LIVY_URL']
VIGILMON_HB    = os.environ['VIGILMON_HEARTBEAT_URL']
SPARK_APP_JAR  = os.environ.get('SPARK_APP_JAR',  'hdfs:///apps/etl-job.jar')
SPARK_APP_CLASS = os.environ.get('SPARK_APP_CLASS', 'com.example.ETLJob')
POLL_INTERVAL  = 15   # seconds
MAX_WAIT       = 7200 # 2 hours

# Submit batch
payload = {
    'file':      SPARK_APP_JAR,
    'className': SPARK_APP_CLASS,
    'numExecutors':  4,
    'executorCores': 2,
    'executorMemory': '4g',
    'driverMemory':   '2g',
    'conf': {
        'spark.app.name': 'vigilmon-heartbeat-etl'
    }
}

r = requests.post(f'{LIVY_URL}/batches', json=payload, timeout=30)
r.raise_for_status()
batch_id = r.json()['id']
print(f"Livy batch {batch_id} submitted.")

# Poll until terminal state
elapsed = 0
final_state = None
while elapsed < MAX_WAIT:
    time.sleep(POLL_INTERVAL)
    elapsed += POLL_INTERVAL
    status_r = requests.get(f'{LIVY_URL}/batches/{batch_id}', timeout=15)
    status_r.raise_for_status()
    state = status_r.json().get('state', 'unknown')
    print(f"  [{elapsed}s] batch {batch_id} state: {state}")
    if state in ('success',):
        final_state = state
        break
    if state in ('dead', 'error'):
        raise RuntimeError(f"Livy batch {batch_id} failed with state: {state}")
else:
    raise TimeoutError(f"Livy batch {batch_id} did not complete within {MAX_WAIT}s.")

# Success — send heartbeat
requests.get(VIGILMON_HB, timeout=10)
print(f"Vigilmon heartbeat sent. Batch {batch_id} completed with state: {final_state}")

Bash Wrapper for Cron-Scheduled Jobs

#!/bin/bash
# livy_batch_cron.sh — wrapper for scheduled Livy batch submissions with heartbeat
set -e

LIVY_URL="${LIVY_URL:-http://livy.example.com:8998}"
VIGILMON_HB="${VIGILMON_HEARTBEAT_URL}"
JAR_PATH="${SPARK_APP_JAR:-hdfs:///apps/etl-job.jar}"
APP_CLASS="${SPARK_APP_CLASS:-com.example.ETLJob}"

echo "[$(date)] Submitting Livy batch job..."

BATCH_ID=$(curl -s -X POST "${LIVY_URL}/batches" \
  -H "Content-Type: application/json" \
  -d "{\"file\":\"${JAR_PATH}\",\"className\":\"${APP_CLASS}\"}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")

echo "[$(date)] Batch ID: ${BATCH_ID}. Polling for completion..."

for i in $(seq 1 240); do
  sleep 30
  STATE=$(curl -s "${LIVY_URL}/batches/${BATCH_ID}" \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['state'])")
  echo "[$(date)] State: ${STATE}"
  if [ "${STATE}" = "success" ]; then
    echo "[$(date)] Batch succeeded. Pinging Vigilmon..."
    curl -s --max-time 10 "${VIGILMON_HB}" > /dev/null
    echo "[$(date)] Heartbeat sent."
    exit 0
  fi
  if [ "${STATE}" = "dead" ] || [ "${STATE}" = "error" ]; then
    echo "[$(date)] ERROR: Batch ${BATCH_ID} failed with state ${STATE}."
    exit 1
  fi
done

echo "[$(date)] ERROR: Batch ${BATCH_ID} did not complete within 2 hours."
exit 1

Step 5: Alert Routing for Livy Failures

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Livy sessions API endpoint | Slack + PagerDuty | P1 | | Session health sidecar | Slack + PagerDuty | P1 | | Batch health sidecar | Slack | P2 | | Heartbeat: scheduled ETL batch | Slack + email | P1 | | Heartbeat: ad-hoc batch pipeline | Email | P2 |

Set response time thresholds for early warning:

  • Alert at 3000ms for the Livy sessions endpoint (slow responses signal Livy JVM pressure or YARN connectivity issues)
  • Alert at 8000ms for the health sidecar (slow sidecar responses may indicate that Livy's session enumeration is blocked)

For data pipelines that feed production dashboards or scheduled reports, configure a tight heartbeat grace period: a batch that runs every hour and takes 20 minutes should alert within 40 minutes of missing its expected completion time, not hours later when business users report missing data.

Summary

Livy failures span service availability, session and batch state health, and end-to-end job completion. External monitoring catches all three:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /sessions | Livy REST service liveness | | HTTP monitor on session health sidecar | Dead/error session accumulation | | HTTP monitor on batch health sidecar | Failed batch job detection | | Heartbeat monitor on batch completion | Scheduled Spark job execution success |

Get started free at vigilmon.online — your first Livy availability monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles