tutorial

How to Monitor Apache Oozie Workflow Scheduler Health with Vigilmon

Oozie failures silently stall all scheduled Hadoop workflows. Learn how to monitor Oozie server health, system status, and workflow job heartbeats with Vigilmon HTTP probes and heartbeat monitors.

Apache Oozie is the workflow orchestration system for Hadoop — it schedules MapReduce, Spark, Hive, Pig, and Shell actions into DAG-based workflows that run on time-based or data-triggered schedules. When Oozie goes down, every scheduled ETL pipeline, batch processing job, and data ingestion workflow silently stops running. When Oozie is degraded — coordinator jobs stuck in WAITING, workflow actions timing out against YARN, or the Oozie DB falling behind — failures appear as stale data or missed SLA windows hours later, not as immediate alerts.

Vigilmon gives you external visibility into Oozie scheduler health through HTTP probe monitoring (via Oozie's built-in REST API) and heartbeat monitoring for coordinator and bundle jobs. This tutorial covers both.

Why Oozie Needs External Monitoring

Oozie has its own Web Console for inspecting job status, but it only helps when you're actively checking. External monitoring with Vigilmon adds:

  • Proactive alerting when the Oozie server REST API becomes unreachable — the earliest signal that your entire workflow schedule has stalled
  • System status checks that confirm Oozie is connected to its database and the YARN resource manager
  • Heartbeat monitoring for coordinator jobs and bundles, so missed scheduled runs trigger an alert before downstream systems notice stale data
  • SLA breach detection — catching workflows that complete too slowly via Oozie's SLA REST API

These layers complement each other: the Oozie server can appear healthy while a coordinator job is stuck in WAITING due to a failed YARN dependency check, which the heartbeat monitor catches before the SLA window closes.

Step 1: Build an Oozie Health Endpoint

Oozie exposes a REST API on port 11000 (HTTP) or 11443 (HTTPS). Key endpoints for monitoring:

Using the Oozie REST API Directly

# Oozie system status — returns NORMAL, SAFEMODE, or SYSTEMMODE
curl -i http://oozie-server.example.com:11000/oozie/v1/admin/status

# Oozie build version — basic liveness check
curl -i http://oozie-server.example.com:11000/oozie/v1/admin/build-version

# List running workflow jobs (confirms DB and YARN connectivity)
curl -i "http://oozie-server.example.com:11000/oozie/v1/jobs?jobtype=wf&filter=status%3DRUNNING&offset=1&len=10"

# Check for coordinator jobs in PAUSED or DONEWITHERROR state
curl -i "http://oozie-server.example.com:11000/oozie/v1/jobs?jobtype=coordinator&filter=status%3DPAUSED&offset=1&len=10"

The /admin/status endpoint is the canonical Oozie health check — NORMAL means the server is healthy, SAFEMODE means it's running but not accepting new job submissions, and any failure to respond means the server is down.

Python Health Sidecar

# oozie_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

OOZIE_URL = os.environ.get('OOZIE_URL', 'http://localhost:11000/oozie')
MAX_PAUSED_COORDINATORS = int(os.environ.get('MAX_PAUSED_COORDS', '0'))

@app.route('/health/oozie')
def oozie_health():
    try:
        # Check system status
        status_resp = requests.get(
            f'{OOZIE_URL}/v1/admin/status',
            timeout=5
        )
        if status_resp.status_code != 200:
            return jsonify({
                'status': 'down',
                'reason': 'admin_api_error',
                'http_status': status_resp.status_code,
            }), 503

        system_status = status_resp.json().get('systemMode', 'UNKNOWN')
        if system_status != 'NORMAL':
            return jsonify({
                'status': 'degraded',
                'reason': 'not_in_normal_mode',
                'system_mode': system_status,
            }), 503

        # Check for paused coordinators (often indicates YARN or dependency issues)
        coord_resp = requests.get(
            f'{OOZIE_URL}/v1/jobs',
            params={'jobtype': 'coordinator', 'filter': 'status=PAUSED', 'offset': 1, 'len': 50},
            timeout=8
        )
        paused_count = 0
        if coord_resp.status_code == 200:
            paused_count = coord_resp.json().get('total', 0)

        if paused_count > MAX_PAUSED_COORDINATORS:
            return jsonify({
                'status': 'degraded',
                'reason': 'coordinators_paused',
                'paused_coordinator_count': paused_count,
            }), 503

        # Check running workflow count (confirms DB and YARN connectivity)
        wf_resp = requests.get(
            f'{OOZIE_URL}/v1/jobs',
            params={'jobtype': 'wf', 'filter': 'status=RUNNING', 'offset': 1, 'len': 1},
            timeout=8
        )
        wf_total = 0
        if wf_resp.status_code == 200:
            wf_total = wf_resp.json().get('total', 0)

        return jsonify({
            'status': 'ok',
            'system_mode': system_status,
            'running_workflows': wf_total,
            'paused_coordinators': paused_count,
        }), 200

    except requests.RequestException as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8092)

Shell Script Health Check

#!/bin/bash
# oozie_health.sh
OOZIE_URL="${OOZIE_URL:-http://localhost:11000/oozie}"
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"

# Check system mode
RESPONSE=$(curl -sf --max-time 10 "${OOZIE_URL}/v1/admin/status")
STATUS=$?

if [ $STATUS -ne 0 ]; then
  echo "Oozie admin status check failed"
  exit 1
fi

SYSTEM_MODE=$(echo "$RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin).get('systemMode','UNKNOWN'))")

if [ "$SYSTEM_MODE" = "NORMAL" ]; then
  curl -s --max-time 5 "${HEARTBEAT_URL}" > /dev/null
  echo "Oozie healthy (mode: $SYSTEM_MODE) — heartbeat sent"
else
  echo "Oozie in non-normal mode: $SYSTEM_MODE — heartbeat NOT sent"
  exit 1
fi

Step 2: Configure Vigilmon HTTP Monitor for Oozie

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Oozie health sidecar: https://your-sidecar.example.com/health/oozie
  4. Set the check interval to 2 minutes
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a direct Oozie admin check (no auth required for the status endpoint):

  • URL: http://oozie-server.example.com:11000/oozie/v1/admin/status
  • Expected response contains: NORMAL
  • Interval: 2 minutes

Vigilmon's multi-region probes confirm that Oozie is accessible from multiple vantage points before alerting, preventing false positives from single-region network issues.

Step 3: Heartbeat Monitoring for Coordinator and Bundle Jobs

Oozie coordinator jobs run on a schedule — hourly, daily, or event-triggered. The most common silent failure is a coordinator job that is technically RUNNING but has no materialized actions for the current time window due to a data dependency or YARN queue backpressure. The Oozie server appears healthy while no actual work is being done.

Heartbeat monitoring detects this: your coordinator's action script pings Vigilmon after each successful execution. Missed pings alert before downstream consumers notice stale data.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: oozie-etl-coordinator (use your actual coordinator name)
  3. Set the expected interval to match your coordinator frequency — for an hourly coordinator, set 65 minutes
  4. Set the grace period: 30 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Oozie Workflow Action Script

Add a heartbeat ping as the last step in your workflow's Shell or Spark action:

#!/bin/bash
# workflow_action.sh — last step in your Oozie workflow
set -e

# ... your actual processing logic here ...
hdfs dfs -cp /data/input/today /data/output/processed/

# Ping Vigilmon on success
# Uses curl with -f so any HTTP error makes the ping fail silently (don't let heartbeat failures fail the workflow)
curl -sf --max-time 5 "${VIGILMON_HEARTBEAT_URL}" > /dev/null || true

echo "Workflow action complete"

Pass VIGILMON_HEARTBEAT_URL as an Oozie workflow property in your job.properties:

# job.properties
nameNode=hdfs://namenode:8020
jobTracker=resourcemanager:8032
oozie.wf.application.path=${nameNode}/user/${user.name}/workflows/etl
VIGILMON_HEARTBEAT_URL=https://vigilmon.online/heartbeat/abc123xyz

Reference it in your workflow.xml:

<workflow-app name="etl-pipeline" xmlns="uri:oozie:workflow:0.5">
  <start to="process-data"/>

  <action name="process-data">
    <shell xmlns="uri:oozie:shell-action:0.3">
      <job-tracker>${jobTracker}</job-tracker>
      <name-node>${nameNode}</name-node>
      <exec>workflow_action.sh</exec>
      <env-var>VIGILMON_HEARTBEAT_URL=${VIGILMON_HEARTBEAT_URL}</env-var>
      <file>workflow_action.sh#workflow_action.sh</file>
      <capture-output/>
    </shell>
    <ok to="end"/>
    <error to="fail"/>
  </action>

  <kill name="fail">
    <message>ETL workflow failed at ${wf:lastErrorNode()}</message>
  </kill>

  <end name="end"/>
</workflow-app>

Step 4: Alert Routing for Oozie Failures

| Monitor | Alert Channel | Priority | |---|---|---| | Oozie health /health/oozie | Slack + PagerDuty | P1 | | Oozie admin status direct | Slack | P1 | | Heartbeat: coordinator jobs | Slack + email | P2 | | Heartbeat: critical ETL workflows | Slack + PagerDuty | P1 |

Set response time thresholds:

  • Alert at 8000ms for health endpoint (slow Oozie queries signal DB pressure or coordinator backlog)
  • Alert at 15000ms for job listing queries (large job histories cause slow DB scans)

For SLA-sensitive pipelines (nightly batch jobs that feed dashboards or downstream systems), use a tighter grace period on the heartbeat monitor — if a workflow is scheduled for midnight and hasn't pinged by 00:15, you want to know before anyone's morning dashboard shows yesterday's data.

Summary

Oozie orchestrates all scheduled Hadoop workloads. Failures are often silent — the server stays up while coordinator jobs stall in dependency resolution or YARN backpressure:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/oozie | Server liveness, system mode, paused coordinator detection | | HTTP monitor on Oozie admin status | API availability, DB connectivity | | Heartbeat monitor | Coordinator job execution, ETL pipeline completion |

Get started free at vigilmon.online — your first Oozie monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles