tutorial

How to Monitor Apache Tez DAG Execution and AM Health with Vigilmon

Silent Tez Application Master crashes and hung DAG tasks block your Hive and Pig workloads for hours without external alerts. Learn how to monitor Tez AM availability, DAG execution health, and session heartbeats with Vigilmon.

Apache Tez is the DAG-based execution framework that replaces MapReduce for Hive, Pig, and Cascading workloads on Hadoop — delivering dramatically faster performance through in-memory data passing and container reuse. But when a Tez Application Master crashes due to an OOM kill, a DAG vertex gets stuck retrying a failed task indefinitely, or a Tez session expires because YARN reclaimed its containers, your Hive queries and Pig scripts hang without any user-visible error. The Tez AM web UI on port 8188 and the YARN tracking UI provide some visibility, but they require active polling and do not alert you proactively.

Vigilmon gives you external visibility into Tez health through the YARN Timeline Server and a custom health sidecar, plus heartbeat monitors for your Tez-backed query workloads. This tutorial covers both.

Why Tez Needs External Monitoring

Tez's built-in observability (the Tez UI on the YARN Timeline Server, individual DAG status via the Tez REST API, and Ambari/Cloudera Manager integration) requires active watching. External monitoring with Vigilmon adds:

  • Proactive alerting when the YARN Timeline Server (required for Tez UI and DAG history) becomes unreachable
  • Application Master crash detection — Tez AM failures surface as failed YARN applications, not HTTP 5xx errors
  • DAG execution health checks confirming sessions are accepting and completing DAGs
  • Session heartbeats for long-running Tez sessions (HiveServer2, Pig Grunt with shared sessions)
  • Multi-region visibility from outside your Hadoop cluster network perimeter

Tez failures are particularly insidious because a shared Tez session can accept new DAG submissions while processing them into an error state — queries queue, appear to run, and return errors only at completion — while all cluster health metrics look normal.

Step 1: Build a Tez Health Endpoint

Tez exposes a REST API via the YARN Timeline Server (port 8188) and through YARN's ResourceManager. Build a sidecar that checks both, plus optionally submits a lightweight test DAG.

Python Health Sidecar (Tez REST + YARN)

# tez_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

YARN_TIMELINE_URL = os.environ.get('YARN_TIMELINE_URL', 'http://timeline-server.example.com:8188')
YARN_RM_HTTP      = os.environ.get('YARN_RM_HTTP',      'http://resourcemanager.example.com:8088')
TEZ_APP_ID        = os.environ.get('TEZ_APP_ID')  # Optional: monitor a specific long-running Tez session

@app.route('/health/tez')
def tez_health():
    errors = []

    # Check YARN Timeline Server (required for Tez DAG history and UI)
    try:
        r = requests.get(f'{YARN_TIMELINE_URL}/ws/v1/timeline/', timeout=5)
        if r.status_code != 200:
            errors.append(f'timeline_server_http_{r.status_code}')
    except Exception as e:
        errors.append(f'timeline_server_unreachable:{e}')

    # Check YARN ResourceManager (Tez AMs run as YARN applications)
    try:
        r = requests.get(f'{YARN_RM_HTTP}/ws/v1/cluster/info', timeout=5)
        if r.status_code != 200:
            errors.append(f'yarn_rm_http_{r.status_code}')
        else:
            state = r.json().get('clusterInfo', {}).get('state', '')
            if state != 'STARTED':
                errors.append(f'yarn_rm_state_{state}')
    except Exception as e:
        errors.append(f'yarn_rm_unreachable:{e}')

    if errors:
        return jsonify({'status': 'down', 'errors': errors}), 503
    return jsonify({'status': 'ok'}), 200

@app.route('/health/tez/session')
def tez_session_health():
    """Check the status of a specific long-running Tez session (e.g. HiveServer2's shared session)."""
    app_id = TEZ_APP_ID or request.args.get('appId')
    if not app_id:
        return jsonify({'status': 'skipped', 'reason': 'no_app_id_configured'}), 200
    try:
        r = requests.get(
            f'{YARN_RM_HTTP}/ws/v1/cluster/apps/{app_id}',
            timeout=5
        )
        if r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'app_not_found_{r.status_code}'}), 503
        app_state = r.json().get('app', {}).get('state', '')
        app_status = r.json().get('app', {}).get('finalStatus', '')
        if app_state not in ('RUNNING', 'ACCEPTED'):
            return jsonify({'status': 'down', 'state': app_state, 'final_status': app_status}), 503
        return jsonify({'status': 'ok', 'app_state': app_state}), 200
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

@app.route('/health/tez/dags')
def tez_recent_dags():
    """Check that recent DAGs are completing successfully (no high error rate)."""
    try:
        # Query Timeline Server for recent Tez DAGs
        r = requests.get(
            f'{YARN_TIMELINE_URL}/ws/v1/timeline/TEZ_DAG_ID',
            params={'limit': 10, 'sortBy': 'CREATED_TIME', 'sortOrder': 'DESCENDING'},
            timeout=8
        )
        if r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'timeline_http_{r.status_code}'}), 503
        dags = r.json().get('entities', [])
        if not dags:
            return jsonify({'status': 'ok', 'recent_dags': 0}), 200
        failed = [d for d in dags
                  if d.get('otherinfo', {}).get('status') == 'FAILED']
        fail_rate = len(failed) / len(dags)
        if fail_rate > 0.5:
            return jsonify({'status': 'degraded', 'recent_fail_rate': round(fail_rate, 2)}), 503
        return jsonify({'status': 'ok', 'recent_dags': len(dags), 'failed': len(failed)}), 200
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8096)

Node.js Health Sidecar

// tez-health.js
const express = require('express');
const axios   = require('axios');

const app = express();

const YARN_TIMELINE_URL = process.env.YARN_TIMELINE_URL || 'http://timeline-server.example.com:8188';
const YARN_RM_HTTP      = process.env.YARN_RM_HTTP      || 'http://resourcemanager.example.com:8088';

app.get('/health/tez', async (req, res) => {
  const errors = [];

  // Check Timeline Server
  try {
    await axios.get(`${YARN_TIMELINE_URL}/ws/v1/timeline/`, { timeout: 5000 });
  } catch (e) {
    errors.push(`timeline_server_unreachable:${e.message}`);
  }

  // Check YARN RM
  try {
    const rm = await axios.get(`${YARN_RM_HTTP}/ws/v1/cluster/info`, { timeout: 5000 });
    const state = rm.data?.clusterInfo?.state;
    if (state !== 'STARTED') errors.push(`yarn_rm_state_${state}`);
  } catch (e) {
    errors.push(`yarn_rm_unreachable:${e.message}`);
  }

  if (errors.length) return res.status(503).json({ status: 'down', errors });
  return res.status(200).json({ status: 'ok' });
});

app.listen(3012, () => console.log('tez-health listening on 3012'));

Step 2: Configure Vigilmon HTTP Monitor for Tez

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Tez health endpoint: https://your-app.example.com/health/tez
  4. Set the check interval to 2 minutes
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 8000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for DAG error rate tracking:

  • URL: https://your-app.example.com/health/tez/dags
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert channel: data-platform team Slack channel

And a direct YARN Timeline Server probe:

  • URL: http://timeline-server.example.com:8188/ws/v1/timeline/
  • Expected: 200
  • Interval: 1 minute
  • Alert channel: Hadoop infrastructure on-call

If you run a persistent Tez session for HiveServer2, add a session monitor:

  • URL: https://your-app.example.com/health/tez/session
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert channel: Hive on-call channel

Step 3: Heartbeat Monitoring for Tez Query Pipelines

Infrastructure health confirms Tez can run DAGs — but it does not confirm your specific workloads are completing. Your Hive ETL jobs (which use Tez under the hood), Pig pipelines configured to use the Tez execution engine, or custom Tez applications may be failing silently due to data quality issues, schema changes in source tables, or vertex-level configuration problems.

Vigilmon heartbeat monitors detect these silent stalls: your pipeline pings Vigilmon after each successful DAG completion (or batch of completions). If pings stop, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: tez-hive-etl-pipeline
  3. Set the expected interval: 6 hours (or your ETL cadence)
  4. Set the grace period: 1 hour
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Hive/Tez ETL Script

#!/bin/bash
# hive_tez_etl.sh — Hive-on-Tez ETL with Vigilmon heartbeat

set -e

HIVE_URL="jdbc:hive2://${HIVE_HOST}:10000/${HIVE_DATABASE}"
VIGILMON_HB="${VIGILMON_HEARTBEAT_URL}"

echo "[$(date)] Running Hive-on-Tez ETL queries..."

beeline \
  --hiveconf hive.execution.engine=tez \
  -u "$HIVE_URL" \
  -n "${HIVE_USERNAME}" \
  -f /opt/etl/transform.hql

if [ $? -eq 0 ]; then
  echo "[$(date)] ETL complete. Pinging Vigilmon heartbeat..."
  curl -s --max-time 10 "$VIGILMON_HB" > /dev/null
  echo "[$(date)] Done."
else
  echo "[$(date)] ETL FAILED. Heartbeat NOT sent."
  exit 1
fi

Direct Tez Application with Heartbeat (Java / DAGClient)

// TezJobRunner.java — simplified Tez DAG submission with heartbeat
import org.apache.tez.client.TezClient;
import org.apache.tez.dag.api.*;
import org.apache.tez.dag.api.client.*;
import java.net.HttpURLConnection;
import java.net.URL;

public class TezJobRunner {

    public static void main(String[] args) throws Exception {
        TezConfiguration tezConf = new TezConfiguration();
        TezClient tezClient = TezClient.newBuilder("my-tez-app", tezConf).build();
        tezClient.start();

        try {
            DAG dag = buildMyDAG(); // build your vertex/edge graph
            DAGClient dagClient = tezClient.submitDAG(dag);
            DAGStatus status = dagClient.waitForCompletion();

            if (status.getState() == DAGStatus.State.SUCCEEDED) {
                pingVigilmon(System.getenv("VIGILMON_HEARTBEAT_URL"));
                System.out.println("DAG succeeded. Heartbeat sent.");
            } else {
                System.err.println("DAG failed: " + status.getState());
                System.exit(1);
            }
        } finally {
            tezClient.stop();
        }
    }

    private static void pingVigilmon(String heartbeatUrl) throws Exception {
        if (heartbeatUrl == null || heartbeatUrl.isEmpty()) return;
        HttpURLConnection conn = (HttpURLConnection) new URL(heartbeatUrl).openConnection();
        conn.setRequestMethod("GET");
        conn.setConnectTimeout(10_000);
        conn.getResponseCode();
        conn.disconnect();
    }

    private static DAG buildMyDAG() {
        // ... construct vertices and edges ...
        return DAG.create("my-dag");
    }
}

Pig on Tez with Heartbeat

#!/bin/bash
# pig_tez.sh — Pig using Tez execution engine + heartbeat

set -e

pig -x tez \
    -param INPUT="/user/pig/input" \
    -param OUTPUT="/user/pig/output/$(date +%Y%m%d)" \
    /opt/pipelines/transform.pig

curl -s --max-time 10 "$VIGILMON_HEARTBEAT_URL" > /dev/null
echo "Pig-on-Tez job complete. Heartbeat sent."

Step 4: Alert Routing for Tez Failures

Tez failures have distinct urgency levels: Timeline Server outages break Tez UI and DAG tracking but don't immediately block query execution; YARN ResourceManager failures block all new DAG submissions; high DAG failure rates indicate query-level issues.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | YARN ResourceManager (port 8088) | Slack + PagerDuty | P1 | | Tez cluster health /health/tez | Slack + PagerDuty | P1 | | YARN Timeline Server (port 8188) | Slack | P2 | | Tez session health /health/tez/session | Slack + PagerDuty | P1 (if HiveServer2) | | DAG error rate /health/tez/dags | Slack | P2 | | Heartbeat: Hive ETL pipeline | Slack + email | P2 | | Heartbeat: Pig-on-Tez pipeline | Email | P3 |

Set response time thresholds for early warning:

  • Alert at 6000ms for cluster health (slow YARN REST responses signal RM GC or network partition)
  • Alert at 12000ms for the DAG error rate check (slow Timeline Server queries signal ATS load or compaction delays)

For latency-sensitive Hive workloads, configure a DAG degradation threshold: if more than 30% of recent DAGs are failing, escalate to the data platform team even before a complete outage — partial failures indicate a systematic problem (schema drift, HDFS quota exhaustion, or executor memory misconfiguration) that will worsen without intervention.

Summary

Tez failures span infrastructure layers (YARN, Timeline Server), session-level (AM crashes), and workload-level (high DAG failure rates). External monitoring catches all three:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on YARN ResourceManager | Cluster availability, container scheduling | | HTTP monitor on Tez cluster health | Timeline Server + YARN combined reachability | | HTTP monitor on Tez session endpoint | Long-running AM liveness (HiveServer2 sessions) | | HTTP monitor on DAG error rate | Recent DAG success/failure ratio | | Heartbeat monitor | End-to-end pipeline completion, data freshness |

Get started free at vigilmon.online — your first Tez workload monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles