tutorial

How to Monitor Apache Pig Script Execution and Grunt Shell Health with Vigilmon

Silent Pig script failures and hung Grunt sessions can stall your Hadoop ETL pipelines for hours without anyone noticing. Learn how to monitor Apache Pig job liveness, YARN submission health, and data pipeline heartbeats with Vigilmon.

Apache Pig is the high-level data flow scripting language that compiles Pig Latin statements into MapReduce (or Tez) jobs on Hadoop — but when a Pig script hangs waiting for YARN containers, fails silently mid-job due to a UDF class-not-found error, or never gets submitted because the Grunt shell lost its HDFS connection, your data pipelines stall without firing any obvious alarm. Pig's default exit codes and log verbosity make these failures easy to miss in automated pipelines.

Vigilmon gives you external visibility into Pig job health through a lightweight health sidecar that checks HDFS connectivity and YARN reachability, plus heartbeat monitors that confirm each Pig pipeline successfully completes on schedule. This tutorial covers both.

Why Pig Needs External Monitoring

Pig's built-in observability (YARN history server, job counters, Oozie workflow logs) requires active polling. External monitoring with Vigilmon adds:

  • Proactive alerting when Pig pipelines stop running or completing successfully
  • HDFS connectivity detection before Pig jobs are even submitted
  • YARN reachability checks so you know the cluster can accept new jobs
  • Pipeline heartbeats that fire only after a Pig script fully succeeds — catching partial failures that exit 0 incorrectly
  • Multi-region visibility from outside your Hadoop cluster network perimeter

These layers matter because Pig failures span multiple dimensions: HDFS I/O errors surface at runtime (not submission time), UDF failures surface after the map phase starts, and YARN resource exhaustion causes indefinite queuing that looks identical to normal job startup from the outside.

Step 1: Build a Pig Ecosystem Health Endpoint

Pig itself does not expose an HTTP health port, but you can build a thin sidecar that probes the services Pig depends on: HDFS NameNode, YARN ResourceManager, and optionally a test Pig job submission.

Python Health Sidecar (hdfs + requests)

# pig_health.py
from flask import Flask, jsonify
import requests
import subprocess
import os

app = Flask(__name__)

NAMENODE_HTTP = os.environ.get('NAMENODE_HTTP', 'http://namenode.example.com:9870')
YARN_RM_HTTP  = os.environ.get('YARN_RM_HTTP',  'http://resourcemanager.example.com:8088')

@app.route('/health/pig')
def pig_health():
    errors = []

    # Check HDFS NameNode web UI
    try:
        r = requests.get(f'{NAMENODE_HTTP}/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus',
                         timeout=5)
        if r.status_code != 200:
            errors.append(f'namenode_http_{r.status_code}')
        else:
            state = r.json().get('beans', [{}])[0].get('State', '')
            if state != 'active':
                errors.append(f'namenode_state_{state}')
    except Exception as e:
        errors.append(f'namenode_unreachable:{e}')

    # Check YARN ResourceManager web UI
    try:
        r = requests.get(f'{YARN_RM_HTTP}/ws/v1/cluster/info', timeout=5)
        if r.status_code != 200:
            errors.append(f'yarn_rm_http_{r.status_code}')
        else:
            rm_state = r.json().get('clusterInfo', {}).get('state', '')
            if rm_state != 'STARTED':
                errors.append(f'yarn_rm_state_{rm_state}')
    except Exception as e:
        errors.append(f'yarn_rm_unreachable:{e}')

    if errors:
        return jsonify({'status': 'down', 'errors': errors}), 503
    return jsonify({'status': 'ok'}), 200

@app.route('/health/pig/hdfs')
def pig_hdfs_health():
    """Check that the Pig working directory exists and is writable in HDFS."""
    pig_hdfs_dir = os.environ.get('PIG_HDFS_DIR', '/user/pig')
    try:
        result = subprocess.run(
            ['hdfs', 'dfs', '-test', '-d', pig_hdfs_dir],
            capture_output=True, timeout=10
        )
        if result.returncode != 0:
            return jsonify({'status': 'down', 'error': f'hdfs_dir_missing:{pig_hdfs_dir}'}), 503
        return jsonify({'status': 'ok', 'hdfs_dir': pig_hdfs_dir}), 200
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8094)

Node.js Health Sidecar

// pig-health.js
const express = require('express');
const axios   = require('axios');

const app = express();

const NAMENODE_HTTP = process.env.NAMENODE_HTTP || 'http://namenode.example.com:9870';
const YARN_RM_HTTP  = process.env.YARN_RM_HTTP  || 'http://resourcemanager.example.com:8088';

app.get('/health/pig', async (req, res) => {
  const errors = [];

  try {
    const nn = await axios.get(
      `${NAMENODE_HTTP}/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus`,
      { timeout: 5000 }
    );
    const state = nn.data?.beans?.[0]?.State;
    if (state !== 'active') errors.push(`namenode_state_${state}`);
  } catch (e) {
    errors.push(`namenode_unreachable:${e.message}`);
  }

  try {
    const rm = await axios.get(`${YARN_RM_HTTP}/ws/v1/cluster/info`, { timeout: 5000 });
    const rmState = rm.data?.clusterInfo?.state;
    if (rmState !== 'STARTED') errors.push(`yarn_rm_state_${rmState}`);
  } catch (e) {
    errors.push(`yarn_rm_unreachable:${e.message}`);
  }

  if (errors.length) return res.status(503).json({ status: 'down', errors });
  return res.status(200).json({ status: 'ok' });
});

app.listen(3010, () => console.log('pig-health listening on 3010'));

Step 2: Configure Vigilmon HTTP Monitor for Pig

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Pig health endpoint: https://your-app.example.com/health/pig
  4. Set the check interval to 2 minutes
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 8000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the HDFS directory check:

  • URL: https://your-app.example.com/health/pig/hdfs
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert channel: data-engineering on-call channel

And a direct YARN ResourceManager probe:

  • URL: http://resourcemanager.example.com:8088/ws/v1/cluster/info
  • Expected: 200
  • Interval: 1 minute
  • Alert channel: Hadoop infrastructure team

Step 3: Heartbeat Monitoring for Pig Pipelines

HDFS and YARN availability are necessary — but not sufficient. Your Pig scripts that perform data transformation (nightly aggregations, log parsing, feature extraction for ML pipelines) can be silently failing to complete: the script exits non-zero but your scheduler marks it as succeeded, or a UDF throws an exception that Pig swallows in a tryRelation block. Vigilmon heartbeat monitors detect these silent stalls.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: pig-nightly-aggregation
  3. Set the expected interval: 24 hours (or your pipeline cadence)
  4. Set the grace period: 2 hours
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Pig Pipeline Script

#!/bin/bash
# pig_pipeline.sh — Pig job wrapper with Vigilmon heartbeat

set -e

PIG_SCRIPT="/opt/pipelines/daily_agg.pig"
VIGILMON_HB="${VIGILMON_HEARTBEAT_URL}"

echo "[$(date)] Submitting Pig job: $PIG_SCRIPT"

pig -x mapreduce \
    -param INPUT_DATE="${INPUT_DATE:-$(date +%Y-%m-%d)}" \
    -param OUTPUT_PATH="/user/pig/output/daily" \
    "$PIG_SCRIPT"

EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ]; then
  echo "[$(date)] Pig job succeeded. Pinging Vigilmon heartbeat..."
  curl -s --max-time 10 "$VIGILMON_HB" > /dev/null
  echo "[$(date)] Done."
else
  echo "[$(date)] Pig job FAILED (exit $EXIT_CODE). Heartbeat NOT sent."
  exit $EXIT_CODE
fi

Python Pipeline Orchestrator with Heartbeat

# pig_orchestrator.py
import subprocess
import requests
import os
import sys
from datetime import date

PIG_SCRIPT   = os.environ['PIG_SCRIPT']
HDFS_OUTPUT  = os.environ.get('HDFS_OUTPUT', '/user/pig/output')
VIGILMON_HB  = os.environ['VIGILMON_HEARTBEAT_URL']

def run_pig_job():
    result = subprocess.run(
        [
            'pig', '-x', 'mapreduce',
            '-param', f'RUN_DATE={date.today().isoformat()}',
            '-param', f'OUTPUT={HDFS_OUTPUT}',
            PIG_SCRIPT,
        ],
        capture_output=True,
        text=True,
    )
    if result.returncode != 0:
        print(f"Pig FAILED:\n{result.stderr}", file=sys.stderr)
        raise RuntimeError(f"pig exited {result.returncode}")
    print(result.stdout)

if __name__ == '__main__':
    run_pig_job()
    # Ping Vigilmon only after successful completion
    requests.get(VIGILMON_HB, timeout=10)
    print("Vigilmon heartbeat sent.")

Oozie Workflow Integration

If you orchestrate Pig via Oozie, add a fork action that pings the heartbeat URL after the Pig action succeeds:

<!-- workflow.xml snippet -->
<action name="ping-vigilmon">
  <shell xmlns="uri:oozie:shell-action:0.2">
    <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
    <exec>curl</exec>
    <argument>-s</argument>
    <argument>${vigilmonHeartbeatUrl}</argument>
  </shell>
  <ok to="end"/>
  <error to="kill"/>
</action>

Step 4: Alert Routing for Pig Failures

Pig failures have distinct urgency levels: HDFS or YARN failures block all Pig jobs immediately, while pipeline heartbeat misses cause data freshness degradation that becomes critical over hours.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | YARN ResourceManager (port 8088) | Slack + PagerDuty | P1 | | HDFS NameNode health /health/pig | Slack + PagerDuty | P1 | | HDFS directory check /health/pig/hdfs | Slack | P2 | | Heartbeat: nightly aggregation pipeline | Slack + email | P2 | | Heartbeat: log parsing pipeline | Email | P3 |

Set response time thresholds for early warning:

  • Alert at 5000ms for the cluster health endpoint (slow responses signal NameNode GC pressure or network partitions)
  • Alert at 10000ms for the HDFS directory check (slow HDFS stat calls signal DataNode issues)

For pipelines with SLA requirements, configure a data freshness budget: if the ETL heartbeat misses beyond 1.5× its expected interval, escalate to the data platform team immediately — downstream reports and ML feature stores will be reading stale data before the next scheduled check.

Summary

Pig failures span cluster infrastructure (HDFS, YARN) and pipeline execution (script errors, UDF failures, resource starvation). External monitoring catches both layers:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on cluster health endpoint | HDFS NameNode + YARN ResourceManager availability | | HTTP monitor on HDFS directory check | Pig working directory existence and accessibility | | HTTP monitor on YARN ResourceManager directly | Cluster state, container availability | | Heartbeat monitor | Pig script completion, data pipeline liveness |

Get started free at vigilmon.online — your first Pig pipeline monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles