tutorial

How to Monitor Apache Giraph Jobs with Vigilmon

Apache Giraph runs iterative graph algorithms over billions of vertices on YARN — but job stalls, worker failures, and ZooKeeper timeouts fail silently inside a cluster. Here's how to monitor Giraph job health end-to-end with Vigilmon.

How to Monitor Apache Giraph Jobs with Vigilmon

Apache Giraph is Google Pregel's open-source equivalent — a BSP (Bulk Synchronous Parallel) graph processing system that runs iterative algorithms over massive graphs on Hadoop YARN. It powers graph analytics at Facebook's scale, executing PageRank, shortest-path, and community detection jobs over hundreds of billions of edges. But Giraph jobs fail in ways that standard cluster monitoring misses: a superstep can stall waiting on a slow worker, ZooKeeper session timeouts can orphan a computation, and YARN resource negotiation can leave a job queued indefinitely with no alert.

This tutorial wires up monitoring for Giraph workloads:

  • A job submission wrapper that emits heartbeats tied to superstep progress
  • HTTP uptime monitoring for the Giraph master's web UI with Vigilmon
  • Heartbeat-based job completion monitoring
  • ZooKeeper health checks
  • Slack alerts when jobs stall or miss SLAs

Why Monitor Giraph?

| Signal | What it catches | |---|---| | Master web UI availability | YARN application master died | | Superstep heartbeat | BSP stall, slow worker blocking global synchronization | | Job completion heartbeat | Job failed, exceeded wall-clock limit, queued indefinitely | | ZooKeeper availability | Worker coordination failure | | HDFS health | Input/output path unavailable, write failures |

YARN shows "RUNNING" for a job that's been stuck in the same superstep for 6 hours. Vigilmon heartbeats tell you if forward progress has stopped.

Step 1: Monitor the Giraph Master Web UI

Giraph's Application Master exposes a web UI on a dynamic YARN port. Use YARN's ResourceManager REST API to find the AM web URL and monitor it:

#!/usr/bin/env python3
# scripts/giraph_health_proxy.py
# Lightweight Flask proxy that exposes a stable health endpoint for Vigilmon

from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

RM_URL = os.environ.get('YARN_RM_URL', 'http://resourcemanager:8088')
APP_ID = os.environ.get('GIRAPH_APP_ID', '')  # e.g. application_1234567890_0001

@app.route('/health')
def health():
    if not APP_ID:
        return jsonify(status='error', reason='GIRAPH_APP_ID not set'), 503

    try:
        r = requests.get(f'{RM_URL}/ws/v1/cluster/apps/{APP_ID}', timeout=5)
        r.raise_for_status()
        app_data = r.json()['app']
        state = app_data['state']
        progress = app_data.get('progress', 0)

        if state in ('RUNNING', 'FINISHED') and app_data.get('finalStatus') != 'FAILED':
            return jsonify(
                status='ok',
                state=state,
                progress=progress,
                elapsedSeconds=app_data.get('elapsedTime', 0) // 1000
            )
        return jsonify(status='failed', state=state, finalStatus=app_data.get('finalStatus')), 503
    except Exception as e:
        return jsonify(status='error', reason=str(e)), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=9200)

Run the proxy alongside your job submission pipeline:

pip install flask requests
export YARN_RM_URL="http://resourcemanager:8088"
export GIRAPH_APP_ID="application_1234567890_0001"
python3 giraph_health_proxy.py &

Test it:

curl http://localhost:9200/health
# => {"status":"ok","state":"RUNNING","progress":67.5,"elapsedSeconds":3420}

Step 2: Emit Superstep Heartbeats

The most critical Giraph health signal is superstep progress. Wrap your job submission to emit a heartbeat after each superstep completes using Giraph's MasterCompute hook:

// src/main/java/com/example/giraph/HeartbeatMasterCompute.java
package com.example.giraph;

import org.apache.giraph.master.DefaultMasterCompute;
import org.apache.hadoop.conf.Configuration;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class HeartbeatMasterCompute extends DefaultMasterCompute {

    public static final String HEARTBEAT_URL_KEY = "giraph.vigilmon.heartbeat.url";
    private HttpClient httpClient;
    private String heartbeatUrl;

    @Override
    public void initialize() throws InstantiationException, IllegalAccessException {
        httpClient = HttpClient.newHttpClient();
        heartbeatUrl = getConf().get(HEARTBEAT_URL_KEY, "");
    }

    @Override
    public void compute() {
        long superstep = getSuperstep();
        long totalMessages = getTotalNumMessages();

        // Emit heartbeat after each superstep to prove forward progress
        if (!heartbeatUrl.isBlank()) {
            try {
                HttpRequest req = HttpRequest.newBuilder()
                    .uri(URI.create(heartbeatUrl))
                    .GET()
                    .build();
                httpClient.sendAsync(req, HttpResponse.BodyHandlers.discarding());
            } catch (Exception ignored) {
                // Never block the BSP superstep barrier on monitoring
            }
        }

        // Halt when no messages remain (algorithm converged)
        if (totalMessages == 0 && superstep > 0) {
            haltComputation();
        }
    }
}

Register it in your job configuration:

// src/main/java/com/example/giraph/MyGraphJob.java
GiraphConfiguration conf = new GiraphConfiguration(new Configuration());
conf.setMasterComputeClass(HeartbeatMasterCompute.class);
conf.set(HeartbeatMasterCompute.HEARTBEAT_URL_KEY,
         System.getenv("VIGILMON_HEARTBEAT_URL"));
// ... rest of your job config

Step 3: Add Vigilmon HTTP Uptime Monitoring

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your health proxy URL: http://your-edge-node:9200/health.
  4. Set Check interval to 2 minutes.
  5. Set Expected HTTP status to 200.
  6. Under Advanced, add a Response body contains check: "status":"ok".
  7. Click Save.

Vigilmon now polls YARN state every 2 minutes and alerts you if the Giraph job enters FAILED or KILLED state.

Step 4: Add a Superstep Heartbeat Monitor

The heartbeat monitor proves superstep-level progress, not just job existence.

  1. Click Add Monitor → Cron / Heartbeat.
  2. Set Expected ping interval based on your typical superstep duration (e.g., 5 minutes for large graphs).
  3. Set Grace period to 2× the superstep duration to account for stragglers.
  4. Copy the generated heartbeat URL into VIGILMON_HEARTBEAT_URL.
  5. Click Save.

If a superstep stalls at the BSP barrier — one slow worker blocking all others — the heartbeat stops and Vigilmon alerts you before the job times out at the YARN level.

Step 5: Monitor ZooKeeper Coordination

Giraph uses ZooKeeper for worker coordination and master election. A ZooKeeper outage orphans running jobs:

# scripts/zookeeper_health.sh
# Simple ZooKeeper four-letter-word check exposed as a health endpoint

#!/bin/bash
ZK_HOST="${ZK_HOST:-localhost}"
ZK_PORT="${ZK_PORT:-2181}"

RUOK=$(echo "ruok" | nc -w 2 "$ZK_HOST" "$ZK_PORT" 2>/dev/null)

if [ "$RUOK" = "imok" ]; then
    echo '{"status":"ok","zookeeper":"imok"}'
    exit 0
else
    echo '{"status":"error","zookeeper":"unreachable"}'
    exit 1
fi

Serve it behind a lightweight HTTP wrapper:

# scripts/zk_health_server.py
from flask import Flask, jsonify
import subprocess

app = Flask(__name__)

@app.route('/health/zookeeper')
def zk_health():
    result = subprocess.run(['bash', 'zookeeper_health.sh'],
                            capture_output=True, text=True, timeout=5)
    if result.returncode == 0:
        return jsonify(status='ok')
    return jsonify(status='error', detail=result.stdout), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=9201)

Add a third Vigilmon monitor pointing to http://your-edge-node:9201/health/zookeeper.

Step 6: Set Up Alerts

In Vigilmon, configure alert channels for your Giraph monitors:

  1. Go to Alert Channels → Add Channel.
  2. Choose Slack, Email, or PagerDuty.
  3. Set alert thresholds: notify after 1 failure for job health (failed jobs need immediate attention).
  4. For ZooKeeper, use 2 consecutive failures (a brief blip shouldn't wake the on-call team).
  5. Assign channels to all three monitors.

For batch pipeline orchestration, create a Status Page showing the health of the entire graph processing stack:

  1. Go to Status Pages → Create Page.
  2. Add YARN job health, superstep heartbeat, and ZooKeeper monitors.
  3. Share with the data engineering team.

Key Metrics to Watch

| Metric | Vigilmon feature | What to alert on | |---|---|---| | Job state | HTTP monitor on /health | Any non-200 (FAILED/KILLED state) | | Superstep progress | Heartbeat monitor | Missing heartbeat > 2× superstep interval | | ZooKeeper health | HTTP monitor on /health/zookeeper | Any 5xx or timeout | | YARN ResourceManager | HTTP monitor on RM REST API | Any 5xx or timeout | | Job wall-clock SLA | Heartbeat + grace period | Alert when job completion heartbeat is missing |

Conclusion

Giraph's BSP model is powerful but opaque — a stalled superstep looks identical to a running one from the YARN web UI. By emitting a heartbeat from MasterCompute after each superstep completes, you turn BSP progress into a concrete, monitorable signal. Add ZooKeeper health checks and YARN application state monitoring, and you have full visibility into your graph processing infrastructure — catching stalls, failures, and SLA breaches before they affect downstream consumers.

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles