tutorial

How to Monitor Apache Heron Health and Topology Availability with Vigilmon

Heron topology failures and tmaster crashes silently stall your real-time analytics pipelines. Learn how to monitor Heron tmaster health, topology liveness, and client application health with Vigilmon HTTP probes and heartbeat monitors.

Apache Heron is Twitter's battle-tested distributed real-time analytics system — the successor to Apache Storm — designed for higher throughput, lower latency, and simpler debugging at scale. But when the Topology Master (tmaster) crashes, a scheduler loses contact with its workers, or a topology's spouts fall behind, your event processing pipeline stalls silently. Heron's topology state transitions don't surface as obvious errors; they appear as missing downstream records, growing consumer lag, or stale aggregates that look valid but are hours behind.

Vigilmon gives you external visibility into Heron cluster health through HTTP probe monitoring (via Heron's built-in tracker REST API or a custom health sidecar) and heartbeat monitoring for your Heron topology client applications. This tutorial covers both.

Why Heron Needs External Monitoring

Heron's built-in tooling (Heron UI on port 8889, Tracker REST API on port 8888, metrics API) provides rich internal metrics — but only if someone is watching. External monitoring with Vigilmon adds:

  • Proactive alerting when the Heron Tracker API becomes unreachable (early sign of tmaster failure or cluster partition)
  • Topology liveness detection via the Tracker REST API that checks active topology status and component health
  • Heartbeat monitoring so you know immediately when a client application or spout stops successfully processing events
  • Multi-region availability checking from outside your Heron cluster network

These layers work together: the Tracker API can report a topology as RUNNING while the tmaster is restarting, and a topology can appear active while individual bolt threads are deadlocked on downstream I/O.

Step 1: Build a Heron Health Endpoint

Heron exposes a Tracker REST API (default port 8888) and a UI on port 8889. Use them directly or wrap in a custom health sidecar for richer checks.

Using the Heron Tracker REST API Directly

# Check all topologies in the default cluster
curl -i http://heron-tracker.example.com:8888/topologies

# Check a specific topology's runtime state
curl -i "http://heron-tracker.example.com:8888/topologies/runtime?cluster=default&environ=prod&topology=my-topology"

# Check tmaster location for a topology
curl -i "http://heron-tracker.example.com:8888/topologies/logicalplan?cluster=default&environ=prod&topology=my-topology"

Point a Vigilmon monitor directly at http://heron-tracker.example.com:8888/topologies for a quick reachability check.

Custom Python Health Sidecar

For deeper health checks — topology status, stmgr connectivity, tmaster liveness — build a lightweight sidecar that aggregates Heron Tracker responses:

# heron_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

TRACKER_URL = os.environ.get('HERON_TRACKER_URL', 'http://localhost:8888')
CLUSTER = os.environ.get('HERON_CLUSTER', 'default')
ENVIRON = os.environ.get('HERON_ENVIRON', 'prod')
REQUIRED_TOPOLOGY = os.environ.get('HERON_TOPOLOGY_NAME', 'production-pipeline')

@app.route('/health/heron')
def heron_health():
    try:
        # Check tracker is reachable and list topologies
        resp = requests.get(
            f'{TRACKER_URL}/topologies',
            params={'cluster': CLUSTER, 'environ': ENVIRON},
            timeout=5
        )
        resp.raise_for_status()
        data = resp.json()

        topologies = [
            t.get('name') for t in data.get('result', [])
        ]

        if REQUIRED_TOPOLOGY not in topologies:
            return jsonify({
                'status': 'down',
                'reason': f'topology_{REQUIRED_TOPOLOGY}_not_found',
            }), 503

        # Check topology runtime state
        runtime_resp = requests.get(
            f'{TRACKER_URL}/topologies/runtime',
            params={
                'cluster': CLUSTER,
                'environ': ENVIRON,
                'topology': REQUIRED_TOPOLOGY,
            },
            timeout=5
        )
        runtime_resp.raise_for_status()
        runtime = runtime_resp.json().get('result', {})

        stmgrs = runtime.get('stmgrs', {})
        total = len(stmgrs)
        alive = sum(1 for s in stmgrs.values() if s.get('is_alive', False))

        if alive == 0:
            return jsonify({
                'status': 'down',
                'reason': 'no_alive_stmgrs',
                'stmgrs_total': total,
                'stmgrs_alive': alive,
            }), 503

        if alive < total:
            return jsonify({
                'status': 'degraded',
                'stmgrs_total': total,
                'stmgrs_alive': alive,
            }), 503

        return jsonify({
            'status': 'ok',
            'topology': REQUIRED_TOPOLOGY,
            'stmgrs_total': total,
            'stmgrs_alive': alive,
        }), 200

    except requests.RequestException as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8092)

Java Health Sidecar (Heron Client)

// HeronHealthController.java
@RestController
public class HeronHealthController {

    @Value("${heron.tracker.url:http://localhost:8888}")
    private String trackerUrl;

    @Value("${heron.cluster:default}")
    private String cluster;

    @Value("${heron.environ:prod}")
    private String environ;

    @Value("${heron.topology.name}")
    private String topologyName;

    private final RestTemplate restTemplate = new RestTemplate();

    @GetMapping("/health/heron")
    public ResponseEntity<Map<String, Object>> checkHealth() {
        Map<String, Object> status = new LinkedHashMap<>();
        try {
            String listUrl = trackerUrl + "/topologies?cluster=" + cluster + "&environ=" + environ;
            ResponseEntity<Map> listResp = restTemplate.getForEntity(listUrl, Map.class);

            List<Map<String, Object>> result =
                (List<Map<String, Object>>) listResp.getBody().get("result");

            boolean found = result.stream()
                .anyMatch(t -> topologyName.equals(t.get("name")));

            if (!found) {
                status.put("status", "down");
                status.put("reason", "topology_not_found");
                return ResponseEntity.status(503).body(status);
            }

            String runtimeUrl = trackerUrl + "/topologies/runtime?cluster=" + cluster
                + "&environ=" + environ + "&topology=" + topologyName;
            ResponseEntity<Map> runtimeResp = restTemplate.getForEntity(runtimeUrl, Map.class);
            Map<String, Object> runtimeResult =
                (Map<String, Object>) runtimeResp.getBody().get("result");
            Map<String, Object> stmgrs =
                (Map<String, Object>) runtimeResult.get("stmgrs");

            long alive = stmgrs.values().stream()
                .filter(s -> Boolean.TRUE.equals(((Map<?, ?>) s).get("is_alive")))
                .count();

            if (alive == 0) {
                status.put("status", "down");
                status.put("reason", "no_alive_stmgrs");
                return ResponseEntity.status(503).body(status);
            }

            status.put("status", "ok");
            status.put("topology", topologyName);
            status.put("stmgrs_alive", alive);
            status.put("stmgrs_total", stmgrs.size());
            return ResponseEntity.ok(status);

        } catch (Exception e) {
            status.put("status", "down");
            status.put("error", e.getMessage());
            return ResponseEntity.status(503).body(status);
        }
    }
}

Step 2: Configure Vigilmon HTTP Monitor for Heron

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Heron health endpoint: https://your-app.example.com/health/heron
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms (Tracker API calls aggregate tmaster and stmgr lookups)
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the Heron Tracker API directly:

  • URL: http://heron-tracker.example.com:8888/topologies
  • Expected: 200
  • Interval: 2 minutes
  • Alert channel: on-call channel

Vigilmon's multi-region probe consensus prevents false positives from transient network blips between probe nodes and your Heron cluster.

Step 3: Heartbeat Monitoring for Heron Topology Clients

Tracker API and topology health checks are necessary — but not sufficient. A Heron spout can be stuck in retry loops for failed tuples, a bolt can be processing tuples and acking them without producing correct output, or a topology can appear healthy while processing at a fraction of normal throughput. Vigilmon heartbeat monitors detect these silent processing stalls: your topology pings Vigilmon after successfully processing each batch of tuples. If pings stop, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: heron-topology-spout
  3. Set the expected interval: 5 minutes (adjust to your tuple throughput)
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Heron Spout (Java)

// MonitoredHeronSpout.java
import org.apache.heron.api.spout.BaseRichSpout;
import org.apache.heron.api.spout.SpoutOutputCollector;
import org.apache.heron.api.topology.TopologyContext;
import org.apache.heron.api.tuple.Values;
import java.net.http.*;
import java.net.URI;
import java.util.Map;

public class MonitoredHeronSpout extends BaseRichSpout {

    private SpoutOutputCollector collector;
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final HttpClient httpClient = HttpClient.newHttpClient();
    private long tuplesEmitted = 0;
    private static final int HEARTBEAT_EVERY = 1000;

    @Override
    public void open(Map<String, Object> conf, TopologyContext context,
                     SpoutOutputCollector collector) {
        this.collector = collector;
    }

    @Override
    public void nextTuple() {
        String event = fetchNextEvent();
        if (event != null) {
            collector.emit(new Values(event));
            tuplesEmitted++;

            if (tuplesEmitted % HEARTBEAT_EVERY == 0) {
                pingVigilmon();
            }
        }
    }

    private void pingVigilmon() {
        try {
            HttpRequest req = HttpRequest.newBuilder()
                .uri(URI.create(heartbeatUrl))
                .GET().build();
            httpClient.sendAsync(req, HttpResponse.BodyHandlers.discarding());
        } catch (Exception e) {
            // Non-fatal: don't interrupt tuple emission
        }
    }

    private String fetchNextEvent() {
        // Your event source logic here
        return null;
    }

    @Override
    public void declareOutputFields(
            org.apache.heron.api.topology.OutputFieldsDeclarer declarer) {
        declarer.declare(new org.apache.heron.api.tuple.Fields("event"));
    }
}

Python Heron Bolt with Heartbeat (heronpy)

# monitored_bolt.py
import heronpy.api.bolt.bolt as bolt
import requests
import os
import time

class MonitoredProcessingBolt(bolt.Bolt):

    HEARTBEAT_INTERVAL = 60  # seconds

    def initialize(self, config, context):
        self.heartbeat_url = os.environ['VIGILMON_HEARTBEAT_URL']
        self.last_heartbeat = time.time()

    def process(self, tup):
        result = process_event(tup.values[0])
        self.emit([result])
        self.ack(tup)

        if time.time() - self.last_heartbeat > self.HEARTBEAT_INTERVAL:
            try:
                requests.get(self.heartbeat_url, timeout=5)
            except Exception:
                pass
            self.last_heartbeat = time.time()

Step 4: Alert Routing for Heron Failures

Heron failures cascade: a tmaster crash causes stmgrs to lose coordination, which causes containers to restart, which causes bolt and spout instances to become unavailable. Alert routing should reflect this cascade order.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Heron health /health/heron | Slack + PagerDuty | P1 | | Tracker API /topologies | Slack | P2 | | Heartbeat: production topology spout | Slack + email | P2 | | Heartbeat: analytics bolt output sink | Email | P3 |

Set response time thresholds for early warning:

  • Alert at 3000ms for the health endpoint (slow tracker responses signal tmaster pressure)
  • Alert at 8000ms for the tracker endpoint (slow aggregation signals stmgr communication delay)

For event-critical topologies, enable two consecutive failures before alerting — Heron's container restarts during autoscaling events can briefly cause the tracker to report degraded stmgr status without representing a full outage.

Summary

Heron failures are silent at the data layer. External monitoring catches tmaster crashes, topology stalls, and spout processing halts before they become event pipeline outages:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/heron | Stmgr liveness, topology presence, alive container count | | HTTP monitor on Tracker API | Tracker availability, topology list reachability | | Heartbeat monitor | Spout/bolt liveness, tuple processing throughput |

Get started free at vigilmon.online — your first Heron monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles