tutorial

How to Monitor Apache Storm Health and Topology Availability with Vigilmon

Storm topology failures and nimbus crashes silently stall your real-time stream processing pipelines. Learn how to monitor Storm nimbus health, topology liveness, and client application health with Vigilmon HTTP probes and heartbeat monitors.

Apache Storm is the real-time distributed stream processing engine powering low-latency event pipelines at massive scale — but when the nimbus crashes, a supervisor loses its connection to ZooKeeper, or a topology's spouts stop emitting, your event processing pipeline silently halts. Unlike batch systems, Storm failures don't produce error logs at the point of data loss; they surface as missing downstream records, increasing consumer lag, or stale aggregates that look correct but are hours out of date.

Vigilmon gives you external visibility into Storm cluster health through HTTP probe monitoring (via Storm UI API or a custom health sidecar) and heartbeat monitoring for your Storm topology client applications. This tutorial covers both.

Why Storm Needs External Monitoring

Storm's built-in tooling (Storm UI on port 8080, metrics API, ZooKeeper state) provides rich internal metrics — but only if someone is watching. External monitoring with Vigilmon adds:

  • Proactive alerting when the Storm UI API becomes unreachable (early sign of nimbus failure or network partition)
  • Topology liveness detection via the REST API that checks active topology status and component health
  • Heartbeat monitoring so you know immediately when a client application or spout stops successfully processing events
  • Multi-region availability checking from outside your Storm cluster network

These layers work together: the Storm UI can be accessible while a topology is stuck in a REBALANCING state, and a topology can appear active while individual bolt threads are deadlocked on downstream I/O.

Step 1: Build a Storm Health Endpoint

Storm exposes a REST API on the Storm UI server (default port 8080). Use it directly or wrap it in a custom health sidecar for richer checks.

Using the Storm UI REST API Directly

# Check nimbus summary — returns nimbus host, uptime, and status
curl -i http://storm-ui.example.com:8080/api/v1/nimbus/summary

# List all active topologies
curl -i http://storm-ui.example.com:8080/api/v1/topology/summary

# Check a specific topology's status
curl -i http://storm-ui.example.com:8080/api/v1/topology/my-topology-id

Point a Vigilmon monitor directly at http://storm-ui.example.com:8080/api/v1/nimbus/summary for a quick reachability check.

Custom Python Health Sidecar

For deeper health checks — topology status, bolt executor health, ZooKeeper connectivity — build a lightweight sidecar that aggregates Storm REST API responses:

# storm_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

STORM_UI_URL = os.environ.get('STORM_UI_URL', 'http://localhost:8080')
REQUIRED_TOPOLOGY = os.environ.get('STORM_TOPOLOGY_NAME', 'production-pipeline')

@app.route('/health/storm')
def storm_health():
    try:
        # Check nimbus is alive
        nimbus = requests.get(
            f'{STORM_UI_URL}/api/v1/nimbus/summary', timeout=5
        ).json()

        if not nimbus.get('nimbuses'):
            return jsonify({'status': 'down', 'reason': 'no_nimbus'}), 503

        # Check required topology is ACTIVE
        topologies = requests.get(
            f'{STORM_UI_URL}/api/v1/topology/summary', timeout=5
        ).json()

        target = next(
            (t for t in topologies.get('topologies', [])
             if t.get('name') == REQUIRED_TOPOLOGY),
            None
        )

        if target is None:
            return jsonify({'status': 'down',
                            'reason': f'topology_{REQUIRED_TOPOLOGY}_not_found'}), 503

        if target.get('status') != 'ACTIVE':
            return jsonify({'status': 'degraded',
                            'topology_status': target.get('status')}), 503

        return jsonify({
            'status': 'ok',
            'topology': REQUIRED_TOPOLOGY,
            'topology_status': target.get('status'),
            'workers': target.get('workersTotal'),
            'executors': target.get('executorsTotal'),
        }), 200

    except requests.RequestException as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8092)

Java Health Sidecar (Storm Client)

// StormHealthController.java
@RestController
public class StormHealthController {

    @Value("${storm.ui.url:http://localhost:8080}")
    private String stormUiUrl;

    @Value("${storm.topology.name}")
    private String topologyName;

    private final RestTemplate restTemplate = new RestTemplate();

    @GetMapping("/health/storm")
    public ResponseEntity<Map<String, Object>> checkHealth() {
        Map<String, Object> status = new LinkedHashMap<>();
        try {
            // Verify nimbus is reachable
            ResponseEntity<Map> nimbus = restTemplate.getForEntity(
                stormUiUrl + "/api/v1/nimbus/summary", Map.class);

            if (!nimbus.getStatusCode().is2xxSuccessful()) {
                status.put("status", "down");
                status.put("reason", "nimbus_unreachable");
                return ResponseEntity.status(503).body(status);
            }

            // Check topology status
            ResponseEntity<Map> topoSummary = restTemplate.getForEntity(
                stormUiUrl + "/api/v1/topology/summary", Map.class);

            List<Map<String, Object>> topologies =
                (List<Map<String, Object>>) topoSummary.getBody().get("topologies");

            Map<String, Object> target = topologies.stream()
                .filter(t -> topologyName.equals(t.get("name")))
                .findFirst().orElse(null);

            if (target == null) {
                status.put("status", "down");
                status.put("reason", "topology_not_found");
                return ResponseEntity.status(503).body(status);
            }

            String topoStatus = (String) target.get("status");
            if (!"ACTIVE".equals(topoStatus)) {
                status.put("status", "degraded");
                status.put("topology_status", topoStatus);
                return ResponseEntity.status(503).body(status);
            }

            status.put("status", "ok");
            status.put("topology_status", topoStatus);
            status.put("workers", target.get("workersTotal"));
            return ResponseEntity.ok(status);

        } catch (Exception e) {
            status.put("status", "down");
            status.put("error", e.getMessage());
            return ResponseEntity.status(503).body(status);
        }
    }
}

Step 2: Configure Vigilmon HTTP Monitor for Storm

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Storm health endpoint: https://your-app.example.com/health/storm
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms (Storm UI REST calls aggregate multiple ZooKeeper lookups)
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the Storm UI API directly:

  • URL: http://storm-ui.example.com:8080/api/v1/nimbus/summary
  • Expected: 200
  • Interval: 2 minutes
  • Alert channel: on-call channel

Vigilmon's multi-region probe consensus prevents false positives from transient network blips between probe nodes and your Storm cluster.

Step 3: Heartbeat Monitoring for Storm Topology Clients

Nimbus and topology health checks are necessary — but not sufficient. A Storm spout can be stuck in a retry loop for failed tuples, a bolt can be processing tuples incorrectly and acking them as success, or a topology can be alive but processing at 1% of normal throughput, while the cluster metrics look healthy.

Vigilmon heartbeat monitors detect silent processing stalls: your topology pings Vigilmon after successfully processing each batch of tuples. If pings stop, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: storm-topology-spout
  3. Set the expected interval: 5 minutes (adjust to your tuple throughput)
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Storm Spout

// MonitoredSpout.java
import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.IRichSpout;
import org.apache.storm.topology.OutputFieldsDeclarer;
import java.net.http.*;
import java.net.URI;
import java.util.Map;

public class MonitoredSpout implements IRichSpout {

    private SpoutOutputCollector collector;
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final HttpClient httpClient = HttpClient.newHttpClient();
    private long tuplesEmitted = 0;
    private static final int HEARTBEAT_EVERY = 1000;

    @Override
    public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
        this.collector = collector;
    }

    @Override
    public void nextTuple() {
        String event = fetchNextEvent();
        if (event != null) {
            collector.emit(new Values(event));
            tuplesEmitted++;

            if (tuplesEmitted % HEARTBEAT_EVERY == 0) {
                pingVigilmon();
            }
        }
    }

    private void pingVigilmon() {
        try {
            HttpRequest req = HttpRequest.newBuilder()
                .uri(URI.create(heartbeatUrl))
                .GET().build();
            httpClient.sendAsync(req, HttpResponse.BodyHandlers.discarding());
        } catch (Exception e) {
            // Non-fatal: don't let heartbeat failure interrupt tuple emission
        }
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("event"));
    }
}

Python Storm Bolt with Heartbeat (streamparse)

# monitored_bolt.py
from streamparse.bolt import Bolt
import requests
import os
import time

class MonitoredProcessingBolt(Bolt):

    HEARTBEAT_INTERVAL = 60  # seconds

    def initialize(self, conf, ctx):
        self.heartbeat_url = os.environ['VIGILMON_HEARTBEAT_URL']
        self.last_heartbeat = time.time()
        self.tuples_processed = 0

    def process(self, tup):
        # Process the tuple
        result = process_event(tup.values[0])
        self.emit([result])
        self.ack(tup)

        self.tuples_processed += 1
        if time.time() - self.last_heartbeat > self.HEARTBEAT_INTERVAL:
            try:
                requests.get(self.heartbeat_url, timeout=5)
            except Exception:
                pass
            self.last_heartbeat = time.time()

Step 4: Alert Routing for Storm Failures

Storm failures cascade: a ZooKeeper session expiry causes the nimbus to disconnect, which causes supervisors to stop accepting new assignments, which causes worker processes to exit, which causes topology bolts and spouts to become unreachable. Alert routing should reflect this cascade order.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Storm health /health/storm | Slack + PagerDuty | P1 | | Storm UI API /api/v1/nimbus/summary | Slack | P2 | | Heartbeat: production topology spout | Slack + email | P2 | | Heartbeat: analytics bolt output sink | Email | P3 |

Set response time thresholds for early warning:

  • Alert at 3000ms for the health endpoint (slow UI responses signal ZooKeeper quorum pressure)
  • Alert at 8000ms for the nimbus summary endpoint (slow aggregation signals worker communication delay)

For event-critical topologies, enable two consecutive failures before alerting — Storm's automatic rebalancing after supervisor restarts can briefly cause the UI to report degraded status without representing a full outage.

Summary

Storm failures are silent at the data layer. External monitoring catches nimbus crashes, topology stalls, and spout processing halts before they become event pipeline outages:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/storm | Nimbus liveness, topology status, worker and executor count | | HTTP monitor on Storm UI API | UI availability, nimbus reachability | | Heartbeat monitor | Spout/bolt liveness, tuple processing throughput |

Get started free at vigilmon.online — your first Storm monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles