Apache Druid is built for sub-second queries over billions of events — but its distributed multi-service architecture means there are a lot of moving parts that can fail silently. A Historical node dropping out shrinks your query capacity without any error to your users. A stalled Overlord task queue backs up ingestion while dashboards keep serving stale data. A Coordinator that stops balancing segments fragments your cluster health across nodes.
Vigilmon gives you external visibility into each Druid service tier through HTTP probe monitoring and heartbeat monitors for your ingestion pipelines. This tutorial covers the full stack: Historical nodes, Broker query latency, Coordinator segment assignment, Overlord task health, and real-time ingestion lag.
Why Druid Needs External Monitoring
Druid's web console and built-in JMX metrics give you rich internal telemetry — but only for what's currently reachable. External monitoring with Vigilmon adds:
- Service liveness probes that detect when a Historical, Broker, Coordinator, or Overlord process dies or stops responding
- Ingestion lag alerting via health sidecars that expose supervisor lag as an HTTP endpoint
- Heartbeat monitoring so you know immediately when a real-time ingestion supervisor stalls — even if the Druid process itself is alive
- Multi-region availability checking that confirms your Broker query layer is reachable from outside your internal network
Internal dashboards can't catch the failure mode where Druid's process is up but not serving queries — a corrupted segment cache, an OOM-killed JVM, or a full GC pause. Vigilmon catches these from the outside.
Step 1: Build Druid Service Health Endpoints
Druid's built-in status API is your first probe target. Each service exposes /status/health on its HTTP port.
Native Druid Health Endpoints
# Coordinator (default port 8081)
curl -i http://druid-coordinator:8081/status/health
# Broker (default port 8082)
curl -i http://druid-broker:8082/status/health
# Historical (default port 8083)
curl -i http://druid-historical:8083/status/health
# Overlord (default port 8090)
curl -i http://druid-overlord:8090/status/health
# Middle Manager (default port 8091)
curl -i http://druid-middlemanager:8091/status/health
# Router (default port 8888)
curl -i http://druid-router:8888/status/health
Each returns HTTP 200 with {"healthy": true} when the process is healthy. Vigilmon can probe these directly — no sidecar needed for basic liveness.
Broker Query Latency Health Endpoint
The Broker's /status/health only tells you the process is alive. For query latency monitoring, add a lightweight sidecar that issues a test query and measures response time:
# druid_health.py
from flask import Flask, jsonify
import requests
import time
import os
app = Flask(__name__)
BROKER_URL = os.environ.get('DRUID_BROKER_URL', 'http://localhost:8082')
TEST_DATASOURCE = os.environ.get('DRUID_TEST_DATASOURCE', 'health_check')
LATENCY_THRESHOLD_MS = int(os.environ.get('DRUID_LATENCY_THRESHOLD_MS', '2000'))
@app.route('/health/druid/broker')
def broker_health():
start = time.time()
try:
resp = requests.post(
f'{BROKER_URL}/druid/v2/sql',
json={
'query': f"SELECT COUNT(*) FROM \"{TEST_DATASOURCE}\" LIMIT 1",
'context': {'timeout': 5000}
},
timeout=6
)
elapsed_ms = (time.time() - start) * 1000
if resp.status_code != 200:
return jsonify({'status': 'down', 'error': f'HTTP {resp.status_code}'}), 503
if elapsed_ms > LATENCY_THRESHOLD_MS:
return jsonify({
'status': 'degraded',
'reason': 'high_query_latency',
'latency_ms': round(elapsed_ms)
}), 503
return jsonify({'status': 'ok', 'latency_ms': round(elapsed_ms)}), 200
except requests.exceptions.Timeout:
return jsonify({'status': 'down', 'error': 'query_timeout'}), 503
except Exception as e:
return jsonify({'status': 'down', 'error': str(e)}), 503
if __name__ == '__main__':
app.run(port=9090)
Coordinator Segment Assignment Health
The Coordinator manages segment replication and balance. Query its segment loading status:
@app.route('/health/druid/coordinator')
def coordinator_health():
try:
resp = requests.get(
f'{os.environ["DRUID_COORDINATOR_URL"]}/druid/coordinator/v1/loadstatus',
timeout=5
)
if resp.status_code != 200:
return jsonify({'status': 'down', 'error': f'HTTP {resp.status_code}'}), 503
load_status = resp.json()
# loadstatus returns datasource -> percentage loaded (0-100)
under_loaded = {
ds: pct for ds, pct in load_status.items() if pct < 100
}
if under_loaded:
return jsonify({
'status': 'degraded',
'reason': 'segments_not_fully_loaded',
'datasources': under_loaded
}), 503
return jsonify({'status': 'ok', 'datasources_loaded': len(load_status)}), 200
except Exception as e:
return jsonify({'status': 'down', 'error': str(e)}), 503
Overlord Task Health and Ingestion Lag
The Overlord manages ingestion tasks. Check for failed tasks and supervisor lag:
@app.route('/health/druid/ingestion')
def ingestion_health():
overlord_url = os.environ['DRUID_OVERLORD_URL']
try:
# Check for running supervisors
supervisors = requests.get(
f'{overlord_url}/druid/indexer/v1/supervisor',
timeout=5
).json()
issues = []
for supervisor_id in supervisors:
status = requests.get(
f'{overlord_url}/druid/indexer/v1/supervisor/{supervisor_id}/status',
timeout=5
).json()
# Check supervisor state
state = status.get('payload', {}).get('state', '')
if state not in ('RUNNING', 'IDLE'):
issues.append({'supervisor': supervisor_id, 'state': state})
# Check ingestion lag
lag = status.get('payload', {}).get('aggregateLag', 0)
lag_threshold = int(os.environ.get('DRUID_LAG_THRESHOLD', '50000'))
if lag > lag_threshold:
issues.append({
'supervisor': supervisor_id,
'reason': 'high_lag',
'lag': lag
})
if issues:
return jsonify({'status': 'degraded', 'issues': issues}), 503
return jsonify({'status': 'ok', 'supervisors': len(supervisors)}), 200
except Exception as e:
return jsonify({'status': 'down', 'error': str(e)}), 503
Step 2: Configure Vigilmon HTTP Monitors for Druid
- Log in to vigilmon.online and go to Monitors → New Monitor
- Choose HTTP / HTTPS
- Add one monitor per service tier:
| Monitor Name | URL | Expected Status | Interval |
|---|---|---|---|
| Druid Broker liveness | http://druid-broker:8082/status/health | 200, body "healthy":true | 1 min |
| Druid Broker query latency | https://your-sidecar/health/druid/broker | 200, body "status":"ok" | 1 min |
| Druid Coordinator | http://druid-coordinator:8081/status/health | 200 | 1 min |
| Druid Coordinator segments | https://your-sidecar/health/druid/coordinator | 200 | 5 min |
| Druid Historical | http://druid-historical:8083/status/health | 200 | 1 min |
| Druid Overlord | http://druid-overlord:8090/status/health | 200 | 1 min |
| Druid ingestion lag | https://your-sidecar/health/druid/ingestion | 200 | 2 min |
| Druid Router (external) | https://your-druid.example.com/status/health | 200 | 1 min |
For the Broker query latency monitor, set:
- Response time threshold:
3000ms(alert before the 5s timeout threshold) - Body must contain:
"status":"ok"
For the coordinator segment monitor, a 5-minute interval is sufficient — segment rebalancing happens on that cadence.
Step 3: Heartbeat Monitoring for Druid Ingestion Pipelines
A Druid supervisor appearing "RUNNING" doesn't mean data is actually flowing. The supervisor state machine can stall on a Kafka partition rebalance or a transient ZooKeeper hiccup while reporting healthy.
Vigilmon heartbeat monitors detect ingestion stalls at the application level: your ingestion pipeline sends a ping after each successful batch of events is confirmed committed. If pings stop, Vigilmon fires an alert.
Set Up the Heartbeat Monitor
- In Vigilmon, go to Monitors → New Monitor → Heartbeat
- Set the name:
druid-kafka-ingestion-pipeline - Set the expected interval: 5 minutes (adjust to your ingestion throughput)
- Set the grace period: 10 minutes
- Save — copy the heartbeat URL, e.g.
https://vigilmon.online/heartbeat/abc123xyz
Kafka Producer Ingestion Pipeline (Python)
# druid_ingestion.py
import os
import requests
import time
from confluent_kafka import Consumer
KAFKA_BROKERS = os.environ['KAFKA_BROKERS']
KAFKA_TOPIC = os.environ['KAFKA_TOPIC']
DRUID_TRANQUILITY_URL = os.environ.get('DRUID_TRANQUILITY_URL')
VIGILMON_HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']
HEARTBEAT_INTERVAL = int(os.environ.get('HEARTBEAT_INTERVAL_SECONDS', '300'))
consumer = Consumer({
'bootstrap.servers': KAFKA_BROKERS,
'group.id': 'druid-ingestion-monitor',
'auto.offset.reset': 'latest',
})
consumer.subscribe([KAFKA_TOPIC])
last_heartbeat = time.time()
events_since_heartbeat = 0
try:
while True:
msg = consumer.poll(timeout=1.0)
if msg and not msg.error():
process_and_send_to_druid(msg)
events_since_heartbeat += 1
# Ping Vigilmon every HEARTBEAT_INTERVAL seconds of active processing
now = time.time()
if now - last_heartbeat >= HEARTBEAT_INTERVAL and events_since_heartbeat > 0:
requests.get(VIGILMON_HEARTBEAT_URL, timeout=5)
last_heartbeat = now
events_since_heartbeat = 0
finally:
consumer.close()
Batch Ingestion Job Wrapper
For scheduled batch ingestion tasks (Hadoop, native batch), wrap the job submission:
#!/bin/bash
# submit_druid_task.sh
TASK_FILE=$1
OVERLORD_URL=${DRUID_OVERLORD_URL:-http://localhost:8090}
VIGILMON_URL=${VIGILMON_HEARTBEAT_URL}
# Submit task
TASK_ID=$(curl -s -X POST \
-H 'Content-Type: application/json' \
-d @"$TASK_FILE" \
"${OVERLORD_URL}/druid/indexer/v1/task" | jq -r '.task')
echo "Submitted task: $TASK_ID"
# Poll for completion
while true; do
STATUS=$(curl -s "${OVERLORD_URL}/druid/indexer/v1/task/${TASK_ID}/status" | jq -r '.status.status')
echo "Task status: $STATUS"
if [ "$STATUS" = "SUCCESS" ]; then
# Ping heartbeat on successful completion
curl -s "$VIGILMON_URL" > /dev/null
echo "Task succeeded, heartbeat sent"
exit 0
elif [ "$STATUS" = "FAILED" ]; then
echo "Task failed — no heartbeat sent, alert will fire"
exit 1
fi
sleep 30
done
Step 4: Alert Routing for Druid Service Failures
Druid failures cascade: a Historical node failure redistributes segment load to remaining nodes, increasing their query load, slowing Broker response times, and eventually causing query timeouts visible to end users.
Configure alert routing in Vigilmon so broker failures page on-call immediately:
| Monitor | Alert Channel | Priority | |---|---|---| | Druid Broker liveness | Slack + PagerDuty | P1 | | Druid Broker query latency | Slack + PagerDuty | P1 | | Druid Historical | Slack | P2 | | Druid Coordinator segments | Slack | P2 | | Druid Overlord | Slack | P2 | | Druid ingestion lag | Slack + email | P2 | | Druid batch ingestion heartbeat | Email | P3 |
Set a response time alert on the Router and Broker monitors at 2000ms — a degraded Historical node cluster shows up as elevated Broker latency before queries start failing outright.
Summary
Druid's distributed architecture means no single health signal tells the full story. You need monitoring at every service tier to catch failures before they cascade to query latency and stale data:
| Monitor Type | What It Covers |
|---|---|
| HTTP probe on /status/health | Service process liveness per tier |
| HTTP probe on broker query sidecar | End-to-end query latency |
| HTTP probe on coordinator load status | Segment replication health |
| HTTP probe on ingestion sidecar | Supervisor state and ingestion lag |
| Heartbeat monitor | Real-time ingestion pipeline liveness |
Get started free at vigilmon.online — your first Druid monitor is running in under two minutes.