tutorial

How to Monitor Apache Pinot with Vigilmon

Apache Pinot powers real-time OLAP at companies like LinkedIn and Uber — but Controller failures, Broker query degradation, and Server segment load issues are silent without external monitoring. Learn how to monitor every Pinot tier with Vigilmon.

Apache Pinot is designed for user-facing analytics at scale — think LinkedIn's "Who Viewed Your Profile" or Uber's real-time operational dashboards. Its real-time ingestion and sub-second query serving make it uniquely powerful, but also uniquely risky to operate: a Controller failure orphans segment assignment, a Server node going dark reduces query capacity silently, and ingestion lag from Kafka can serve stale data to users without any obvious error signal.

Vigilmon gives you external visibility into every Pinot service tier through HTTP probe monitoring and heartbeat monitors for your real-time ingestion. This tutorial covers Controller health, Broker query latency, Server segment load status, Minion task health, and real-time ingestion lag.

Why Pinot Needs External Monitoring

Pinot's web UI and JMX metrics provide rich internal telemetry — but only from within your network perimeter. External monitoring with Vigilmon adds:

  • Controller liveness probes that detect when cluster metadata and segment assignment become unavailable
  • Broker query latency tracking that catches degradation before users report slow dashboards
  • Server segment load validation that alerts when a node joins but hasn't yet loaded its assigned segments
  • Minion task failure detection for offline table maintenance jobs (purge, merge, backfill)
  • Real-time ingestion lag alerts via sidecar endpoints that expose Kafka consumer offset lag as HTTP health

When a Pinot Server goes OOM and gets restarted by Kubernetes, Pinot re-routes queries to surviving servers — but those servers now carry extra load and query latency climbs. Vigilmon catches this as elevated Broker response times before dashboards visibly degrade.

Step 1: Build Pinot Health Endpoints

Pinot exposes a built-in health endpoint on each component. Use these as your first probe targets.

Native Pinot Health Endpoints

# Controller (default port 9000)
curl -i http://pinot-controller:9000/health

# Broker (default port 8099)
curl -i http://pinot-broker:8099/health

# Server (default port 8097)
curl -i http://pinot-server:8097/health

# Minion (default port 9514)
curl -i http://pinot-minion:9514/health

Each returns OK with HTTP 200 when healthy. Configure one Vigilmon monitor per component pointing at these endpoints.

Broker Query Latency Sidecar

The Broker's /health endpoint only checks process liveness. Build a sidecar that runs a test query and reports latency:

// pinot-health-sidecar.js
const express = require('express');
const axios = require('axios');
const app = express();

const BROKER_URL = process.env.PINOT_BROKER_URL || 'http://localhost:8099';
const TEST_TABLE = process.env.PINOT_TEST_TABLE || 'health_check';
const LATENCY_THRESHOLD_MS = parseInt(process.env.PINOT_LATENCY_THRESHOLD_MS || '1500');

app.get('/health/pinot/broker', async (req, res) => {
  const start = Date.now();
  try {
    const response = await axios.post(
      `${BROKER_URL}/query/sql`,
      { sql: `SELECT COUNT(*) FROM ${TEST_TABLE} LIMIT 1` },
      { timeout: 5000 }
    );

    const latencyMs = Date.now() - start;

    if (response.status !== 200) {
      return res.status(503).json({ status: 'down', error: `HTTP ${response.status}` });
    }

    if (latencyMs > LATENCY_THRESHOLD_MS) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'high_query_latency',
        latency_ms: latencyMs,
        threshold_ms: LATENCY_THRESHOLD_MS,
      });
    }

    return res.status(200).json({ status: 'ok', latency_ms: latencyMs });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

Controller Cluster Health

The Controller manages cluster metadata, table configs, and segment assignment. Query its cluster state:

app.get('/health/pinot/controller', async (req, res) => {
  const controllerUrl = process.env.PINOT_CONTROLLER_URL || 'http://localhost:9000';
  try {
    // Check controller health
    const health = await axios.get(`${controllerUrl}/health`, { timeout: 5000 });
    if (health.data !== 'OK') {
      return res.status(503).json({ status: 'down', reason: 'controller_unhealthy' });
    }

    // Check for segments in ERROR state via cluster health endpoint
    const clusterHealth = await axios.get(
      `${controllerUrl}/health/controller`,
      { timeout: 5000 }
    );

    return res.status(200).json({
      status: 'ok',
      controller: clusterHealth.data
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

Server Segment Load Status

After a Server node restarts, it needs time to load its segments from deep storage. During this window it answers queries with partial data. Monitor segment load completeness:

app.get('/health/pinot/server/segments', async (req, res) => {
  const controllerUrl = process.env.PINOT_CONTROLLER_URL || 'http://localhost:9000';
  try {
    const segmentStatus = await axios.get(
      `${controllerUrl}/debug/timeBoundary`,
      { timeout: 10000 }
    );

    // A simpler check: verify the server health endpoint returns 200
    const serverUrl = process.env.PINOT_SERVER_URL || 'http://localhost:8097';
    const serverHealth = await axios.get(`${serverUrl}/health`, { timeout: 5000 });

    if (serverHealth.data !== 'OK') {
      return res.status(503).json({ status: 'degraded', reason: 'server_unhealthy' });
    }

    return res.status(200).json({ status: 'ok' });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

Real-Time Ingestion Lag (Kafka to Pinot)

For real-time tables consuming from Kafka, check the consumer lag via Pinot's consuming segment API:

app.get('/health/pinot/ingestion/lag', async (req, res) => {
  const controllerUrl = process.env.PINOT_CONTROLLER_URL || 'http://localhost:9000';
  const tableName = process.env.PINOT_REALTIME_TABLE;
  const lagThreshold = parseInt(process.env.PINOT_LAG_THRESHOLD || '10000');

  try {
    const consumingSegments = await axios.get(
      `${controllerUrl}/tables/${tableName}/consumingSegmentsInfo`,
      { timeout: 10000 }
    );

    const segments = consumingSegments.data.segmentToConsumingInfoMap || {};
    let maxLag = 0;
    const lagBySegment = {};

    for (const [segmentName, info] of Object.entries(segments)) {
      const lag = info.partitionOffsetInfo?.recordsLag || 0;
      lagBySegment[segmentName] = lag;
      if (lag > maxLag) maxLag = lag;
    }

    if (maxLag > lagThreshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'high_ingestion_lag',
        max_lag: maxLag,
        threshold: lagThreshold,
        lag_by_segment: lagBySegment,
      });
    }

    return res.status(200).json({ status: 'ok', max_lag: maxLag });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(9090, () => console.log('Pinot health sidecar on :9090'));

Step 2: Configure Vigilmon HTTP Monitors for Pinot

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add one monitor per tier:

| Monitor Name | URL | Expected | Interval | |---|---|---|---| | Pinot Controller liveness | http://pinot-controller:9000/health | 200, body OK | 1 min | | Pinot Controller cluster | https://your-sidecar/health/pinot/controller | 200, body "ok" | 2 min | | Pinot Broker liveness | http://pinot-broker:8099/health | 200, body OK | 1 min | | Pinot Broker query latency | https://your-sidecar/health/pinot/broker | 200, body "ok" | 1 min | | Pinot Server | http://pinot-server:8097/health | 200, body OK | 1 min | | Pinot Minion | http://pinot-minion:9514/health | 200 | 5 min | | Pinot ingestion lag | https://your-sidecar/health/pinot/ingestion/lag | 200, body "ok" | 1 min |

For the Broker query latency monitor, set the response time alert threshold to 2000ms — this fires before users notice dashboard slowness.

For the ingestion lag monitor, tune PINOT_LAG_THRESHOLD to a value appropriate for your SLA. For user-facing dashboards with a "near real-time" SLA, 10,000 records is a reasonable alert boundary.

Step 3: Heartbeat Monitoring for Pinot Ingestion Pipelines

Pinot's real-time ingestion health endpoint tells you about lag in the current consuming segment — but it doesn't tell you if your data producer has stopped writing to Kafka, or if the ingestion pipeline upstream of Pinot has stalled.

Vigilmon heartbeat monitors give you liveness checking at the producer level: your ingestion service pings Vigilmon after successfully writing events to Kafka. If pings stop, the alert fires regardless of what Pinot's segment lag looks like.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: pinot-kafka-producer
  3. Set the expected interval: 5 minutes
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL

Producer Service Integration (Java)

// PinotIngestionService.java
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
import java.time.Instant;

public class PinotIngestionService {
    private final HttpClient httpClient = HttpClient.newHttpClient();
    private final String vigilmonHeartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private long lastHeartbeatTime = 0;
    private static final long HEARTBEAT_INTERVAL_MS = 5 * 60 * 1000; // 5 minutes

    public void processEvents(List<Event> events) throws Exception {
        for (Event event : events) {
            writeToKafka(event);
        }

        // Ping Vigilmon every 5 minutes of successful writes
        long now = Instant.now().toEpochMilli();
        if (now - lastHeartbeatTime >= HEARTBEAT_INTERVAL_MS) {
            sendHeartbeat();
            lastHeartbeatTime = now;
        }
    }

    private void sendHeartbeat() {
        try {
            HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(vigilmonHeartbeatUrl))
                .GET()
                .build();
            httpClient.send(request, HttpResponse.BodyHandlers.discarding());
        } catch (Exception e) {
            // Heartbeat failure is non-fatal — log and continue
            System.err.println("Vigilmon heartbeat failed: " + e.getMessage());
        }
    }
}

Minion Task Heartbeat (for Offline Table Maintenance)

For Pinot Minion tasks (purge, merge, backfill), wrap the task runner:

# minion_task_runner.py
import os
import requests
import subprocess
import sys

VIGILMON_HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']

def run_minion_task(task_config_path):
    result = subprocess.run(
        ['pinot-admin.sh', 'RunSegmentCreationJob', '-jobSpec', task_config_path],
        capture_output=True, text=True
    )

    if result.returncode == 0:
        # Task succeeded — ping heartbeat
        requests.get(VIGILMON_HEARTBEAT_URL, timeout=5)
        print("Minion task succeeded, heartbeat sent")
    else:
        # Task failed — do NOT ping; Vigilmon will fire the alert
        print(f"Minion task failed:\n{result.stderr}", file=sys.stderr)
        sys.exit(1)

if __name__ == '__main__':
    run_minion_task(sys.argv[1])

Step 4: Alert Routing for Pinot Service Failures

Pinot failures have different blast radii depending on tier:

  • Controller failure: new table creation and segment assignment stop; queries continue from existing servers
  • Broker failure: queries fail or reroute to other brokers; user-facing impact within seconds
  • Server failure: query capacity and data coverage shrink; latency increases as remaining servers absorb load
  • Ingestion lag: real-time tables serve progressively staler data without query errors

Configure alert routing in Vigilmon to match severity:

| Monitor | Alert Channel | Priority | |---|---|---| | Pinot Broker liveness | Slack + PagerDuty | P1 | | Pinot Broker query latency | Slack + PagerDuty | P1 | | Pinot Controller liveness | Slack + PagerDuty | P1 | | Pinot Server liveness | Slack | P2 | | Pinot ingestion lag | Slack + email | P2 | | Pinot Minion | Email | P3 |

Set response time thresholds on the Broker monitor at 1500ms. Pinot is designed for sub-second queries; anything approaching 1.5s means Server-tier degradation is already affecting query routing.

Summary

Pinot's real-time architecture means failures are often invisible until dashboards visibly stale or slow down. External monitoring at every tier catches problems at the source before users see them:

| Monitor Type | What It Covers | |---|---| | HTTP probe on /health | Service process liveness per tier | | HTTP probe on query sidecar | End-to-end Broker query latency | | HTTP probe on ingestion sidecar | Real-time Kafka ingestion lag | | Heartbeat monitor | Data producer and pipeline liveness |

Get started free at vigilmon.online — your first Pinot monitor is live in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles