tutorial

How to Monitor Apache DataSketches with Vigilmon (Uptime, APIs, and Alerts)

Apache DataSketches is a probabilistic data structures library for approximate big data analytics, providing production-grade implementations of HyperLogLog ...

Apache DataSketches is a probabilistic data structures library for approximate big data analytics, providing production-grade implementations of HyperLogLog (cardinality estimation), Theta sketches (set operations at scale), KLL sketches (quantile approximation), and frequency sketches — all with mathematically proven error bounds. DataSketches is embedded into query engines like Apache Druid, Apache Hive, and Apache Pinot, and is frequently deployed as a standalone aggregation microservice used for real-time distinct count and percentile computations at petabyte scale.

When the Druid broker node serving your DataSketches-based cardinality queries goes offline, your analytics dashboards show stale counts or fail to load. When the aggregation microservice wrapping DataSketches sketch merges crashes, all downstream distinct-count endpoints return 500s. This tutorial shows you how to set up uptime monitoring for Apache DataSketches deployments using Vigilmon — free tier, no credit card.

Why DataSketches deployments need external monitoring

DataSketches adds monitoring surface area depending on how you deploy it:

  • Embedded in Druid — if Druid Broker, Coordinator, or Historical nodes go down, all DataSketches-based queries fail; the Druid cluster's web console may remain partially available while query serving is broken
  • Sketch aggregation microservices — teams often expose sketch merge endpoints via a REST service (e.g., a Dropwizard or Spring Boot application); those services can crash or OOM when handling very large sketch unions
  • Sketch store failures — sketches serialized to Redis, Cassandra, or object storage need those stores available for incremental updates; a store outage causes sketch writes to fail silently or throw unhandled exceptions
  • Stale sketch windows — in sliding-window analytics, if the update job stops running, the sketch data grows stale but queries continue to return (now incorrect) results with no error signal
  • Druid ingestion stalls — if the Druid ingestion supervisor stops pulling from Kafka, DataSketches aggregations in real-time queries fall behind, and no external alert fires by default

External monitoring from Vigilmon gives you real-time visibility across all these layers.

What you'll need

  • A DataSketches deployment — embedded in Druid/Hive/Pinot, or as a standalone aggregation service
  • Network-reachable health endpoints for the query layer
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose health endpoints for your DataSketches layer

If DataSketches is embedded in Apache Druid, Druid exposes built-in health and status endpoints:

# Druid Router health
curl http://druid-router.example.com:8888/status/health

# Druid Broker health
curl http://druid-broker.example.com:8082/status/health

# Druid Coordinator
curl http://druid-coordinator.example.com:8081/status/health

# Druid Historical
curl http://druid-historical.example.com:8083/status/health

Each of these returns a JSON response when the node is healthy.

If you run a standalone DataSketches aggregation microservice (Spring Boot with Dropwizard Metrics or Spring Actuator), expose a health endpoint:

// Spring Boot Actuator — enabled by default at /actuator/health
// For a custom sketch-aware health check:

@Component
public class SketchServiceHealthIndicator implements HealthIndicator {

    private final SketchStore sketchStore;

    public SketchServiceHealthIndicator(SketchStore sketchStore) {
        this.sketchStore = sketchStore;
    }

    @Override
    public Health health() {
        try {
            // Attempt a read from the sketch store to verify connectivity
            long estimatedCardinality = sketchStore.getEstimate("healthcheck-sketch");
            return Health.up()
                .withDetail("sketch_store", "reachable")
                .withDetail("sample_estimate", estimatedCardinality)
                .build();
        } catch (Exception e) {
            return Health.down(e)
                .withDetail("sketch_store", "unreachable")
                .build();
        }
    }
}

For a lightweight Python microservice:

from flask import Flask, jsonify
from datasketches import hll_sketch, hll_union
import redis, time

app = Flask(__name__)
redis_client = redis.Redis(host="redis.example.com", port=6379)

@app.get("/health")
def health():
    try:
        redis_client.ping()
        return jsonify({"status": "ok", "sketch_store": "reachable"}), 200
    except Exception as e:
        return jsonify({"status": "degraded", "sketch_store": "unreachable", "error": str(e)}), 503

@app.get("/estimate/<sketch_key>")
def estimate(sketch_key):
    raw = redis_client.get(sketch_key)
    if raw is None:
        return jsonify({"error": "sketch not found"}), 404
    sketch = hll_sketch.deserialize(raw)
    return jsonify({"key": sketch_key, "estimate": sketch.get_estimate()})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

For Docker deployments:

services:
  datasketches-api:
    image: your-org/datasketches-api:latest
    ports:
      - "8080:8080"
    environment:
      REDIS_HOST: redis.example.com
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    restart: unless-stopped

Step 2: Monitor the Druid Broker health endpoint

The Druid Broker is the query entry point — if it is down, all DataSketches-based analytics queries fail:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. URL: http://druid-broker.example.com:8082/status/health
  4. Set check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
  6. Save the monitor

Step 3: Monitor the Druid Coordinator and Router

The Coordinator manages segment assignment; the Router is the front door to the cluster:

  1. Go to Monitors → New Monitor

  2. Choose HTTP / HTTPS

  3. URL: http://druid-router.example.com:8888/status/health

  4. Expected status: 200

  5. Save the monitor

  6. Repeat for the Coordinator: http://druid-coordinator.example.com:8081/status/health

Step 4: Monitor your standalone DataSketches aggregation service

  1. Go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://datasketches-api.example.com/health
  4. Expected status: 200
  5. Response body contains: "status":"ok"
  6. Save the monitor

The 503 degraded response fires an incident when the sketch store (Redis, Cassandra) becomes unreachable — catching data-layer failures that would otherwise silently corrupt your cardinality estimates.

Step 5: Add TCP port monitoring for the sketch store

The sketch store is a dependency of your DataSketches service — if it goes down, sketch reads and writes both fail:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Host: redis.example.com, Port: 6379
  4. Save the monitor

| Service | Default port | |---------|-------------| | Redis | 6379 | | Cassandra | 9042 | | Druid Broker | 8082 | | Druid Coordinator | 8081 | | Druid Router | 8888 | | Druid Historical | 8083 | | Kafka (ingestion source) | 9092 |

Step 6: Set up a heartbeat monitor for sketch update jobs

If you run batch jobs that periodically update or merge sketches (for example, a nightly distinct-count rollup), use Vigilmon heartbeats to detect silent job failures:

  1. Go to Monitors → New Monitor
  2. Choose Heartbeat
  3. Name it datasketches-update-job-heartbeat
  4. Set the expected interval to match your job schedule (e.g., 1 hour)
  5. Copy the generated heartbeat URL
  6. Add the ping to your update job:
#!/bin/bash
java -jar datasketches-update-job.jar \
    --input kafka://kafka.example.com:9092/events \
    --output redis://redis.example.com:6379

if [ $? -eq 0 ]; then
    curl -fsS "https://hb.vigilmon.online/your-heartbeat-id" > /dev/null
fi

If the job fails or is never scheduled, Vigilmon opens an incident after the expected interval passes — alerting you that your sketch data is growing stale.

Step 7: Configure webhook alerts

DataSketches outages affect analytics dashboards and real-time distinct-count APIs that stakeholders rely on. Configure alerts to reach the right people immediately.

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your Slack, PagerDuty, or OpsGenie webhook URL
  3. Assign the channel to all your DataSketches monitors

Example alert payload from Vigilmon:

{
  "monitor_name": "DataSketches API Health",
  "status": "down",
  "url": "https://datasketches-api.example.com/health",
  "started_at": "2026-05-20T03:14:00Z",
  "duration_seconds": 60
}

Consider separate alert channels for the data infrastructure team (Druid, Redis, Kafka) and the analytics engineering team (sketch update jobs, API endpoints) — they have different response runbooks.

Step 8: Create a status page for your analytics platform

If product and data science teams depend on DataSketches-powered dashboards, give them a status page:

  1. Go to Status Pages → New Status Page
  2. Name it (e.g. "Analytics Platform Status")
  3. Add all your monitors — Druid Broker, Druid Router, sketch API, Redis TCP, heartbeat
  4. Publish the page

Share the URL in your data portal or BI tool documentation so teams can check status before filing incident tickets.

Monitor coverage summary

| Monitor | Type | What it catches | |---------|------|-----------------| | http://druid-broker.example.com:8082/status/health | HTTP | Druid Broker outage, query failures | | http://druid-router.example.com:8888/status/health | HTTP | Druid Router cluster entry failure | | http://druid-coordinator.example.com:8081/status/health | HTTP | Coordinator failure, segment management loss | | https://datasketches-api.example.com/health | HTTP | Sketch service crash, sketch store unreachable | | redis.example.com:6379 | TCP | Sketch store unreachable | | kafka.example.com:9092 | TCP | Ingestion source unreachable | | datasketches-update-job-heartbeat | Heartbeat | Silent sketch update job failure, stale estimates |

What's next

  • SSL certificate monitoring — Vigilmon tracks TLS expiry across all your HTTPS endpoints, including your sketch API
  • Response time tracking — latency spikes on sketch merge endpoints often predict memory pressure before they cause OOM crashes

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles