tutorial

How to Monitor Heroic (Spotify's Time-Series Engine) with Vigilmon

Heroic is Spotify's open-source time-series aggregation engine built on Cassandra and Elasticsearch. Silent failures in its query federation or metadata indexing can blind your metrics stack. Here's how to monitor Heroic availability with Vigilmon.

Heroic is Spotify's open-source time-series aggregation engine — purpose-built for the scale Spotify operates at, with a federated query model that fans out to multiple Cassandra storage backends and uses Elasticsearch for metadata indexing. That federation is both its strength and its failure surface: a single backend shard unavailable, an Elasticsearch index lagging, or a query coordinator timing out can degrade or black-hole metrics silently while the Heroic process itself stays alive.

Vigilmon gives you external visibility into Heroic availability through HTTP probe monitoring and heartbeat monitoring for metric ingestion pipelines. This tutorial walks through both.

Why Heroic Monitoring Matters

Heroic's architecture introduces failure modes that process-level monitoring cannot see:

  • The query federation layer can fail for a subset of shards — queries return partial results without error codes
  • The Elasticsearch metadata index can become unavailable, causing metric tag queries to return empty while raw data still writes
  • Cassandra backend latency can spike, making Heroic queries time out at the federation layer before any storage error is surfaced
  • The ingestion pipeline (typically via FFWD or a custom producer) can back up and drop points silently when Heroic is under write pressure
  • Heroic exposes a /metrics endpoint via Dropwizard — but high JVM GC pause rates are a precursor to full query engine stalls that aren't visible until users report missing dashboards

External monitoring through Vigilmon tests the actual path your dashboards and alerting systems use to query Heroic.

Step 1: Use Heroic's Native Health Endpoint

Heroic exposes a /status endpoint when running. Test it:

curl -i http://your-heroic-host:8080/status
# HTTP/1.1 200 OK
# {"ok":true,...}

The /status response includes cluster health information. For a quick Vigilmon probe this is sufficient for process-level availability. For deeper ingestion-and-query coverage, build a sidecar.

Node.js Sidecar Health Endpoint

// heroic-health.js — checks both Heroic query and write paths
const express = require('express');
const axios = require('axios');

const app = express();
const HEROIC_URL = process.env.HEROIC_URL || 'http://localhost:8080';

app.get('/health/heroic', async (req, res) => {
  const results = {};

  // Check 1: cluster status
  try {
    const status = await axios.get(`${HEROIC_URL}/status`, { timeout: 5000 });
    if (!status.data.ok) {
      return res.status(503).json({ status: 'degraded', check: 'cluster', detail: status.data });
    }
    results.cluster = 'ok';
  } catch (err) {
    return res.status(503).json({ status: 'down', check: 'cluster', error: err.message });
  }

  // Check 2: write path — write a synthetic metric
  try {
    const now = Date.now();
    const writePayload = {
      series: [
        {
          key: 'health.probe',
          tags: { source: 'vigilmon' },
          points: [[now, 1.0]],
        },
      ],
    };
    await axios.post(`${HEROIC_URL}/write`, writePayload, { timeout: 5000 });
    results.write = 'ok';
  } catch (err) {
    return res.status(503).json({ status: 'degraded', check: 'write', error: err.message });
  }

  // Check 3: query path — query the synthetic metric back
  try {
    const now = Date.now();
    const queryPayload = {
      range: { type: 'relative', value: 60, unit: 'SECONDS' },
      filter: {
        type: 'and',
        filters: [
          { type: 'matchKey', value: 'health.probe' },
          { type: 'matchTag', tag: 'source', value: 'vigilmon' },
        ],
      },
      aggregators: [{ type: 'sum', sampling: { value: 30, unit: 'SECONDS' } }],
    };
    const result = await axios.post(`${HEROIC_URL}/query/metrics`, queryPayload, { timeout: 10000 });
    if (!result.data || !result.data.result || result.data.result.length === 0) {
      return res.status(503).json({ status: 'degraded', check: 'query', reason: 'no_data_returned' });
    }
    results.query = 'ok';
  } catch (err) {
    return res.status(503).json({ status: 'degraded', check: 'query', error: err.message });
  }

  return res.status(200).json({ status: 'ok', ...results });
});

app.listen(3001);

Python Equivalent

# heroic_health.py — FastAPI sidecar for Heroic health
import os, time
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
HEROIC_URL = os.environ.get("HEROIC_URL", "http://localhost:8080")

@app.get("/health/heroic")
async def heroic_health():
    async with httpx.AsyncClient(timeout=10.0) as client:
        # Cluster status
        try:
            r = await client.get(f"{HEROIC_URL}/status")
            if not r.json().get("ok"):
                return JSONResponse(status_code=503, content={"status": "degraded", "check": "cluster"})
        except Exception as e:
            return JSONResponse(status_code=503, content={"status": "down", "check": "cluster", "error": str(e)})

        # Write path
        now_ms = int(time.time() * 1000)
        payload = {
            "series": [{
                "key": "health.probe",
                "tags": {"source": "vigilmon"},
                "points": [[now_ms, 1.0]],
            }]
        }
        try:
            await client.post(f"{HEROIC_URL}/write", json=payload)
        except Exception as e:
            return JSONResponse(status_code=503, content={"status": "degraded", "check": "write", "error": str(e)})

        # Query path
        query = {
            "range": {"type": "relative", "value": 60, "unit": "SECONDS"},
            "filter": {
                "type": "and",
                "filters": [
                    {"type": "matchKey", "value": "health.probe"},
                    {"type": "matchTag", "tag": "source", "value": "vigilmon"},
                ],
            },
            "aggregators": [{"type": "sum", "sampling": {"value": 30, "unit": "SECONDS"}}],
        }
        try:
            r = await client.post(f"{HEROIC_URL}/query/metrics", json=query, timeout=15.0)
            data = r.json()
            if not data.get("result"):
                return JSONResponse(status_code=503, content={"status": "degraded", "check": "query"})
        except Exception as e:
            return JSONResponse(status_code=503, content={"status": "degraded", "check": "query", "error": str(e)})

    return {"status": "ok", "cluster": "ok", "write": "ok", "query": "ok"}

Deploy and verify:

curl -i https://your-app.example.com/health/heroic
# HTTP/1.1 200 OK
# {"status":"ok","cluster":"ok","write":"ok","query":"ok"}

Step 2: Configure a Vigilmon HTTP Monitor for Heroic

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-app.example.com/health/heroic
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms (federated queries have inherent latency)
  6. Under Alert channels, assign your Slack or email channel
  7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously. A single-region transient failure will not page you — Vigilmon requires multi-region consensus before opening an incident, giving you confident, low-noise alerts.

Failure Coverage

| Failure Mode | Process Monitor | Vigilmon | |---|---|---| | Heroic process crash | ✓ | ✓ | | Cassandra backend shard unavailable | ✗ | ✓ | | Elasticsearch metadata index down | ✗ | ✓ | | Federated query timeout | ✗ | ✓ | | Write path backpressure / drops | ✗ | ✓ |

Step 3: Heartbeat Monitoring for Metric Ingestion Pipelines

Metric ingestion into Heroic — via FFWD, custom producers, or Kafka consumers — fails silently when Heroic is under load or briefly unavailable. Data is simply dropped. Vigilmon's heartbeat monitors detect silent ingestion death.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: heroic-metric-ingestion
  3. Set the expected interval to match your flush frequency (e.g., 2 minutes)
  4. Set the grace period: 5 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Producer

Node.js Metric Producer:

const axios = require('axios');

async function flushToHeroic(batch) {
  await axios.post(`${process.env.HEROIC_URL}/write`, { series: batch }, { timeout: 5000 });
  // Ping only after successful write
  await axios.get(process.env.VIGILMON_HEARTBEAT_URL).catch(() => {});
}

Python Metric Producer:

import requests, os

def flush_to_heroic(batch):
    requests.post(
        f"{os.environ['HEROIC_URL']}/write",
        json={"series": batch},
        timeout=5,
    ).raise_for_status()
    requests.get(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)

Kafka Consumer (Java-style pseudocode):

// After each successful Heroic write in your consumer loop:
httpClient.get(System.getenv("VIGILMON_HEARTBEAT_URL"));

Step 4: Monitor Heroic's Dropwizard Metrics Endpoint

Heroic exposes internal JVM and application metrics at /metrics in Dropwizard/JSON format. You can use Vigilmon's HTTP monitor to check for early warning signs without a sidecar:

  1. Create a second Vigilmon monitor pointing to http://your-heroic-host:8080/metrics
  2. Set the expected status code to 200
  3. Set a response time threshold of 2000ms — a slow /metrics response indicates JVM GC pressure

Separately, watch for "gc.G1-Old-Generation.time" counts growing rapidly in the response body. You can add a response body check in Vigilmon for the shape of the response to catch a completely unhealthy metrics endpoint.

Step 5: Alert Routing for Heroic Outages

Heroic outages typically cascade: write backpressure builds → producers retry → Cassandra write latency increases → queries time out → dashboards go blank → alerting rules stop firing. Page before this cascade completes.

In Vigilmon, configure alert channels:

  1. Heroic health monitor → immediate Slack + PagerDuty (P1 — metrics blind)
  2. Ingestion heartbeat monitors → Slack + email (P1 — data loss in progress)
  3. Heroic /metrics endpoint → Slack (P2 — early warning of JVM pressure)

Set a response time alert at 5000ms on the health monitor. Heroic query slowdowns precede full outages by minutes — catching high latency early gives your team time to scale or shed load.

Summary

Heroic's federated architecture means partial failures are invisible to process monitors — federated shards can fail while the process stays up. Vigilmon covers the full path.

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/heroic | Cluster status, write path, query path | | HTTP monitor on /status | Native Heroic cluster health | | HTTP monitor on /metrics | JVM pressure early warning | | Heartbeat monitor | Ingestion pipeline continuity |

Get started free at vigilmon.online — your first Heroic monitor takes under two minutes to configure.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles