tutorial

How to Monitor Zipkin with Vigilmon

Zipkin is the distributed tracing backbone of many microservices architectures, but when the Zipkin server goes down, traces disappear silently. Learn how to monitor Zipkin availability, storage health, and trace ingestion with Vigilmon.

Zipkin is an open-source distributed tracing system that helps engineers understand latency problems in complex microservices architectures. Instrumented services send span data to a central Zipkin server, which stores and visualizes request traces across service boundaries. Zipkin is widely deployed at companies running distributed Java, Go, Python, and Node.js services and is a foundational piece of the observability stack for many engineering teams. But Zipkin's server is a single point of failure for your entire tracing pipeline: if it goes down, every instrumented service either silently drops traces or backs up its span buffer until it overflows. You lose distributed tracing visibility precisely when a latency incident is happening.

Vigilmon monitors Zipkin server health, storage layer status, and trace ingestion pipelines externally so you know immediately when your distributed tracing backend has a problem.

Why Zipkin Needs External Monitoring

Zipkin's collector, storage, and query layers can fail independently:

  • Server unavailability — the Zipkin process crashes or the JVM runs out of memory; all spans are dropped
  • Storage backend failure — Zipkin supports Cassandra, Elasticsearch, and MySQL; a storage failure makes the server appear healthy while all writes fail
  • Collector rejection — the /api/v2/spans endpoint returns 5xx when the storage or internal queue is overwhelmed
  • Query layer degradation — the server accepts spans but /api/v2/services or /api/v2/traces returns errors; engineers cannot investigate issues
  • Clock skew corruption — trace timestamps arrive out of order, making spans invisible in the UI without alerting on any error endpoint

Vigilmon gives you external HTTP monitoring that confirms the full stack — ingestion endpoint, query API, and storage health — from outside your service mesh.

Step 1: Verify Zipkin Health Endpoints

Zipkin exposes several endpoints useful for health checking:

# Health endpoint (Zipkin 2.x)
curl -s http://localhost:9411/health | jq .

# Check services endpoint (confirms query layer and storage are working)
curl -s http://localhost:9411/api/v2/services | jq .

# Check if server accepts spans (send a minimal test span)
curl -s -X POST http://localhost:9411/api/v2/spans \
  -H 'Content-Type: application/json' \
  -d '[{"traceId":"a2fb4a1d1a96d312","id":"a2fb4a1d1a96d312","name":"test","timestamp":'"$(date +%s%6N)"',"duration":1,"localEndpoint":{"serviceName":"health-check"}}]'

A successful span submission returns HTTP 202. The /health endpoint returns {"status":"UP"} when all storage backends are connected.

Step 2: Build a Health Proxy for Vigilmon

Create a composite health endpoint that validates Zipkin's full stack — storage connection, collector, and query API:

Node.js Health Proxy

// zipkin-health.js
const express = require('express');
const axios = require('axios');

const app = express();
const ZIPKIN_URL = process.env.ZIPKIN_URL || 'http://localhost:9411';

// Minimal test span — required fields only
function buildTestSpan() {
  const nowMicros = Date.now() * 1000;
  const traceId = Math.random().toString(16).slice(2, 18).padStart(16, '0');
  return [{
    traceId,
    id: traceId,
    name: 'vigilmon-health-check',
    timestamp: nowMicros,
    duration: 1000,
    localEndpoint: { serviceName: 'vigilmon-health-check' },
    tags: { 'source': 'vigilmon' },
  }];
}

app.get('/health/zipkin', async (req, res) => {
  const results = {};

  try {
    // 1. Health endpoint
    const healthRes = await axios.get(`${ZIPKIN_URL}/health`, { timeout: 5000 });
    results.health = healthRes.data;
    if (healthRes.data?.status !== 'UP') {
      return res.status(503).json({ status: 'down', reason: 'health_not_up', results });
    }
  } catch (err) {
    return res.status(503).json({ status: 'down', reason: 'health_endpoint_unreachable', error: err.message });
  }

  try {
    // 2. Query API — confirms storage reads work
    const servicesRes = await axios.get(`${ZIPKIN_URL}/api/v2/services`, { timeout: 8000 });
    results.servicesCount = servicesRes.data?.length ?? 0;
  } catch (err) {
    return res.status(503).json({
      status: 'degraded',
      reason: 'query_api_failed',
      error: err.message,
      results,
    });
  }

  try {
    // 3. Collector endpoint — confirms span ingestion works
    const spanRes = await axios.post(
      `${ZIPKIN_URL}/api/v2/spans`,
      buildTestSpan(),
      { headers: { 'Content-Type': 'application/json' }, timeout: 8000 }
    );
    results.collectorStatus = spanRes.status;
    if (spanRes.status !== 202) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'collector_rejected_spans',
        collectorStatus: spanRes.status,
        results,
      });
    }
  } catch (err) {
    return res.status(503).json({
      status: 'degraded',
      reason: 'collector_unreachable',
      error: err.message,
      results,
    });
  }

  return res.status(200).json({ status: 'ok', results });
});

app.listen(3011, () => console.log('Zipkin health proxy on :3011'));

Python Health Proxy (FastAPI)

# zipkin_health.py
from fastapi import FastAPI
from fastapi.responses import JSONResponse
import httpx, os, random, time

app = FastAPI()

ZIPKIN_URL = os.environ.get("ZIPKIN_URL", "http://localhost:9411")

def build_test_span():
    trace_id = format(random.randint(0, 2**64 - 1), '016x')
    return [{
        "traceId": trace_id,
        "id": trace_id,
        "name": "vigilmon-health-check",
        "timestamp": int(time.time() * 1_000_000),
        "duration": 1000,
        "localEndpoint": {"serviceName": "vigilmon-health-check"},
        "tags": {"source": "vigilmon"},
    }]

@app.get("/health/zipkin")
async def zipkin_health():
    results = {}

    async with httpx.AsyncClient(timeout=8.0) as client:
        try:
            health = await client.get(f"{ZIPKIN_URL}/health")
            results["health"] = health.json()
            if health.json().get("status") != "UP":
                return JSONResponse(status_code=503, content={
                    "status": "down", "reason": "health_not_up", "results": results
                })
        except Exception as e:
            return JSONResponse(status_code=503, content={
                "status": "down", "reason": "health_endpoint_unreachable", "error": str(e)
            })

        try:
            services = await client.get(f"{ZIPKIN_URL}/api/v2/services")
            results["servicesCount"] = len(services.json())
        except Exception as e:
            return JSONResponse(status_code=503, content={
                "status": "degraded", "reason": "query_api_failed", "error": str(e), "results": results
            })

        try:
            span_res = await client.post(
                f"{ZIPKIN_URL}/api/v2/spans",
                json=build_test_span(),
                headers={"Content-Type": "application/json"},
            )
            results["collectorStatus"] = span_res.status_code
            if span_res.status_code != 202:
                return JSONResponse(status_code=503, content={
                    "status": "degraded", "reason": "collector_rejected_spans", "results": results
                })
        except Exception as e:
            return JSONResponse(status_code=503, content={
                "status": "degraded", "reason": "collector_unreachable", "error": str(e), "results": results
            })

    return {"status": "ok", "results": results}

Step 3: Monitor Zipkin Storage Backend

If you run Zipkin with Elasticsearch or Cassandra, the storage layer can fail independently of the Zipkin process. Add a storage-specific check:

// Add to zipkin-health.js
app.get('/health/zipkin/storage', async (req, res) => {
  try {
    // Zipkin's /health response includes component-level status in some versions
    const healthRes = await axios.get(`${ZIPKIN_URL}/health`, { timeout: 5000 });
    const components = healthRes.data?.components || {};

    const degradedComponents = Object.entries(components)
      .filter(([, v]) => v?.status && v.status !== 'UP')
      .map(([name, v]) => ({ name, status: v.status, details: v.details }));

    if (degradedComponents.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'storage_components_unhealthy',
        degradedComponents,
      });
    }

    // Also probe the trace query endpoint with a time range that should return data
    const traceRes = await axios.get(`${ZIPKIN_URL}/api/v2/services`, { timeout: 10000 });
    const serviceCount = traceRes.data?.length ?? 0;

    return res.status(200).json({
      status: 'ok',
      components,
      serviceCount,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

This check is particularly important if you store traces in Elasticsearch — an Elasticsearch yellow or red cluster state typically degrades Zipkin's write throughput before it causes complete failure, giving you a window to act.

Step 4: Configure Vigilmon HTTP Monitors

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Zipkin health proxy: https://your-app.example.com/health/zipkin
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms (span submission and storage reads can be slow)
  6. Under Alert channels, assign your Slack or PagerDuty integration
  7. Save the monitor

Add a second monitor for storage health:

  • URL: https://your-app.example.com/health/zipkin/storage
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert channel: observability platform team Slack channel

Also add a direct health endpoint monitor for the fastest possible detection:

  • URL: https://zipkin.example.com/health
  • Expected: 200, body contains "status":"UP"
  • Interval: 30 seconds
  • Alert channel: SRE on-call PagerDuty

Step 5: Heartbeat Monitoring for Trace Ingestion Continuity

HTTP monitors confirm Zipkin is responding, but the critical metric is whether your services are actually delivering traces. A misconfigured SDK, network partition, or buffer overflow in your services can stop traces arriving at Zipkin without the server itself throwing errors.

Use a Vigilmon heartbeat monitor to confirm that traces arrive and are queryable within a time window:

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: zipkin-trace-ingestion-alive
  3. Set the expected interval: 5 minutes
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123xyz

Trace Ingestion Validator

// zipkin-trace-validator.js
const axios = require('axios');

const ZIPKIN_URL = process.env.ZIPKIN_URL || 'http://localhost:9411';
const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;
const SERVICE_NAME = process.env.TRACE_SERVICE_NAME || 'vigilmon-health-check';

async function validateTraceIngestion() {
  const traceId = Math.random().toString(16).slice(2, 18).padStart(16, '0');
  const nowMicros = Date.now() * 1000;

  // 1. Submit a synthetic trace
  await axios.post(`${ZIPKIN_URL}/api/v2/spans`, [{
    traceId,
    id: traceId,
    name: 'ingestion-validator',
    timestamp: nowMicros,
    duration: 2000,
    localEndpoint: { serviceName: SERVICE_NAME },
    tags: { 'vigilmon.check': 'true' },
  }], {
    headers: { 'Content-Type': 'application/json' },
    timeout: 10000,
  });

  // 2. Wait for async storage write
  await new Promise(r => setTimeout(r, 15000));

  // 3. Query for the trace by ID
  try {
    const traceRes = await axios.get(`${ZIPKIN_URL}/api/v2/trace/${traceId}`, { timeout: 10000 });
    if (traceRes.data?.length > 0) {
      if (HEARTBEAT_URL) {
        await axios.get(HEARTBEAT_URL, { timeout: 3000 }).catch(() => {});
      }
      console.log(`Trace ingestion validated: traceId=${traceId}`);
    } else {
      console.error(`Trace NOT found in Zipkin after storage wait: traceId=${traceId}`);
    }
  } catch (err) {
    console.error(`Failed to query trace: ${err.message}`);
  }
}

validateTraceIngestion().catch(console.error);
setInterval(() => validateTraceIngestion().catch(console.error), 5 * 60 * 1000);

This validator sends a synthetic span, waits for storage, then queries by trace ID. If the trace is not retrievable, the heartbeat does not ping and Vigilmon fires an alert.

Step 6: Alert Routing for Zipkin Failures

| Monitor | Alert Channel | Priority | |---|---|---| | /health/zipkin (full stack check) | Slack + PagerDuty | P1 | | /health/zipkin/storage (storage component health) | Slack + email | P1 | | Direct /health check | Slack + PagerDuty | P1 | | Heartbeat: trace-ingestion-alive | Slack + email | P2 |

Zipkin downtime during an incident is a force multiplier for pain — engineers cannot use distributed traces to diagnose the incident while the incident is happening. Configure your Zipkin monitors with zero tolerance for false negatives: use a 1-minute check interval and 0-minute grace period on the direct health check so that an alert fires on the first missed check.

Response time thresholds matter here too. Set the /health/zipkin proxy monitor to alert at 5000ms — slow Zipkin responses indicate storage layer pressure and precede complete failures.

Summary

Zipkin gives distributed tracing visibility to your microservices architecture — but the tracing infrastructure itself needs monitoring. Vigilmon provides the outside-in layer that keeps your Zipkin deployment reliable:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/zipkin | Full stack: health, query API, and span ingestion | | HTTP monitor on /health/zipkin/storage | Storage backend component health | | Direct health endpoint check | Fast Zipkin process availability detection | | Heartbeat monitor | End-to-end trace ingestion and queryability |

Get started free at vigilmon.online — your first Zipkin monitor is live in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles