tutorial

How to Monitor Jaeger Distributed Tracing Health with Vigilmon

Jaeger collector failures and storage backend disconnects silently drop traces in production. Learn how to monitor Jaeger collector health, query service uptime, storage backend connectivity, trace ingestion rate, and sampling strategy health with Vigilmon HTTP probes.

Jaeger is the CNCF-graduated distributed tracing system built by Uber — the backbone of observability for microservice architectures. When Jaeger fails, it fails quietly: the collector stops accepting spans, your storage backend silently disconnects, and your engineers debug production incidents completely blind, without a single trace to show for it.

Vigilmon adds external health monitoring for Jaeger through its built-in health endpoints and optional health aggregation sidecars. This tutorial covers collector health, query service uptime, storage backend connectivity, trace ingestion rate, sampling strategy health, and Vigilmon integration.

Why Jaeger Needs External Monitoring

Jaeger's architecture has multiple failure points that don't self-report clearly:

  • Collector overload or crash — high trace volume can OOM the collector process; traces are dropped silently
  • Storage backend disconnects — if Cassandra or Elasticsearch becomes unreachable, the collector queues traces briefly, then drops them, while still appearing "Running"
  • Query service failure — engineers can't search traces during incidents, exactly when they need them most
  • Silent port failures — the collector listens on multiple ports (14250 gRPC, 14268 HTTP); port binding failures drop all traces from agents using that protocol

Vigilmon external monitoring adds:

  • Collector liveness checks that fire immediately if the collector process crashes or hangs
  • Query service uptime monitoring so engineers can trust trace search during incidents
  • Storage connectivity checks to catch Elasticsearch/Cassandra disconnects before trace data loss
  • Trace ingestion rate monitoring to detect when span volume drops to zero
  • Sampling strategy health for remote sampling configurations
  • Multi-region probe consensus to separate Jaeger failures from network blips

Step 1: Identify Jaeger Health Endpoints

Jaeger exposes health check endpoints on each component. Default ports when running via Docker or binary:

| Component | Port | What to check | |---|---|---| | Collector (HTTP spans) | 14268 | GET / → 405 (healthy — expects POST) | | Collector (admin) | 14269 | GET / → 200 with metrics | | Query (API + UI) | 16686 | GET /api/services → JSON list | | Query (admin) | 16687 | GET / → 200 | | Agent (if deployed) | 14271 | GET / → 200 | | Ingester (Kafka mode) | 14270 | GET / → 200 |

Verify the key endpoints are reachable:

# Query API — confirms query service AND storage are working
curl -s http://localhost:16686/api/services | python3 -m json.tool

# Collector admin
curl -s http://localhost:14269/

# Collector HTTP endpoint (405 is healthy — it expects POST for span ingestion)
curl -v http://localhost:14268/ 2>&1 | grep "< HTTP"

Step 2: Configure Vigilmon HTTP Monitors for Jaeger Components

Monitor 1: Jaeger Query API (Service + Storage Health)

The /api/services endpoint is your best Jaeger health indicator. It queries the storage backend and returns real data — if it responds, both the query service and storage are working.

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://jaeger.yourdomain.com/api/services
  4. Check interval: 1 minute
  5. Expected response: Status 200, body contains "data"
  6. Response time threshold: 5000ms
  7. Alert channel: Slack + PagerDuty
  8. Save

Monitor 2: Jaeger UI

The Jaeger web UI can fail independently of the API (proxy misconfiguration, broken build assets). Monitor it separately:

  • URL: https://jaeger.yourdomain.com/
  • Expected: 200, body contains Jaeger
  • Interval: 2 minutes
  • Alert: Slack (P2 — data loss isn't occurring, but incident response is impaired)

Monitor 3: Jaeger Collector

The collector must stay running or your entire trace pipeline breaks:

  • URL: http://jaeger-collector.yourdomain.com:14268/
  • Expected status: 200 or 405 — both indicate the HTTP server is alive
  • Interval: 1 minute
  • Alert: PagerDuty (P1 — collector down means all incoming spans are dropped)

Add a TCP monitor for the gRPC collector port:

  1. Go to Monitors → New Monitor → TCP Port
  2. Host: jaeger-collector.yourdomain.com
  3. Port: 14250
  4. Interval: 1 minute
  5. Alert: PagerDuty (P1)

Step 3: Monitor Storage Backend Connectivity

Jaeger can appear healthy while silently failing to persist traces because the storage backend is unreachable. Build a sidecar that tests storage connectivity directly:

// jaeger-storage-health.js
const express = require('express');
const axios = require('axios');

const app = express();

const STORAGE_TYPE = process.env.JAEGER_STORAGE_TYPE || 'elasticsearch';
const ES_URL = process.env.ELASTICSEARCH_URL || 'http://elasticsearch:9200';
const CASSANDRA_HEALTH_URL = process.env.CASSANDRA_HEALTH_URL;  // optional HTTP health wrapper

app.get('/health/jaeger/storage', async (req, res) => {
  try {
    if (STORAGE_TYPE === 'elasticsearch') {
      const resp = await axios.get(`${ES_URL}/_cluster/health`, { timeout: 5000 });
      const esHealth = resp.data;

      if (esHealth.status === 'red') {
        return res.status(503).json({
          status: 'down',
          reason: 'elasticsearch_red',
          es_status: esHealth.status,
          unassigned_shards: esHealth.unassigned_shards,
        });
      }
      if (esHealth.status === 'yellow') {
        return res.status(503).json({
          status: 'degraded',
          reason: 'elasticsearch_yellow',
          es_status: esHealth.status,
        });
      }
      return res.status(200).json({
        status: 'ok',
        es_status: esHealth.status,
        num_nodes: esHealth.number_of_nodes,
      });
    }

    // Cassandra: probe a custom health wrapper or use badger (no external check needed)
    if (STORAGE_TYPE === 'cassandra' && CASSANDRA_HEALTH_URL) {
      const resp = await axios.get(CASSANDRA_HEALTH_URL, { timeout: 5000 });
      return res.status(resp.status === 200 ? 200 : 503).json({
        status: resp.status === 200 ? 'ok' : 'down',
        storage_type: 'cassandra',
      });
    }

    return res.status(200).json({ status: 'ok', storage_type: STORAGE_TYPE });
  } catch (err) {
    return res.status(503).json({
      status: 'down',
      reason: 'storage_unreachable',
      error: err.message,
    });
  }
});

app.listen(process.env.PORT || 3011);

Add a Vigilmon monitor for this endpoint:

  • URL: https://your-host.example.com/health/jaeger/storage
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert: Slack + PagerDuty (P1 — storage failure means traces are not being persisted)

Step 4: Monitor Trace Ingestion Rate

Process health checks confirm Jaeger is running — they don't prove traces are actually flowing. Build a trace ingestion check that queries the API for recent data:

// Add to your health sidecar
const JAEGER_QUERY_URL = process.env.JAEGER_QUERY_URL || 'http://jaeger-query:16686';

app.get('/health/jaeger/ingestion', async (req, res) => {
  try {
    // Get list of registered services
    const servicesResp = await axios.get(
      `${JAEGER_QUERY_URL}/api/services`,
      { timeout: 5000 }
    );
    const services = servicesResp.data?.data || [];

    if (services.length === 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'no_services_in_jaeger',
        note: 'no traces have been ingested or all traces have expired',
      });
    }

    // Check for recent traces from the first service
    const testService = services.find(s => !s.includes('jaeger')) || services[0];
    const lookbackMs = 10 * 60 * 1000;  // 10 minutes
    const now = Date.now() * 1000;
    const start = (Date.now() - lookbackMs) * 1000;

    const tracesResp = await axios.get(
      `${JAEGER_QUERY_URL}/api/traces?service=${encodeURIComponent(testService)}&start=${start}&end=${now}&limit=1`,
      { timeout: 5000 }
    );
    const recentTraces = tracesResp.data?.data?.length || 0;

    return res.status(200).json({
      status: 'ok',
      services_registered: services.length,
      recent_traces_in_10m: recentTraces,
      sample_service: testService,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

Add a Vigilmon monitor for ingestion health:

  • URL: https://your-host.example.com/health/jaeger/ingestion
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert: Slack (P2 — zero ingestion indicates complete pipeline stall)

For additional coverage, use a Vigilmon Heartbeat monitor to send a test span to the collector every 5 minutes and verify it returns 200 or 202. Set grace period to 10 minutes — if no heartbeat arrives in 10 minutes, Vigilmon alerts.

Step 5: Monitor Sampling Strategy Health

If you use Jaeger's remote sampling strategy server, engineers' services fetch their sampling configuration from it on startup. If the sampling endpoint is down, new service deployments may fall back to 100% sampling (overwhelming the collector) or zero sampling (losing all traces).

// Add to health sidecar
const SAMPLING_URL = process.env.JAEGER_SAMPLING_URL || 'http://jaeger-agent:5778/sampling';

app.get('/health/jaeger/sampling', async (req, res) => {
  const testService = process.env.SAMPLING_TEST_SERVICE || 'my-service';
  try {
    const resp = await axios.get(
      `${SAMPLING_URL}?service=${encodeURIComponent(testService)}`,
      { timeout: 3000 }
    );
    const strategy = resp.data;

    // Verify strategy contains a valid sampling configuration
    const hasStrategy = strategy.strategyType !== undefined ||
      strategy.probabilisticSampling !== undefined ||
      strategy.rateLimitingSampling !== undefined;

    if (!hasStrategy) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'empty_or_invalid_sampling_strategy',
      });
    }

    return res.status(200).json({
      status: 'ok',
      strategy_type: strategy.strategyType,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

Monitor this at a 5-minute interval with Slack alerting (P3 — failure won't cause immediate data loss but will cause misconfigured sampling in newly deployed services).

Step 6: Alert Routing for Jaeger

| Monitor | Alert Channel | Priority | |---|---|---| | Query API /api/services | Slack + PagerDuty | P1 | | Collector HTTP (14268) | PagerDuty | P1 | | Collector gRPC TCP (14250) | PagerDuty | P1 | | Storage connectivity | Slack + PagerDuty | P1 | | Query UI | Slack | P2 | | Trace ingestion rate | Slack | P2 | | Sampling strategy | Slack | P3 |

Collector + storage are always P1: both must be healthy to accept and persist traces. A collector failure drops spans at the entry point; a storage failure drops spans that have already been accepted.

For multi-replica deployments, create individual monitors per replica and use monitor grouping in Vigilmon to distinguish single-replica failures from cluster-wide outages.

Summary

Jaeger failures are silent and high-impact: when the collector goes down or storage disconnects, you lose distributed traces exactly when you need them most — during production incidents. External monitoring with Vigilmon catches these failures before your on-call engineers discover them the hard way:

| Monitor Type | What It Covers | |---|---| | HTTP on query API | Query service liveness + storage connectivity | | HTTP on collector | Span ingestion availability | | TCP on gRPC port | gRPC collector for SDK clients | | HTTP on storage health | Elasticsearch/Cassandra connectivity | | HTTP on ingestion rate | Active trace flow through the pipeline | | HTTP on sampling health | Remote sampling strategy availability |

Get started free at vigilmon.online — your first Jaeger monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles