How to Monitor the OpenTelemetry Demo Application with Vigilmon

The OpenTelemetry Demo is the CNCF's reference architecture for OpenTelemetry — a constellation of microservices (frontend, cart, checkout, payment, email, product catalog, recommendation, shipping, and more) all instrumented with OTel SDKs. It ships as a Kubernetes-native application and serves as both a learning environment and a real-world model for OTel in production.

Monitoring the OTel Demo (or production systems based on it) requires a different approach than monitoring a single service: you need service-level health checks across all microservices, trace pipeline completeness monitoring, metric export health, and log pipeline verification. Vigilmon adds the external layer of health monitoring that confirms your OTel stack is collecting and exporting telemetry end-to-end.

Why the OTel Demo Needs External Monitoring

The OTel Demo ships with Prometheus, Grafana, and Jaeger included — but those tools only work if the telemetry pipeline itself is healthy. External monitoring with Vigilmon adds:

• Per-service HTTP health checks that fire if any demo microservice crashes or stops responding
• OTel Collector health monitoring to confirm telemetry is flowing through the pipeline
• Export target reachability checks for Prometheus, Jaeger, and any custom backend
• End-to-end trace completeness via a synthetic probe that confirms trace data reaches its destination
• Multi-region availability to catch network-level failures separate from application failures

Step 1: Identify Service Health Endpoints

The OTel Demo exposes health endpoints across its services. When deployed with default ports (via docker compose or Kubernetes), the key endpoints are:

# Frontend (the main entry point)
curl -s http://localhost:8080/    # HTTP 200 confirms frontend is serving

# OpenTelemetry Collector health check
curl -s http://localhost:13133/   # Returns "Server available" when OTel Collector is healthy

# Prometheus (verifies metric exports are reachable)
curl -s http://localhost:9090/-/healthy   # Returns "Prometheus Server is Healthy."

# Grafana
curl -s http://localhost:3000/api/health   # Returns {"database": "ok"}

# Jaeger query
curl -s http://localhost:16686/   # Jaeger UI reachability

# Flagd (feature flag service)
curl -s http://localhost:8013/healthz   # Returns healthy

# OpenTelemetry Collector zpages
curl -s http://localhost:55679/debug/rpcz

For Kubernetes deployments, replace localhost with the appropriate Service/Ingress hostname or ClusterIP.

Step 2: Monitor the OTel Collector Health

The OpenTelemetry Collector is the central hub — all telemetry flows through it. It exposes a dedicated health check extension:

# otel-collector-config.yaml
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
    path: "/health/status"
    check_collector_pipeline:
      enabled: true
      interval: 5m
      exporter_failure_threshold: 5

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

With check_collector_pipeline enabled, the health endpoint returns 503 if the collector detects persistent exporter failures — meaning Vigilmon can alert you when the collector is running but failing to export.

Step 3: Build a Pipeline Completeness Health Endpoint

A healthy OTel Collector process doesn't prove telemetry is making it end-to-end. Build a health sidecar that probes across the full pipeline:

// otel-pipeline-health.js
const express = require('express');
const axios = require('axios');

const app = express();

const CHECKS = {
  otel_collector: process.env.OTEL_COLLECTOR_HEALTH || 'http://otelcol:13133/',
  prometheus: process.env.PROMETHEUS_HEALTH || 'http://prometheus:9090/-/healthy',
  grafana: process.env.GRAFANA_HEALTH || 'http://grafana:3000/api/health',
  jaeger: process.env.JAEGER_HEALTH || 'http://jaeger:16686/',
};

async function checkEndpoint(name, url) {
  try {
    const resp = await axios.get(url, { timeout: 5000 });
    return { name, status: 'ok', code: resp.status };
  } catch (err) {
    const code = err.response?.status || null;
    return { name, status: 'down', code, error: err.message };
  }
}

app.get('/health/otel-pipeline', async (req, res) => {
  const results = await Promise.all(
    Object.entries(CHECKS).map(([name, url]) => checkEndpoint(name, url))
  );

  const failed = results.filter(r => r.status !== 'ok');

  if (failed.length > 0) {
    return res.status(503).json({
      status: 'degraded',
      failed_checks: failed,
      all_checks: results,
    });
  }

  return res.status(200).json({
    status: 'ok',
    checks: results,
  });
});

// Trace completeness: verify Jaeger has received traces recently
app.get('/health/otel-pipeline/traces', async (req, res) => {
  try {
    // Query Jaeger for services — if the demo is working, services should appear
    const servicesResp = await axios.get(
      `${process.env.JAEGER_API || 'http://jaeger:16686'}/api/services`,
      { timeout: 5000 }
    );
    const services = servicesResp.data?.data || [];

    // Expect at least the core OTel Demo services to be visible in Jaeger
    const expectedServices = ['frontend', 'checkoutservice', 'cartservice'];
    const foundServices = services.filter(s => expectedServices.some(e => s.includes(e)));

    if (foundServices.length === 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'no_demo_services_in_jaeger',
        services_found: services.length,
      });
    }

    return res.status(200).json({
      status: 'ok',
      services_in_jaeger: services.length,
      demo_services_found: foundServices,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(process.env.PORT || 3009);

Step 4: Configure Vigilmon HTTP Monitors

Monitor 1: Full Pipeline Health

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: https://your-host.example.com/health/otel-pipeline
4. Check interval: 1 minute
5. Expected response: Status 200, body contains "status":"ok"
6. Response time threshold: 10000ms (pipeline health check fans out to multiple services)
7. Alert channel: Slack + PagerDuty
8. Save

Monitor 2: OTel Collector Direct

• URL: http://your-host:13133/ (or via reverse proxy)
• Expected: 200
• Interval: 1 minute
• Alert: PagerDuty (P1 — collector down means all telemetry stops)

Monitor 3: Frontend Service Availability

For the demo's primary user-facing endpoint:

• URL: https://your-host.example.com/ (the demo storefront)
• Expected: 200
• Interval: 1 minute
• Alert: Slack

This confirms the entire demo stack is functional from a user perspective, not just infrastructure health.

Monitor 4: Trace Pipeline Completeness

• URL: https://your-host.example.com/health/otel-pipeline/traces
• Expected: 200, body contains "status":"ok"
• Interval: 5 minutes (Jaeger data can lag by a minute or two)
• Alert: Slack (P2 — trace loss is serious but not immediate outage)

Step 5: Monitor Individual Demo Microservices

For production OTel deployments based on the Demo architecture, monitor each microservice independently. Create monitors for each service health endpoint:

# Map of OTel Demo services to their health endpoints
frontend:         GET /  → 200
productcatalog:   GET /healthz → 200
cart:             gRPC health (use HTTP wrapper)
checkout:         gRPC health (use HTTP wrapper)
payment:          gRPC health (use HTTP wrapper)
email:            GET /actuator/health → 200 (if Spring Boot)
recommendation:   GET /_ah/health → 200
shipping:         GET /healthz → 200

For gRPC services without HTTP health endpoints, add a lightweight HTTP wrapper:

// grpc-health-wrapper.js
const express = require('express');
const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');

const app = express();

app.get('/health/grpc/:service', async (req, res) => {
  // Uses gRPC standard health check protocol (grpc.health.v1)
  const target = process.env[`${req.params.service.toUpperCase()}_ADDR`];
  if (!target) return res.status(404).json({ error: 'unknown service' });

  // Basic TCP connectivity check as a proxy for gRPC health
  const net = require('net');
  const [host, port] = target.split(':');
  const socket = new net.Socket();
  socket.setTimeout(3000);
  socket.connect(parseInt(port), host, () => {
    socket.destroy();
    res.status(200).json({ status: 'ok', service: req.params.service });
  });
  socket.on('error', (err) => {
    socket.destroy();
    res.status(503).json({ status: 'down', error: err.message });
  });
});

app.listen(3010);

Step 6: Alert Routing for OTel Demo

| Monitor | Alert Channel | Priority | |---|---|---| | Pipeline health /health/otel-pipeline | Slack + PagerDuty | P1 | | OTel Collector direct | PagerDuty | P1 | | Trace completeness | Slack | P2 | | Frontend availability | Slack + PagerDuty | P1 | | Individual microservices | Slack | P2 |

Collector failure is always P1: when the OTel Collector is down, all three telemetry signals (traces, metrics, logs) stop flowing simultaneously. No other component failure has that blast radius.

Summary

The OpenTelemetry Demo (and production OTel deployments) require layered monitoring: not just "is the process running" but "is telemetry actually making it end-to-end." Vigilmon provides the external health layer:

| Monitor Type | What It Covers | |---|---| | HTTP pipeline health | All pipeline components reachable | | OTel Collector direct | Central hub liveness | | Trace completeness | End-to-end trace delivery to Jaeger | | Frontend availability | Full stack user-facing health | | Per-service monitors | Individual microservice crash detection |

Get started free at vigilmon.online — your first OTel pipeline monitor is running in under two minutes.