Monitoring Uptrace (Distributed Tracing) with Vigilmon

Uptrace is an open-source APM and distributed tracing platform built on top of OpenTelemetry and ClickHouse. It receives spans from your services via OTLP (OpenTelemetry Protocol), stores them in ClickHouse for fast analytics queries, and provides a UI for browsing traces, detecting slow operations, and correlating errors across services. When Uptrace's ingestion endpoint goes down — due to ClickHouse running out of disk space, the gRPC listener crashing, or OOM on the Uptrace process itself — all your instrumented services lose their trace sink silently. The services keep running, the instrumentation keeps generating spans, but nothing is recorded. You discover the gap hours later when you try to debug an incident. Vigilmon gives you external monitoring for Uptrace availability, OTLP ingestion health, ClickHouse storage, span throughput, and API query responsiveness so you know when your observability backend needs attention — not when you need it most.

What You'll Set Up

• Uptrace process and HTTP health endpoint monitoring
• OTLP gRPC ingestion endpoint availability
• ClickHouse backend storage and query health
• Span ingestion throughput tracking
• Uptrace query API responsiveness

Prerequisites

• Uptrace 1.6+ running (self-hosted, Docker, or Kubernetes)
• Access to the Uptrace host or cluster
• ClickHouse credentials (used by Uptrace as storage backend)
• A free Vigilmon account

Step 1: Monitor Uptrace HTTP Health Endpoint

Uptrace exposes a built-in /health endpoint that confirms the web server is responding and its internal subsystems are reachable. This is the fastest signal that Uptrace is completely down:

1. In Vigilmon, click Add Monitor → HTTP(S).
2. Set the URL to http://your-uptrace-host:14318/health (adjust the port if you changed the default).
3. Set Check interval to 1 minute.
4. Set Expected status code to 200.
5. Enable Keyword check and set it to ok — Uptrace returns {"status":"ok"} when healthy.

If you're running Uptrace behind a reverse proxy, use its public URL. The health endpoint should not require authentication. If it returns anything other than 200 with ok, Uptrace's internal health check has failed — either ClickHouse is unreachable or a required subsystem has crashed.

For TLS-terminated deployments, add a separate SSL certificate expiry monitor:

1. Click Add Monitor → SSL Certificate.
2. Set the domain to your Uptrace hostname.
3. Set Alert days before expiry to 14.

Step 2: Monitor OTLP gRPC Ingestion Endpoint

OTLP over gRPC (port 14317 by default) is how all your instrumented services send spans to Uptrace. A healthy HTTP endpoint does not guarantee the gRPC listener is working — the web server and the OTLP receiver are independent processes in Uptrace's architecture. Monitor the gRPC port directly:

1. In Vigilmon, click Add Monitor → TCP Port.
2. Set Host to your Uptrace host.
3. Set Port to 14317.
4. Set Check interval to 1 minute.

For deeper validation, send a test span from a cron heartbeat using the OpenTelemetry CLI or a minimal OTLP client:

#!/bin/bash
# /usr/local/bin/uptrace-otlp-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
UPTRACE_HOST="your-uptrace-host"
OTLP_PORT="14317"
TIMEOUT=10

# Test gRPC connectivity with grpc_health_probe or a simple TCP check
if timeout "$TIMEOUT" bash -c "echo >/dev/tcp/${UPTRACE_HOST}/${OTLP_PORT}" 2>/dev/null; then
  echo "OTLP gRPC port $OTLP_PORT is reachable on $UPTRACE_HOST"
  curl -s "$HEARTBEAT_URL"
else
  echo "OTLP gRPC port $OTLP_PORT unreachable on $UPTRACE_HOST"
  exit 1
fi

Also monitor the OTLP HTTP endpoint (port 14318) if your services use OTLP/HTTP instead of gRPC:

# Check OTLP HTTP endpoint returns expected response
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time 10 \
  -X POST http://${UPTRACE_HOST}:14318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{}')

# Uptrace returns 400 for empty/invalid payloads but 200 or 400 both mean the endpoint is alive
if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "400" ]; then
  echo "OTLP HTTP endpoint alive (HTTP $HTTP_CODE)"
  curl -s "$HEARTBEAT_URL"
else
  echo "OTLP HTTP endpoint unexpected response: HTTP $HTTP_CODE"
  exit 1
fi

Create a Vigilmon Cron Heartbeat with a 2 minute expected interval for this script. Run it from a machine outside the Uptrace host to test actual network reachability.

Step 3: Monitor ClickHouse Backend Storage

Uptrace uses ClickHouse as its primary data store for spans, metrics, and logs. If ClickHouse runs out of disk space, the Uptrace ingestion pipeline silently drops incoming spans — no error is returned to clients, the data is simply lost. Monitor ClickHouse storage health directly:

#!/bin/bash
# /usr/local/bin/uptrace-clickhouse-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
CH_HOST="your-clickhouse-host"
CH_PORT="9000"
CH_USER="default"
CH_PASSWORD="your-clickhouse-password"
CH_DATABASE="uptrace"
DISK_THRESHOLD_PCT=80
TIMEOUT=15

# Check ClickHouse is responsive
CH_PING=$(clickhouse-client \
  --host "$CH_HOST" \
  --port "$CH_PORT" \
  --user "$CH_USER" \
  --password "$CH_PASSWORD" \
  --query "SELECT 1" \
  --connect_timeout "$TIMEOUT" \
  2>&1)

if [ "$CH_PING" != "1" ]; then
  echo "ClickHouse unresponsive: $CH_PING"
  exit 1
fi

# Check disk usage for ClickHouse data directory
DISK_USED=$(clickhouse-client \
  --host "$CH_HOST" \
  --port "$CH_PORT" \
  --user "$CH_USER" \
  --password "$CH_PASSWORD" \
  --query "SELECT round(free_space / total_space * 100) FROM system.disks WHERE name = 'default'" \
  2>/dev/null || echo "0")

FREE_PCT="${DISK_USED:-0}"
USED_PCT=$(( 100 - FREE_PCT ))

if [ "$USED_PCT" -ge "$DISK_THRESHOLD_PCT" ]; then
  echo "ClickHouse disk at ${USED_PCT}% (threshold: ${DISK_THRESHOLD_PCT}%)"
  exit 1
fi

# Check Uptrace tables exist and are queryable
TABLE_CHECK=$(clickhouse-client \
  --host "$CH_HOST" \
  --port "$CH_PORT" \
  --user "$CH_USER" \
  --password "$CH_PASSWORD" \
  --query "SELECT count() FROM system.tables WHERE database = '${CH_DATABASE}' AND name LIKE '%span%'" \
  2>/dev/null || echo "0")

if [ "$TABLE_CHECK" = "0" ]; then
  echo "Uptrace span tables missing from ClickHouse database ${CH_DATABASE}"
  exit 1
fi

echo "ClickHouse OK: disk ${USED_PCT}% used, Uptrace tables present"
curl -s "$HEARTBEAT_URL"

Set the Vigilmon heartbeat to 5 minutes. ClickHouse disk fill is the most common cause of silent span loss in Uptrace deployments — set your threshold conservatively at 80% to give yourself time to add storage or purge old data before ingestion fails.

Step 4: Track Span Ingestion Throughput

A sudden drop in span ingestion rate is one of the most actionable signals Uptrace can provide. If your services are generating 50,000 spans per minute and that number drops to zero, either all your services crashed simultaneously (unlikely) or Uptrace's ingestion pipeline broke (far more likely). Monitor throughput via the Uptrace API or ClickHouse query:

#!/bin/bash
# /usr/local/bin/uptrace-throughput-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
CH_HOST="your-clickhouse-host"
CH_PORT="9000"
CH_USER="default"
CH_PASSWORD="your-clickhouse-password"
MIN_SPANS_PER_5MIN=100    # Alert if fewer than 100 spans in the last 5 minutes
TIMEOUT=15

# Count spans ingested in the last 5 minutes
RECENT_SPANS=$(clickhouse-client \
  --host "$CH_HOST" \
  --port "$CH_PORT" \
  --user "$CH_USER" \
  --password "$CH_PASSWORD" \
  --query "
    SELECT count()
    FROM uptrace.spans_index
    WHERE time >= now() - INTERVAL 5 MINUTE
  " \
  --connect_timeout "$TIMEOUT" \
  2>/dev/null || echo "0")

RECENT_SPANS="${RECENT_SPANS:-0}"

if [ "$RECENT_SPANS" -ge "$MIN_SPANS_PER_5MIN" ]; then
  echo "Span throughput OK: ${RECENT_SPANS} spans in last 5 minutes"
  curl -s "$HEARTBEAT_URL"
else
  echo "Span throughput low: ${RECENT_SPANS} spans in last 5 minutes (min: ${MIN_SPANS_PER_5MIN})"
  # Don't exit 1 if this is a development environment that may be idle
  # Adjust MIN_SPANS_PER_5MIN to 0 for dev, higher for production
  exit 1
fi

Run this check every 5 minutes with a Vigilmon heartbeat at 8 minute expected interval. Calibrate MIN_SPANS_PER_5MIN against your normal traffic baseline — a production system with many instrumented services might generate millions of spans per 5 minutes; adjust the threshold accordingly. For development environments, set it to 0 to skip the throughput check and only verify Uptrace is alive.

Step 5: Monitor Uptrace Query API Responsiveness

Uptrace's UI and API allow developers to search traces, view service maps, and analyze latency distributions. When ClickHouse is under heavy query load or the Uptrace application server is overloaded, query responses slow dramatically — developers experience timeouts when trying to debug incidents. Monitor query response time:

#!/bin/bash
# /usr/local/bin/uptrace-query-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
UPTRACE_URL="https://your-uptrace-host"
UPTRACE_TOKEN="your-uptrace-api-token"   # From Uptrace Settings > API tokens
MAX_RESPONSE_MS=5000   # Alert if queries take more than 5 seconds
TIMEOUT=15

# Time a query to the Uptrace API
START_TIME=$(date +%s%3N)

HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time "$TIMEOUT" \
  -H "Authorization: Bearer $UPTRACE_TOKEN" \
  "${UPTRACE_URL}/api/v1/projects")

END_TIME=$(date +%s%3N)
ELAPSED=$(( END_TIME - START_TIME ))

if [ "$HTTP_CODE" != "200" ]; then
  echo "Uptrace API returned HTTP $HTTP_CODE (expected 200)"
  exit 1
fi

if [ "$ELAPSED" -gt "$MAX_RESPONSE_MS" ]; then
  echo "Uptrace API slow: ${ELAPSED}ms (max: ${MAX_RESPONSE_MS}ms)"
  exit 1
fi

echo "Uptrace API OK: HTTP $HTTP_CODE in ${ELAPSED}ms"
curl -s "$HEARTBEAT_URL"

For more realistic query-load testing, execute a trace search against a known project:

# Query recent traces for a specific service
START_TIME=$(date +%s%3N)

HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time 20 \
  -H "Authorization: Bearer $UPTRACE_TOKEN" \
  -G "${UPTRACE_URL}/api/v1/tracing/${YOUR_PROJECT_ID}/spans" \
  --data-urlencode "time_gte=$(date -u -d '5 minutes ago' '+%Y-%m-%dT%H:%M:%SZ')" \
  --data-urlencode "time_lt=$(date -u '+%Y-%m-%dT%H:%M:%SZ')" \
  --data-urlencode "per_page=10")

END_TIME=$(date +%s%3N)
ELAPSED=$(( END_TIME - START_TIME ))

if [ "$HTTP_CODE" = "200" ] && [ "$ELAPSED" -lt 8000 ]; then
  echo "Trace query OK: ${ELAPSED}ms"
  curl -s "$HEARTBEAT_URL"
else
  echo "Trace query failed or slow: HTTP $HTTP_CODE in ${ELAPSED}ms"
  exit 1
fi

Set the Vigilmon heartbeat to 5 minutes. Query API degradation often precedes full ClickHouse instability — catching slow queries early gives you time to reduce ClickHouse load before the backend falls over.

Step 6: Set Up Alert Channels

Route Uptrace alerts to your platform team with appropriate urgency levels:

1. Go to Alert Channels in Vigilmon.
2. Add a Slack webhook to #observability-alerts — the platform team must know when their tracing backend is down.
3. Add PagerDuty for the HTTP health and OTLP ingestion monitors — a dead ingestion endpoint means live incident data is being lost right now.
4. Add email notification for storage and throughput monitors — these indicate impending failure rather than current outage.
5. Set Consecutive failures before alert to 2 for the throughput monitor to avoid false alerts during brief traffic lulls.

Add a maintenance window during Uptrace or ClickHouse upgrades:

# Pause monitors during planned maintenance
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "UPTRACE_MONITOR_ID",
    "duration_minutes": 30,
    "reason": "Uptrace version upgrade"
  }'

# Upgrade Uptrace (example for Docker Compose)
docker compose pull uptrace
docker compose up -d uptrace

# Verify health before ending maintenance window
curl -s http://your-uptrace-host:14318/health

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health | /health endpoint | Uptrace process crash, subsystem failure | | TCP port check | OTLP gRPC port 14317 | Ingestion receiver down, network partition | | Cron heartbeat (ClickHouse) | CH ping + disk + tables | Storage full, backend unresponsive, missing tables | | Cron heartbeat (throughput) | Spans ingested per 5 min | Silent span loss, pipeline break | | Cron heartbeat (query API) | API response time | ClickHouse overload, UI degradation | | SSL certificate | Uptrace TLS cert | Expiry causing SDK connection failures |

The most dangerous failure mode in Uptrace is silent span loss: the OTLP endpoint accepts connections and returns success, but spans are dropped at the ClickHouse write layer because storage is full or a table schema migration failed. External throughput monitoring via ClickHouse queries is the only way to detect this without Uptrace-internal instrumentation. Vigilmon's cron heartbeats give you that external view — you see the span count drop before the gap in your traces becomes an incident-investigation blocker.

Start monitoring for free at vigilmon.online.