Monitoring Grafana Tempo with Vigilmon: Web UI, Ready Endpoint, Trace Ingestion & SSL Alerts

Grafana Tempo is the distributed tracing backend for the Grafana observability stack — receiving trace spans from OpenTelemetry collectors, Jaeger agents, and Zipkin forwarders, and serving them to Grafana's Explore view for request tracing. When Tempo's distributor goes down, traces stop being written and every engineer trying to trace a slow request sees nothing. When the query frontend becomes unavailable, Tempo's TraceQL queries fail silently in Grafana. Vigilmon gives you external visibility into Tempo's health from outside your observability infrastructure.

What You'll Build

• A monitor on Tempo's /ready endpoint to detect ingestion failures
• A web UI availability check to catch reverse proxy and networking failures
• A trace query API health check that validates the query frontend
• SSL certificate monitoring for your Tempo domain
• An alerting setup that distinguishes ingestion failures from query failures

Prerequisites

• Grafana Tempo running (v2.x) in single-binary or microservices mode
• A domain or IP for Tempo (e.g., https://tempo.example.com or http://host:3200)
• A free account at vigilmon.online

Step 1: Verify Tempo's Ready Endpoint

Tempo's HTTP server runs on port 3200 by default and exposes a /ready endpoint that confirms the instance is fully initialized and ready to accept trace data:

curl http://localhost:3200/ready

A healthy Tempo returns HTTP 200 with the plain-text body:

ready

Tempo returns HTTP 503 with a descriptive message when:

• The component is still starting up and loading its ring membership
• The backend store (S3, GCS, Azure Blob) is unreachable
• A microservices component lost its connection to memberlist

# Example not-ready response
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8

Not ready: waiting to be an active ring member

Port mapping: If Tempo runs behind a reverse proxy (nginx, Traefik, Caddy), use the proxied HTTPS URL. If accessing Tempo directly, port 3200 is the HTTP API port. The gRPC receiver ports (4317 for OTLP, 14250 for Jaeger) are separate from the HTTP management port.

Step 2: Create the /ready Monitor in Vigilmon

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://tempo.example.com/ready.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ready (the exact plain-text body of a healthy Tempo instance).
7. Label: Tempo ready.
8. Click Save.

Alert threshold: Set to alert after 1 consecutive failure. When Tempo is not ready, the OTLP and Jaeger receivers stop accepting spans — trace data is lost for the duration of the outage with no replay mechanism. Every minute of missed traces is permanent data loss.

This monitor catches:

• Tempo process crash or OOM kill (common when ingesting high-volume trace workloads)
• Backend storage failures (S3 permission errors, GCS quota limits)
• Ring membership failures in microservices deployments
• Configuration errors after a reload that cause Tempo to exit

Step 3: Monitor the Tempo Web UI

Tempo's web UI at the root path serves the status page and basic Tempo information. Monitoring it catches reverse proxy issues and networking failures that may not affect the /ready endpoint on the direct port:

curl -o /dev/null -s -w "%{http_code}" https://tempo.example.com/

A running Tempo returns HTTP 200.

1. Add Monitor → HTTP.
2. URL: https://tempo.example.com/.
3. Check interval: 5 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Label: Tempo web UI.
7. Click Save.

Step 4: Monitor the Trace Query API

Tempo serves TraceQL queries and trace lookups via its HTTP API. The /api/search/tags endpoint lists available tag keys — a lightweight call that exercises the query path without requiring specific trace IDs:

curl "https://tempo.example.com/api/search/tags"

A healthy Tempo query frontend returns:

{
  "tagNames": ["service.name", "http.method", "http.url", "span.kind", ...]
}

1. Add Monitor → HTTP.
2. URL: https://tempo.example.com/api/search/tags.
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: tagNames (always present when the query frontend is healthy).
7. Label: Tempo query API.
8. Click Save.

When the query API monitor fires but /ready is still green, you have a query frontend failure separate from the ingestion pipeline — common in microservices mode where the querier component is degraded while the distributor/ingester continues accepting spans.

Alternative probe: If your Tempo has tenant authentication, use /api/v2/search/tags with an X-Scope-OrgID header. Or use the /status endpoint (https://tempo.example.com/status) which doesn't require auth and returns Tempo's component ring status as JSON.

Step 5: Monitor the Trace Ingestion Path

Verify that Tempo's OTLP HTTP receiver is reachable. The OTLP HTTP endpoint at /v1/traces accepts span batches:

curl -o /dev/null -s -w "%{http_code}" \
  -X POST https://tempo.example.com/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'

An available OTLP receiver returns HTTP 200 with an empty response for an empty span batch — confirming the ingestion path is routable without writing actual trace data.

1. Add Monitor → HTTP (POST).
2. URL: https://tempo.example.com/v1/traces.
3. Method: POST.
4. Headers: Content-Type: application/json.
5. Body: {"resourceSpans":[]}.
6. Expected status: 200.
7. Check interval: 2 minutes.
8. Label: Tempo OTLP ingestion.
9. Click Save.

Port note: The OTLP HTTP receiver may run on port 4318 rather than 3200 depending on your Tempo configuration. Check your tempo.yaml under receivers.otlp.protocols.http for the correct port. If using a reverse proxy, the proxy path should route /v1/traces to the correct backend port.

Step 6: Monitor SSL Certificates

Tempo's TLS certificate must stay valid for:

• OpenTelemetry SDK clients (applications) to send spans over HTTPS
• Grafana's Tempo data source connection to work
• Tempo-to-Tempo federation in multi-cluster setups

1. Add Monitor → SSL Certificate.
2. Domain: tempo.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again at: 14 days, 7 days, 3 days.
5. Click Save.

An expired Tempo SSL certificate causes all instrumented applications to fail TLS and stop sending traces. Since most SDKs don't log this loudly, trace data disappears silently — you may not notice until a developer complains that traces are missing during an investigation.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Response | |---|---|---| | /ready | Non-200 or ready keyword missing | Tempo not ready; check backend storage, ring membership | | Web UI | Non-200 or timeout | Reverse proxy issue; Tempo may still be accepting spans | | Query API (/api/search/tags) | Non-200 or keyword missing | Query frontend down; Grafana trace lookups failing | | OTLP ingestion | Non-200 | Ingestion path broken; applications can't send spans | | SSL certificate | < 30 days to expiry | Renew cert; SDK clients will fail TLS validation |

Priority: The /ready monitor is your highest-priority alert — a not-ready Tempo means trace data is being silently dropped. Send this directly to your on-call channel.

Common Tempo Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Tempo OOM kill (high-cardinality spans) | /ready and web UI fire within 60 s | | S3 / GCS storage backend unreachable | /ready returns 503; ingestion stops | | Distributor ring membership lost | /ready returns 503; restart restores ring | | Querier crash (microservices) | Query API monitor fires; /ready stays green | | OTLP gRPC receiver crash (HTTP receiver fine) | HTTP ingestion passes; gRPC spans are dropped silently | | SSL certificate expires | SSL monitor alerts; SDK clients stop sending spans | | Tempo upgrade migration fails | /ready may loop; query API fails | | TraceQL query timeouts under load | Query API response time increases; timeout-based alerts fire | | DNS misconfiguration | All monitors fire simultaneously | | Backend object store bucket permissions revoked | /ready returns not-ready after in-flight spans drain |

Tempo sits at the end of your distributed tracing pipeline — and like Loki, it creates an observability paradox: if Tempo is down, you lose the tool you'd use to trace what went wrong with Tempo. Vigilmon breaks that dependency by monitoring Tempo's /ready endpoint, query frontend, ingestion path, and SSL certificate from outside your observability stack. You'll know Tempo is dropping traces before your engineers notice that the Grafana Explore trace search is returning nothing.

Start monitoring Tempo for free — register at vigilmon.online.