Monitoring Grafana Tempo with Vigilmon

Grafana Tempo is a distributed tracing backend designed to store and query traces at petabyte scale using only object storage — no indexing required. It ingests spans from OpenTelemetry, Jaeger, Zipkin, and Zipkin via separate receiver ports, fans writes through a distributor-ingester pipeline, and serves queries through a querier that reads from both the in-memory ingester blocks and long-term object storage. Each layer can fail independently: a crashed ingester loses recent spans, a failed compactor balloons storage costs, and an unavailable querier breaks trace lookups without touching the write path. Vigilmon gives you the external monitoring layer to detect Tempo component failures before your developers notice missing traces in Grafana.

What You'll Set Up

• HTTP monitor for Tempo's write path (distributor readiness)
• Querier health check via the Tempo API
• Ingester ring status monitoring
• Compactor readiness check
• Metrics-generator health monitoring
• SSL certificate expiry monitoring and alert configuration

Prerequisites

• Grafana Tempo deployed via Helm (grafana/tempo-distributed) or as a single binary
• The Tempo API accessible at a stable URL (e.g. https://tempo.example.com)
• Port 3200 or your configured HTTP listen port open for health probes
• A free Vigilmon account

Step 1: Monitor the Write Path (Distributor Readiness)

Tempo's distributor accepts all incoming spans from trace collectors — OpenTelemetry Collector, Grafana Agent, or Alloy — and routes them to ingesters using a consistent hash ring. If the distributor is down, no new traces are written and your distributed tracing data silently disappears.

Tempo exposes a readiness probe at /ready:

GET https://tempo.example.com/ready

A healthy distributor returns HTTP 200 with body ready. Configure Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter URL: https://tempo.example.com/ready.
4. Set Expected HTTP status to 200.
5. Set Expected body contains to ready.
6. Set Check interval to 2 minutes.
7. Click Save.

In the tempo-distributed Helm chart, the distributor runs as its own Deployment. If you expose it through an ingress, use that URL. For internal Kubernetes environments, add a Vigilmon agent probe inside the cluster and monitor http://tempo-distributor.tempo.svc:3200/ready.

Step 2: Monitor the Trace Query API

Tempo's query API lets developers search for traces by service name, span attributes, or trace ID. A healthy query path confirms that the querier is reaching both live ingesters and long-term object storage blocks.

Check the Tempo API search endpoint:

GET https://tempo.example.com/api/search?limit=1&start=<now-300>&end=<now>

A healthy response returns HTTP 200 with a JSON body containing a traces array (which may be empty but must be present):

{
  "traces": [],
  "metrics": { "inspectedTraces": 0 }
}

Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://tempo.example.com/api/search?limit=1.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to "traces".
5. Set Check interval to 3 minutes.
6. Click Save.

This check exercises the full read path: querier → ingester gRPC → object storage. If this check fails while the /ready check passes, the write path is healthy but trace search is broken.

Step 3: Monitor the Ingester Ring

Tempo ingesters buffer recent spans in memory (default: last 15 minutes) before flushing blocks to object storage. The ingester ring page shows the state of all replicas:

GET https://tempo.example.com/ingester/ring

This returns an HTML page with a table of ingester states. Configure Vigilmon to assert the ring page loads with healthy replicas:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://tempo.example.com/ingester/ring.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to ACTIVE.
5. Set Check interval to 5 minutes.
6. Click Save.

With Tempo's default replication factor of 3, losing a single ingester causes a replication-factor warning but no data loss. Losing two causes write failures for affected ring segments. Seeing ACTIVE for all ingesters is the fastest external signal that the ingester ring is fully healthy.

Step 4: Monitor the Compactor

Tempo's compactor merges small trace blocks flushed by ingesters into larger, more storage-efficient blocks, and enforces retention policies. A stalled compactor causes:

• Object storage block count grows unbounded, increasing listing and read overhead
• Retention policies are not applied, producing unexpected storage costs
• Bloom filters for tag-based search become stale

The compactor exposes its own readiness endpoint in distributed deployments:

GET https://tempo-compactor.example.com/ready

Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://tempo-compactor.example.com/ready (or https://tempo.example.com/compactor/ready in single-binary mode).
3. Set Expected HTTP status to 200.
4. Set Expected body contains to ready.
5. Set Check interval to 10 minutes.
6. Click Save.

In tempo-distributed, the compactor runs as a Deployment. If the compactor pod enters CrashLoopBackOff or its object storage credentials expire, it stops processing silently — Vigilmon's readiness check surfaces this before your storage bill surprises you.

Step 5: Monitor the Metrics-Generator

Tempo's optional metrics-generator component derives RED metrics (rate, errors, duration) from ingested spans and writes them to a Prometheus-compatible remote_write target. This powers Grafana's Service Graph and Span Metrics features. If the metrics-generator is down, service graphs stop updating and RED dashboard panels show stale or missing data.

GET https://tempo.example.com/metrics-generator/ring

Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://tempo.example.com/metrics-generator/ring.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to ACTIVE.
5. Set Check interval to 5 minutes.
6. Click Save.

If the metrics-generator ring page returns 404, it means the component is not deployed — which is valid if you don't use Span Metrics or Service Graphs. Only add this monitor if your Tempo configuration includes metrics_generator.enabled: true.

Step 6: Monitor the Tempo Status API

Tempo exposes a /status endpoint that reports component versions, build info, and configuration state. This is a lightweight check that confirms the entire Tempo process is alive and serving admin traffic:

GET https://tempo.example.com/status

A healthy response returns HTTP 200 with a plain-text status page or JSON containing the Tempo version. Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://tempo.example.com/status.
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

Pair this monitor with the /ready check: if /status returns 200 but /ready returns a non-ready body, Tempo is up but still initializing — for example, waiting for the ingester ring to reach quorum after a rolling restart.

Step 7: SSL Certificate Monitoring and Alert Configuration

Add SSL expiry monitoring for Tempo's API endpoint:

1. Add Monitor → SSL Certificate.
2. URL: https://tempo.example.com.
3. Set Alert when certificate expires in less than 30 days.
4. Click Save.

Configure alert channels:

1. Go to Alert Channels → add Slack, PagerDuty, or email.
2. For the distributor /ready and query API checks: set Consecutive failures before alert to 1 — these are critical write/read path monitors.
3. For ingester ring, compactor, and metrics-generator checks: set to 2 consecutive failures — brief ring state transitions during rolling restarts are normal.
4. Add Maintenance windows for Tempo Helm chart upgrades; a full rolling restart takes 5–15 minutes.

Going Further

• Receiver endpoint health: Tempo accepts spans on multiple ports — OTLP gRPC (4317), OTLP HTTP (4318), Jaeger gRPC (14250), Zipkin HTTP (9411). Add Vigilmon TCP monitors for each receiver port to confirm your trace collectors can reach Tempo's receivers even if the HTTP API is unavailable.
• Object storage connectivity: All of Tempo's components — ingesters, compactor, querier — depend on object storage (S3, GCS, Azure Blob). Add a lightweight HEAD request to a known Tempo object prefix in your bucket to distinguish object storage failures from Tempo component failures.
• Trace pipeline E2E health: Deploy a synthetic span generator (a simple cron job that sends one test span every 5 minutes) and use Vigilmon to poll the Tempo search API for that synthetic trace. A missing synthetic trace is the earliest signal that the full span pipeline is broken end-to-end.
• Query-frontend load: In large Tempo deployments, the query-frontend shards large trace searches across multiple queriers. Monitor its /ready endpoint separately — a failed query-frontend forces all reads through direct querier routes, increasing latency under load.

With Vigilmon monitoring Tempo's distributor, ingesters, querier, compactor, and metrics-generator, you have full external visibility into each layer of the distributed tracing pipeline — so any component failure is caught before your developers notice missing traces in their Grafana dashboards.