Monitoring Thanos with Vigilmon

Thanos extends Prometheus with unlimited long-term metric retention, a global query view across multiple Prometheus instances, and high-availability deduplication — all backed by cheap object storage like S3 or GCS. It does this through a set of loosely coupled components: the Sidecar uploads blocks from Prometheus and exposes a StoreAPI, the Store Gateway serves historical blocks from object storage, the Querier fans out PromQL across all StoreAPI endpoints, the Compactor merges and downsamples remote blocks, and the Ruler evaluates recording and alerting rules against the global view. Each component can silently degrade — a stalled Compactor increases query latency over time, and a failed Store Gateway makes months of data inaccessible. Vigilmon gives you the external monitoring layer to catch Thanos component failures before they corrupt your long-term observability data.

What You'll Set Up

• HTTP monitor for Thanos Querier (global read path)
• Sidecar readiness check (write path from Prometheus to object storage)
• Store Gateway health monitoring
• Compactor readiness check
• Ruler health monitoring
• SSL certificate expiry monitoring and alert configuration

Prerequisites

• Thanos components deployed via Helm (bitnami/thanos) or as individual binaries
• Thanos HTTP endpoints accessible at stable URLs (e.g. https://thanos.example.com)
• Port 10902 (HTTP) open for health probes per component
• A free Vigilmon account

Step 1: Monitor the Querier (Global Read Path)

The Thanos Querier is the entry point for all PromQL queries in a Thanos deployment. It fans out queries across all registered StoreAPI endpoints (Sidecars, Store Gateways, Rulers) and deduplicates results. If the Querier is unhealthy, your Grafana dashboards that use Thanos as a data source go blank.

Thanos exposes a readiness probe at /-/ready:

GET https://thanos-querier.example.com/-/ready

A healthy Querier returns HTTP 200 with body Thanos Querier is ready. Configure Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter URL: https://thanos-querier.example.com/-/ready.
4. Set Expected HTTP status to 200.
5. Set Expected body contains to ready.
6. Set Check interval to 2 minutes.
7. Click Save.

You can also check the liveness probe at /-/healthy to confirm the process is alive even before it completes initialization:

GET https://thanos-querier.example.com/-/healthy

Add a second monitor for /-/healthy with a 1 minute check interval as an early-warning signal.

Step 2: Monitor the Querier Store Discovery

The Querier needs at least one healthy StoreAPI endpoint (Sidecar, Store Gateway, or Ruler) to serve queries. Thanos exposes a /api/v1/stores endpoint that lists all discovered StoreAPI endpoints and their health:

GET https://thanos-querier.example.com/api/v1/stores

A healthy response returns HTTP 200 with a JSON object listing all connected stores. Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://thanos-querier.example.com/api/v1/stores.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to "type".
5. Set Check interval to 5 minutes.
6. Click Save.

An empty store list ({} or {"thanos-store":[]}") while the Querier itself is healthy means all Sidecars and Store Gateways are unreachable — queries will succeed on currently-ingested data only, silently returning empty results for historical data.

Step 3: Monitor the Sidecar (Prometheus Upload Path)

The Thanos Sidecar runs alongside each Prometheus instance. It uploads completed 2-hour blocks from Prometheus's local TSDB to object storage and exposes a StoreAPI endpoint over gRPC so the Querier can access Prometheus's most recent data. A failed Sidecar stops block uploads — object storage falls behind, and once the Prometheus TSDB retention window passes, that data is permanently lost.

GET https://thanos-sidecar.example.com/-/ready

Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://thanos-sidecar.example.com/-/ready.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to ready.
5. Set Check interval to 3 minutes.
6. Click Save.

If you run multiple Prometheus/Sidecar pairs (for HA), add a separate monitor for each Sidecar instance. A failed Sidecar on a replica is not immediately catastrophic, but two failed Sidecars means block uploads stop entirely.

Step 4: Monitor the Store Gateway

The Thanos Store Gateway serves historical metric blocks stored in object storage to the Querier. It downloads block index files locally and answers gRPC StoreAPI queries against them. A failed Store Gateway makes all historical data inaccessible — current data from Sidecars is still available, so dashboards appear to work but long-range queries (7 days, 30 days) silently return incomplete results.

GET https://thanos-store.example.com/-/ready

Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://thanos-store.example.com/-/ready.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to ready.
5. Set Check interval to 3 minutes.
6. Click Save.

Store Gateway startup involves downloading block index files from object storage — for large deployments this can take several minutes. If your Store Gateway frequently restarts (e.g., due to OOM), the readiness check will show degraded state during each restart. Monitor the /-/ready endpoint (not /-/healthy) to distinguish an initializing Store Gateway from a crashed one.

Step 5: Monitor the Compactor

The Thanos Compactor runs as a singleton (it does not support horizontal scaling). It merges small blocks uploaded by Sidecars into larger blocks, applies retention policies, and generates downsampled blocks for fast long-range queries. A stalled Compactor causes:

• Block count grows without bound → Store Gateway startup time increases
• Downsampled blocks are not generated → long-range queries degrade significantly
• Retention policies are not enforced → unexpected object storage costs

GET https://thanos-compactor.example.com/-/ready

Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://thanos-compactor.example.com/-/ready.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to ready.
5. Set Check interval to 10 minutes.
6. Click Save.

The Compactor is intentionally designed to run as a Job or CronJob rather than a long-running Deployment — it completes a compaction cycle and optionally sleeps. If you run it as a CronJob, use a Vigilmon HTTP monitor against a dedicated health port exposed during the compaction run, or monitor via a synthetic canary that checks whether downsampled blocks exist in your S3 bucket.

Step 6: Monitor the Ruler

The Thanos Ruler evaluates Prometheus recording rules and alerting rules against the global Thanos query view — not just a single Prometheus instance. This allows alerting rules that span multiple clusters or regions. A failed Ruler means:

• Recording rules don't compute global pre-aggregated metrics
• Cross-cluster alerting rules don't fire
• Alerts are not forwarded to Alertmanager

GET https://thanos-ruler.example.com/-/ready

Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://thanos-ruler.example.com/-/ready.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to ready.
5. Set Check interval to 3 minutes.
6. Click Save.

Check rule health via the Ruler's API:

GET https://thanos-ruler.example.com/api/v1/rules

A healthy response returns a 200 with a groups array. An empty groups array means the Ruler loaded but found no rules — typically a configuration error.

Step 7: SSL Certificate Monitoring and Alert Configuration

Add SSL expiry monitoring for each Thanos component's HTTPS endpoint:

1. Add Monitor → SSL Certificate.
2. URL: https://thanos-querier.example.com.
3. Set Alert when certificate expires in less than 30 days.
4. Click Save.

Repeat for thanos-store.example.com and thanos-ruler.example.com.

Configure alert channels:

1. Go to Alert Channels → add Slack, PagerDuty, or email.
2. For the Querier /-/ready check: Consecutive failures before alert = 1 — a failed Querier means zero PromQL queries work.
3. For Sidecar and Store Gateway checks: 2 consecutive failures — brief restarts during rolling upgrades are normal.
4. For Compactor: 3 consecutive failures — it is expected to be idle between compaction cycles.
5. Add Maintenance windows for Thanos Helm chart upgrades; a rolling restart of Querier and Store Gateway components takes 5–10 minutes.

Going Further

• Object storage connectivity: All Thanos components depend on object storage (S3, GCS, Azure Blob). Add a lightweight HEAD request to a known Thanos block prefix in your bucket to distinguish object storage outages from Thanos component failures. If the bucket is unreachable, Sidecars stop uploading, Store Gateways stop serving historical data, and Compactor stalls — but the Querier's /ready check may still pass.
• Block freshness monitoring: The Querier exposes a /api/v1/query?query=max(thanos_objstore_bucket_last_successful_upload_time) endpoint you can poll with Vigilmon. If the last successful upload timestamp is more than 3 hours old, block uploads have stalled.
• Query-frontend layer: For large deployments, the Thanos Query Frontend sits in front of the Querier and caches and splits long-range queries. Add a /-/ready monitor for the Query Frontend separately — a failed Query Frontend causes cache misses and query fan-out storms on the Querier.
• Receive component: If you use Thanos Receive (instead of Sidecar) to accept Prometheus remote_write traffic, monitor its /ready endpoint and its remote_write receiver port (default: 19291) with a TCP monitor.

With Vigilmon monitoring Thanos's Querier, Sidecars, Store Gateway, Compactor, and Ruler, you have full external visibility into every layer of your long-term Prometheus storage pipeline — so any component failure is caught before developers notice missing or incomplete historical metrics in their dashboards.