tutorial

Monitoring Grafana Mimir with Vigilmon

Grafana Mimir is the scalable long-term storage backend for Prometheus — but if Mimir's ingesters, compactor, or object storage connections fail, your metrics pipeline silently drops data. Here's how to monitor every Mimir component with Vigilmon.

Grafana Mimir solves the hardest problem in Prometheus at scale: single-node storage limits. Where Prometheus stores metrics locally on disk with a 15-day default retention, Mimir distributes ingestion, storage, and querying across a fleet of microservices backed by object storage (S3, GCS, or Azure Blob), giving you unlimited retention and horizontal scalability.

The tradeoff is operational complexity. Mimir has ten distinct microservice roles. Any one of them failing can cause silent metric loss, query errors, or alert rule evaluation gaps — and because Mimir is your metrics backend, it can't alert on its own failures. Vigilmon provides the external vantage point: continuous health checks on every Mimir component so you know before your engineers do.

What You'll Set Up

  • HTTP health monitors for Mimir's main API and each microservice role
  • Heartbeat monitors for ingester, compactor, and ruler health
  • Object storage connectivity monitoring
  • Alert thresholds for data loss, query timeouts, and ring degradation

Prerequisites

  • Grafana Mimir deployed (monolithic mode on port 8080, or microservice mode with per-component ports)
  • Object storage backend configured (S3/GCS/Azure)
  • A free Vigilmon account

Step 1: Monitor the Mimir HTTP API

Mimir exposes a /ready endpoint on port 8080 that returns 200 OK once all required components have started and joined the ring. It's the canonical liveness check for the entire instance in monolithic mode, and for each microservice pod in distributed mode.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your Mimir URL: http://mimir.yourdomain.com:8080/ready.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

For Kubernetes deployments where Mimir is not externally exposed, add an ingress or use a NodePort service for this health endpoint only, or run the Vigilmon probe from inside the cluster.

Mimir also exposes /metrics for its own internal Prometheus metrics. Add a second monitor if you want to confirm the metrics endpoint itself is serving:

http://mimir.yourdomain.com:8080/metrics

Step 2: Monitor Distributor Health (Remote Write Ingestion)

The Distributor is the entry point for all remote_write traffic from Prometheus. If the distributor is down or shedding requests, Prometheus stacks up unsent WAL data and eventually loses metrics after its WAL retention window expires.

In microservice deployments, the Distributor runs separately on its own service. Add a dedicated monitor:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://mimir-distributor.yourdomain.com:8080/ready.
  3. Check interval: 1 minute.
  4. Expected status: 200.

For single-instance (monolithic) deployments, the /ready check from Step 1 covers the distributor. To verify the remote_write ingestion path directly, add a heartbeat from a write probe script:

#!/bin/bash
# Confirms the distributor is accepting remote_write requests
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DISTRIBUTOR_ID"

STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -X POST http://mimir.yourdomain.com:8080/api/v1/push \
  -H "X-Scope-OrgID: probe" \
  --data-binary @/opt/monitoring/test_metrics.pb \
  -H "Content-Type: application/x-protobuf")

if [ "$STATUS" = "204" ] || [ "$STATUS" = "200" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

Schedule every 5 minutes.

Step 3: Monitor Ingester Health

Ingesters buffer the most recent metric data in memory before flushing to object storage. An ingester OOM kill or crash causes data loss for the series it owned — the WAL prevents total loss, but replay takes time and in-memory data since the last flush is gone.

Add a heartbeat that checks the ingester ring status via Mimir's ring API:

#!/usr/bin/env python3
import requests
import sys

MIMIR_API = "http://mimir.yourdomain.com:8080"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_INGESTER_ID"
MIN_HEALTHY = 3  # adjust to match your replication factor

resp = requests.get(
    f"{MIMIR_API}/ingester/ring",
    headers={"Accept": "application/json"},
    timeout=10,
)
resp.raise_for_status()
ring = resp.json()

healthy = sum(
    1 for member in ring.get("shards", [])
    if member.get("state") == "ACTIVE"
)

if healthy >= MIN_HEALTHY:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print(f"Only {healthy} healthy ingesters (need {MIN_HEALTHY})")
    sys.exit(1)

Schedule every 2 minutes. Adjust MIN_HEALTHY to match your replication factor (commonly 3).

Step 4: Monitor Object Storage Connectivity

Mimir's durability depends entirely on object storage. Ingesters flush blocks to S3/GCS/Azure; the Store Gateway reads historical blocks back for queries. An object storage disruption causes ingesters to accumulate data in memory until they OOM, and query results become incomplete as historical blocks become inaccessible.

Add a heartbeat that probes object storage reachability:

#!/usr/bin/env python3
"""
Verifies Mimir can reach object storage by listing the root prefix.
Requires the same credentials Mimir uses (via environment or IAM role).
"""
import boto3
import requests
import sys

BUCKET = "your-mimir-bucket"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_OBJSTORE_ID"

try:
    s3 = boto3.client("s3")
    s3.list_objects_v2(Bucket=BUCKET, MaxKeys=1)
    requests.get(HEARTBEAT_URL, timeout=5)
except Exception as e:
    print(f"Object storage probe failed: {e}", file=sys.stderr)
    sys.exit(1)

For GCS, replace with google.cloud.storage.Client(). For Azure, use azure.storage.blob.BlobServiceClient.

Schedule every 5 minutes. Object storage outages are rare but catastrophic — early warning lets you switch Mimir to a degraded-but-alive mode before data is lost.

Step 5: Monitor Query Frontend Health

The Query Frontend caches PromQL query results and splits large range queries into parallel sub-queries. A degraded query frontend causes dashboard timeouts and alert evaluation failures — users see blank Grafana panels and alerts stop firing.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://mimir-query-frontend.yourdomain.com:8080/ready.
  3. Check interval: 1 minute.
  4. Expected status: 200.

For latency-sensitive alerting, add a probe that executes a real PromQL query:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_QF_ID"
MIMIR_QUERY="http://mimir.yourdomain.com:8080/prometheus"
MAX_LATENCY_MS=5000

START=$(date +%s%3N)
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -G "$MIMIR_QUERY/api/v1/query" \
  --data-urlencode "query=vector(1)" \
  -H "X-Scope-OrgID: probe" \
  --max-time 10)
END=$(date +%s%3N)
LATENCY=$((END - START))

if [ "$STATUS" = "200" ] && [ "$LATENCY" -lt "$MAX_LATENCY_MS" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
else
  echo "Query probe: status=$STATUS latency=${LATENCY}ms"
fi

Schedule every 5 minutes. A query latency spike above 5 seconds typically indicates compaction falling behind or Store Gateway block load failures.

Step 6: Monitor Compactor Health

The Compactor merges small block files into larger ones in object storage, keeping query performance stable as data ages. A stalled compactor doesn't cause immediate data loss, but query performance degrades over weeks as object storage fills with tiny unmerged blocks.

Add a heartbeat that checks the compactor ring:

#!/usr/bin/env python3
import requests
import sys

MIMIR_API = "http://mimir.yourdomain.com:8080"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_COMPACTOR_ID"

resp = requests.get(
    f"{MIMIR_API}/compactor/ring",
    headers={"Accept": "application/json"},
    timeout=10,
)
resp.raise_for_status()
ring = resp.json()

healthy_compactors = sum(
    1 for m in ring.get("shards", []) if m.get("state") == "ACTIVE"
)

if healthy_compactors >= 1:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print("No healthy compactors in ring", file=sys.stderr)
    sys.exit(1)

Schedule every 10 minutes. Compactor failures are often the first sign of object storage permission issues — the compactor can't write merged blocks back.

Step 7: Monitor Store Gateway and Ruler Health

Store Gateway serves historical blocks from object storage for queries beyond the ingester retention window. Ruler evaluates Prometheus recording rules and alerting rules on a schedule.

Add HTTP monitors for both:

http://mimir-store-gateway.yourdomain.com:8080/ready
http://mimir-ruler.yourdomain.com:8080/ready

Both should return 200. Set check interval to 1 minute.

For the Ruler, add a heartbeat confirming rule evaluation is occurring:

#!/usr/bin/env python3
"""Checks that the ruler reports at least one rule group successfully evaluated."""
import requests
import sys

MIMIR_API = "http://mimir.yourdomain.com:8080"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_RULER_ID"

resp = requests.get(
    f"{MIMIR_API}/prometheus/api/v1/rules",
    headers={"X-Scope-OrgID": "default"},
    timeout=10,
)
resp.raise_for_status()
groups = resp.json().get("data", {}).get("groups", [])

if groups:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print("Ruler reports zero rule groups — rules may not be evaluating")
    sys.exit(1)

Schedule every 5 minutes. A Ruler with zero rule groups means your alerting rules have silently stopped firing — the most dangerous failure mode in any observability stack.

Step 8: Configure Alert Routing

Mimir failures have very different urgency levels:

| Monitor | Urgency | Alert Action | |---------|---------|-------------| | Distributor down | Critical | Page on-call immediately | | Ingester count below RF | Critical | Page on-call immediately | | Query frontend down | High | Alert within 5 minutes | | Object storage connectivity | Critical | Page on-call immediately | | Compactor stalled | Medium | Slack alert to ops team | | Store Gateway down | High | Alert within 5 minutes | | Ruler — zero rules | Critical | Page on-call immediately |

In Vigilmon:

  1. Go to Settings → Alerts.
  2. Add your PagerDuty, Slack, or email channels.
  3. Assign escalation policies per monitor using the urgency table above.

For Mimir, the most dangerous silent failure is the Ruler evaluating zero rules — your entire alerting system stops working without any visible error. Make this your highest-priority Vigilmon alert.

Key Metrics Summary

| Component | Monitor Type | Check Interval | Alert Threshold | |-----------|-------------|---------------|----------------| | Mimir API /ready | HTTP | 1 min | 1 failure | | Distributor | HTTP | 1 min | 1 failure | | Ingester ring health | Heartbeat | 2 min | 1 miss | | Object storage | Heartbeat | 5 min | 2 misses | | Query frontend | HTTP | 1 min | 1 failure | | Query latency probe | Heartbeat | 5 min | 1 miss | | Compactor ring | Heartbeat | 10 min | 3 misses | | Store Gateway | HTTP | 1 min | 1 failure | | Ruler | HTTP | 1 min | 1 failure | | Ruler rule groups | Heartbeat | 5 min | 1 miss |

Conclusion

Grafana Mimir is the foundation of your long-term metrics storage — every Grafana dashboard, every Prometheus alert rule, and every historical trend analysis depends on it being healthy. The irony of a metrics system is that it can't monitor itself.

Vigilmon fills that gap with external HTTP and TCP checks for process-level health, plus cron heartbeats for logical health (ingesters in the ring, ruler evaluating rules, object storage reachable). Together they give you the early warning you need before a Mimir component failure becomes a metrics outage your engineers discover by noticing their dashboards are blank.

Get started at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles