tutorial

Monitoring Grafana Loki with Vigilmon

Grafana Loki is the log aggregation system designed for Kubernetes — here's how to monitor ingester health, distributor metrics, querier performance, chunk store latency, and retention policy with Vigilmon.

Grafana Loki is a log aggregation system purpose-built for cloud-native environments. Unlike traditional log systems that index everything, Loki only indexes metadata labels and compresses raw log chunks — dramatically lowering storage costs while keeping query performance fast via LogQL. But like any distributed system, Loki has multiple critical components: distributors, ingesters, queriers, compactors, and chunk stores. If any of these fail silently, your logs stop flowing without any obvious error to the users sending them. Vigilmon adds the uptime monitoring layer to ensure every part of your Loki deployment stays healthy.

What You'll Set Up

  • Vigilmon HTTP checks on Loki's ready, health, and ring endpoints
  • Ingester and distributor health monitoring via Prometheus metrics
  • Querier performance and chunk store latency tracking
  • Retention policy compactor health checks
  • Heartbeat monitoring for log-forwarding agents (Promtail, Alloy, Fluentd)

Prerequisites

  • Grafana Loki (2.9+) running via Docker or Kubernetes
  • An object store backend (S3, GCS, Azure Blob, or local filesystem for testing)
  • A free Vigilmon account
  • Optional: Grafana for metrics dashboards

Step 1: Deploy Loki with Monitoring Enabled

Deploy Loki using the official Helm chart in microservices mode for production:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm upgrade --install loki grafana/loki \
  --namespace loki \
  --create-namespace \
  --set loki.auth_enabled=false \
  --set loki.commonConfig.replication_factor=3 \
  --set loki.storage.type=s3 \
  --set loki.storage.s3.region=us-east-1 \
  --set loki.storage.s3.bucketnames=my-loki-chunks \
  --set monitoring.serviceMonitor.enabled=true \
  --set monitoring.selfMonitoring.enabled=true

Verify all components are running:

kubectl get pods -n loki
# NAME                           READY   STATUS    RESTARTS   AGE
# loki-backend-0                 2/2     Running   0          3m
# loki-read-6b8c9d9f7-xj2nk     1/1     Running   0          3m
# loki-write-5c7d8b8c6-kqvbr    1/1     Running   0          3m
# loki-gateway-7f9b6c9d8-p4mnc  1/1     Running   0          3m

Step 2: Check Loki Health Endpoints

Loki exposes several health endpoints:

# Overall readiness
curl http://loki-gateway.loki.svc.cluster.local/ready
# ready

# Build info (detailed component status)
curl http://loki-gateway.loki.svc.cluster.local/loki/api/v1/status/buildinfo
# {"version":"2.9.4","revision":"...","branch":"HEAD","goVersion":"..."}

# Ring status (shows all distributed component members)
curl http://loki-gateway.loki.svc.cluster.local/ring
# HTML page showing ring members and their status

Step 3: Add Vigilmon Health Checks

Configure Vigilmon monitors for Loki's critical endpoints:

3a. Gateway Readiness Check

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the URL: https://loki.yourdomain.com/ready
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Set Expected response body contains to ready.
  7. Click Save.

3b. Ingester Ring Health

The Loki ingester ring shows which ingester instances are active:

curl http://loki.yourdomain.com/ingester/ring | grep -c "ACTIVE"
# 3  (should match your replication factor)

Add a Vigilmon monitor:

  • URL: https://loki.yourdomain.com/ingester/ring
  • Expected HTTP status: 200
  • Check interval: 2 minutes

3c. Distributor Ring Health

curl http://loki.yourdomain.com/distributor/ring

Add a separate monitor for the distributor ring — it's the entry point for all incoming logs and failures here mean log ingestion stops entirely.

Step 4: Prometheus Metrics Monitoring

Loki exposes rich Prometheus metrics. Configure Prometheus to scrape all components:

# prometheus.yml
scrape_configs:
  - job_name: 'loki'
    static_configs:
      - targets:
          - 'loki-write.loki.svc.cluster.local:3100'
          - 'loki-read.loki.svc.cluster.local:3100'
          - 'loki-backend.loki.svc.cluster.local:3100'
    metrics_path: /metrics
    scrape_interval: 15s

Critical metrics dashboard:

| Metric | Description | Alert threshold | |--------|-------------|-----------------| | loki_ingester_streams_created_total | New log streams per second | sudden drop to 0 | | loki_distributor_lines_received_total | Log lines received by distributor | rate drop > 50% | | loki_ingester_chunk_utilization | Chunk fill efficiency (0.0–1.0) | < 0.3 (wasting memory) | | loki_request_duration_seconds | Query latency histogram | p99 > 5s | | loki_chunk_store_index_entries_per_chunk | Chunks written to object store | should be > 0 | | loki_compactor_runs_completed_total | Compactor runs for retention | should increment daily | | loki_ingester_flush_queue_length | Pending chunks waiting to flush | > 500 indicates pressure |

Step 5: Ingester Health Deep Dive

The ingester is Loki's most memory-sensitive component — it holds active chunks in memory before flushing to object storage. Monitor it closely:

# Check ingester-specific metrics
curl -s http://loki-write.loki.svc.cluster.local:3100/metrics \
  | grep loki_ingester | head -30

Write an ingester health check script:

#!/usr/bin/env python3
# loki_ingester_check.py
import requests
import sys

LOKI_URL = "http://loki-write.loki.svc.cluster.local:3100"
VIGILMON_HEARTBEAT = "https://vigilmon.online/api/v1/heartbeat/<YOUR_TOKEN>"

FLUSH_QUEUE_THRESHOLD = 500
MEMORY_CHUNKS_THRESHOLD = 10000

def parse_metric(metrics_text, metric_name):
    for line in metrics_text.splitlines():
        if line.startswith(metric_name + " "):
            return float(line.split(" ")[1])
    return None

def main():
    try:
        resp = requests.get(f"{LOKI_URL}/metrics", timeout=10)
        resp.raise_for_status()
        metrics = resp.text

        flush_queue = parse_metric(metrics, "loki_ingester_flush_queue_length")
        memory_chunks = parse_metric(metrics, "loki_ingester_memory_chunks")

        issues = []
        if flush_queue is not None and flush_queue > FLUSH_QUEUE_THRESHOLD:
            issues.append(f"flush_queue={flush_queue:.0f} > {FLUSH_QUEUE_THRESHOLD}")

        if memory_chunks is not None and memory_chunks > MEMORY_CHUNKS_THRESHOLD:
            issues.append(f"memory_chunks={memory_chunks:.0f} > {MEMORY_CHUNKS_THRESHOLD}")

        if issues:
            print("WARNING: " + ", ".join(issues))
            sys.exit(1)

        requests.get(VIGILMON_HEARTBEAT, timeout=5)
        print(f"OK: flush_queue={flush_queue:.0f}, memory_chunks={memory_chunks:.0f}")

    except requests.RequestException as e:
        print(f"ERROR: {e}")
        sys.exit(1)

if __name__ == "__main__":
    main()

Deploy as a Kubernetes CronJob running every 5 minutes.

Step 6: Querier Performance Monitoring

The Loki querier handles LogQL queries from Grafana and the Loki API. Slow queries can cascade into Grafana dashboard timeouts:

# Send a test query and measure latency
time curl -s -G 'http://loki.yourdomain.com/loki/api/v1/query_range' \
  --data-urlencode 'query={app="probe-app"}' \
  --data-urlencode 'start=1h' \
  --data-urlencode 'end=now' \
  --data-urlencode 'limit=10'

Add a Vigilmon monitor that checks query response time:

#!/usr/bin/env python3
# loki_query_check.py
import requests
import time
import sys
import datetime

LOKI_URL = "https://loki.yourdomain.com"
VIGILMON_HEARTBEAT = "https://vigilmon.online/api/v1/heartbeat/<YOUR_TOKEN>"
QUERY_LATENCY_THRESHOLD_MS = 3000

def main():
    now = datetime.datetime.utcnow()
    start = now - datetime.timedelta(minutes=5)

    params = {
        "query": '{app="probe-app"}',
        "start": start.strftime("%Y-%m-%dT%H:%M:%SZ"),
        "end": now.strftime("%Y-%m-%dT%H:%M:%SZ"),
        "limit": "1"
    }

    t0 = time.monotonic()
    try:
        resp = requests.get(
            f"{LOKI_URL}/loki/api/v1/query_range",
            params=params,
            timeout=10
        )
        elapsed_ms = (time.monotonic() - t0) * 1000

        if resp.status_code != 200:
            print(f"ERROR: query returned {resp.status_code}")
            sys.exit(1)

        if elapsed_ms > QUERY_LATENCY_THRESHOLD_MS:
            print(f"SLOW QUERY: {elapsed_ms:.0f}ms > {QUERY_LATENCY_THRESHOLD_MS}ms threshold")
            sys.exit(1)

        requests.get(VIGILMON_HEARTBEAT, timeout=5)
        print(f"OK: query latency {elapsed_ms:.0f}ms")

    except requests.RequestException as e:
        print(f"ERROR: {e}")
        sys.exit(1)

if __name__ == "__main__":
    main()

Step 7: Chunk Store and Retention Monitoring

Loki's chunk store (S3, GCS, Azure Blob, etc.) is where all logs ultimately live. Monitor it via the compactor component:

# Check compactor status
curl http://loki-backend.loki.svc.cluster.local:3100/compactor/ring

Track retention policy execution:

# loki-config.yaml snippet — enable retention
compactor:
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  working_directory: /data/loki/compactor

limits_config:
  retention_period: 30d  # global default
  per_tenant_override_config: /etc/loki/overrides.yaml

Monitor retention metrics:

| Metric | Description | |--------|-------------| | loki_compactor_runs_completed_total | Successful compaction runs | | loki_compactor_runs_failed_total | Failed compaction runs (alert immediately) | | loki_boltdb_shipper_compact_tables_duration_seconds | Time spent compacting index tables | | loki_store_series_total | Active series count in store |

Add a Vigilmon monitor for the compactor endpoint:

  • URL: https://loki.yourdomain.com/compactor/ring
  • Expected HTTP status: 200
  • Check interval: 5 minutes

Step 8: Log Agent Heartbeat Monitoring

Promtail, Grafana Alloy, or Fluentd agents forward logs to Loki. If an agent stops, logs stop — but Loki stays healthy, making silent data loss easy to miss. Add heartbeat checks for each agent.

Configure Promtail to write a heartbeat log line and validate it with Vigilmon:

# promtail-config.yaml
scrape_configs:
  - job_name: heartbeat
    static_configs:
      - targets: [localhost]
        labels:
          job: heartbeat
          __path__: /var/log/heartbeat.log

Write a heartbeat log every minute:

# /etc/cron.d/loki-heartbeat
* * * * * root echo "loki-heartbeat-$(hostname)-$(date +%s)" >> /var/log/heartbeat.log

Validate via the Loki query API:

# Check that heartbeat logs arrived in the last 2 minutes
curl -s -G 'https://loki.yourdomain.com/loki/api/v1/query' \
  --data-urlencode 'query={job="heartbeat"} |= "loki-heartbeat"' \
  --data-urlencode 'time=now' \
  | jq '.data.result | length'
# Should return 1 or more

Set up a Vigilmon Cron / Heartbeat monitor that expects a ping at least once per 5 minutes, and trigger it from a wrapper script that validates the log arrived.

Step 9: SSL Certificate Monitoring

Every Promtail and Alloy agent connects to Loki via TLS. An expired SSL certificate causes all agents to fail TLS validation and stop shipping logs — silently.

  1. In Vigilmon, click Add Monitor → SSL Certificate.
  2. Set Domain to loki.yourdomain.com.
  3. Set Alert when expiry is within to 30 days.
  4. Set additional reminders at 14 days, 7 days, and 1 day.
  5. Click Save.

Alerting Configuration

Configure Vigilmon alert channels in Settings → Notifications:

  • PagerDuty: immediate page when /ready returns non-200 (all log ingestion stopped)
  • Slack: #observability-alerts for querier latency threshold breaches
  • Email: daily digest of compactor run status to the platform team
  • Webhooks: trigger auto-scaling of ingester pods on flush queue pressure

Use alert labels to distinguish between ingester failures (critical — logs being lost) and querier slowness (high — dashboards affected but logs preserved).

Troubleshooting

Loki returning "too many outstanding requests" The querier queue is full. Increase querier.max_outstanding_requests_per_tenant:

limits_config:
  max_outstanding_requests_per_tenant: 4096

Ingester OOMKilled High memory pressure from too many active streams. Check:

kubectl top pods -n loki -l app.kubernetes.io/component=write

Reduce ingester.max_streams_per_user or scale up ingester replicas.

Compactor not deleting old data

kubectl logs -n loki statefulset/loki-backend | grep compactor

Verify retention_enabled: true is set and S3/GCS permissions allow object deletion.

Distributor dropping log lines

# Check for rate limit errors
curl -s http://loki-write:3100/metrics | grep loki_distributor_ingestion_rate_bytes

The tenant may be exceeding ingestion_rate_mb limits. Increase limits or scale distributors.

Summary

You now have comprehensive Grafana Loki monitoring with Vigilmon:

  • Gateway readiness and ring health checks every 1-2 minutes
  • Ingester health monitoring with flush queue and memory chunk alerts
  • Querier performance probes with 3-second latency threshold
  • Compactor and retention monitoring to ensure logs are being pruned correctly
  • Log agent heartbeats to detect silent Promtail or Alloy failures
  • SSL certificate monitoring to prevent TLS-induced ingestion failures

Loki is the nervous system of your observability stack — if it fails silently, you lose the very logs you need to debug other failures. Vigilmon ensures that never happens.

Ready to monitor your Loki deployment? Create a free Vigilmon account and add your first log aggregation health check in minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles