Monitoring Grafana Loki with Vigilmon

Grafana Loki gives you Prometheus-style log aggregation without the full-text index cost — logs are stored compressed and indexed only by labels. But that lightweight architecture spans several interdependent components: distributors, ingesters, queriers, a compactor, and an optional ruler. When any component silently degrades, queries stall and alerts stop firing. Vigilmon keeps watch over every Loki endpoint so you know before your oncall engineers start complaining about missing logs.

What You'll Set Up

• HTTP availability monitor for the Loki server API
• Ingestion (write) API health check
• Query (read) API response time monitoring
• Compactor and ruler service availability checks
• Promtail agent heartbeat for scrape health
• Distributed component status monitoring

Prerequisites

• Grafana Loki 2.8+ running (single-binary or microservices mode)
• Loki HTTP API accessible on port 3100
• A free Vigilmon account

Step 1: Monitor the Loki HTTP API (Port 3100)

Loki exposes its main API on port 3100. The /ready endpoint returns a 200 OK when all components are ready and a 5xx when the service is still initialising or degraded.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter: http://your-loki-host:3100/ready
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Loki also exposes a Prometheus-compatible metrics endpoint at /metrics. You can use this in Grafana dashboards, but for availability alerting the /ready probe is the right signal.

Step 2: Monitor the Ingestion (Push) API

If Promtail or any other log shipper cannot reach Loki's push endpoint, logs are silently dropped. Monitor the write path directly:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-loki-host:3100/loki/api/v1/push
3. Set Method to POST.
4. Set Request body to a minimal valid push payload:

{"streams": [{"stream": {"job": "vigilmon-probe"}, "values": []}]}

5. Set Expected HTTP status to 204.
6. Set Check interval to 2 minutes.
7. Click Save.

A 204 No Content response means Loki accepted the push. If the ingester pool is overloaded you'll get a 429 or 500 instead.

Step 3: Monitor the Query (Read) API

Loki's query frontend sits in front of ingesters and the object store. A slow or broken querier means Grafana dashboards time out even when logs are still being ingested.

1. Add an HTTP / HTTPS monitor.
2. URL: http://your-loki-host:3100/loki/api/v1/labels
3. Set Expected HTTP status to 200.
4. Set Check interval to 1 minute.
5. Under Advanced, enable Response time alert and set the threshold to 5000 ms.
6. Click Save.

The /loki/api/v1/labels endpoint exercises the full read path (querier → ingester → object store) without requiring a specific log stream to exist.

Step 4: Monitor Distributed Components (gRPC Port 9095)

In microservices mode, Loki's distributor, ingester, and querier each expose a gRPC port on 9095 for inter-component communication. Add a TCP monitor for each component:

1. Click Add Monitor → TCP Port.
2. Host: IP or hostname of the distributor node.
3. Port: 9095.
4. Check interval: 1 minute.
5. Click Save.

Repeat for each ingester and querier node. A TCP failure on port 9095 before the HTTP API shows a problem means a component is still starting or has crashed.

Step 5: Monitor the Compactor Health

The Loki compactor merges chunks in the object store to reduce storage costs. A stalled compactor causes the store to grow unboundedly but does not affect query availability immediately — making it easy to miss.

1. Add an HTTP / HTTPS monitor.
2. URL: http://your-loki-host:3100/compactor/ring
3. Set Expected HTTP status to 200.
4. Set Check interval to 5 minutes.
5. Click Save.

The /compactor/ring endpoint shows the compactor's ring membership. A 404 means the compactor is not running (check your Loki config's compactor block). A 500 indicates a ring health problem.

Step 6: Monitor the Ruler Alerting Service

Loki's ruler evaluates LogQL alerting rules and fires alerts through an Alertmanager. If the ruler is down, your log-based alerts stop firing silently.

1. Add an HTTP / HTTPS monitor.
2. URL: http://your-loki-host:3100/ruler/ring
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

If you are not using Loki ruler, skip this step. If the endpoint returns 404, the ruler component is not enabled in your Loki configuration.

Step 7: Promtail Agent Heartbeat

Promtail scrapes log files and ships them to Loki. If Promtail crashes or its scrape targets dry up, no new logs appear in Loki — but Loki itself stays healthy. Use a cron heartbeat to verify Promtail is still running.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Expected interval to 2 minutes.
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).
4. Add a scrape config entry to your promtail.yaml that pings the heartbeat after each successful scrape cycle:

# promtail.yaml snippet
scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*.log
    pipeline_stages:
      - match:
          selector: '{job="varlogs"}'
          stages:
            - output:
                source: message

Then wrap Promtail in a supervisor script that pings Vigilmon on each successful cycle:

#!/bin/bash
promtail -config.file=/etc/promtail/config.yaml &
PROMTAIL_PID=$!

while kill -0 $PROMTAIL_PID 2>/dev/null; do
  sleep 120
  curl -s https://vigilmon.online/heartbeat/abc123
done

If Promtail crashes and stops pinging, Vigilmon alerts after the 2-minute window expires.

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. For the push API monitor, set Consecutive failures before alert to 2 — transient network blips occasionally cause a 500.
3. For the compactor monitor, set Consecutive failures before alert to 3 — the compactor ring can take a minute to stabilise after a restart.
4. For the Loki /ready endpoint, set Consecutive failures before alert to 1 — this endpoint only returns 200 when fully initialised.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Loki readiness | :3100/ready | Full service failure, init stuck | | Ingestion API | :3100/loki/api/v1/push | Write path degradation | | Query API | :3100/loki/api/v1/labels | Read path failure, slow querier | | Compactor ring | :3100/compactor/ring | Storage cost runaway | | Ruler ring | :3100/ruler/ring | Log-based alerts silenced | | Distributed gRPC | :9095 per node | Component crash (microservices mode) | | Promtail heartbeat | Heartbeat URL | Agent crash, scrape stall |

Loki's cost advantage comes from its distributed, label-based design — but distributed systems fail in distributed ways. With Vigilmon watching every component endpoint and the Promtail agent heartbeat, you get early warning on every failure mode before the missing logs become a production incident.