Monitoring Promtail with Vigilmon

Promtail is the official log shipping agent for Grafana Loki. It discovers log files, systemd journal entries, and Kubernetes pod logs, then tails and ships them to Loki with consistent labels. When Promtail is healthy, your logs flow into Loki and you can query them in Grafana. When Promtail fails — whether the process crashes, loses its Loki connection, or silently drops a log target — your logs disappear without any visible error on the applications producing them. Vigilmon monitors Promtail's own HTTP endpoints so you know when log shipping breaks before you discover the gap hours later during an incident investigation.

Note: Grafana is deprecating Promtail in favor of Grafana Alloy. Promtail remains widely deployed in existing Loki installations and fully functional — new deployments should consider Alloy. This tutorial applies to all active Promtail deployments.

What You'll Set Up

• HTTP readiness monitor for Promtail's /ready endpoint (port 9080)
• Keyword check confirming Promtail is connected to Loki and ready to ship
• Metrics monitor checking active targets and sent-entries rate
• SSL certificate alerts for TLS-enabled Promtail deployments
• Cron heartbeat confirming the full Promtail → Loki pipeline is operational

Prerequisites

• Promtail installed and running (version 2.x or later)
• Promtail's built-in HTTP server active (default port 9080)
• Loki backend reachable from the Promtail host
• A free Vigilmon account

Step 1: Verify Promtail's HTTP Server is Active

Promtail runs a built-in HTTP server by default on port 9080. Confirm it is reachable:

curl -s http://localhost:9080/ready
# Expected output: Ready

curl -s http://localhost:9080/metrics | head -20

If the HTTP server is not responding, check your Promtail configuration (/etc/promtail/config.yml) for the server block:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

Restart Promtail after any config change:

sudo systemctl restart promtail
sudo systemctl status promtail

Step 2: Monitor the Promtail Readiness Endpoint

The /ready endpoint is Promtail's canonical health check. It returns 200 Ready\n when Promtail has successfully connected to Loki and is ready to ship logs, and 503 if not ready.

Add a Vigilmon monitor for this endpoint:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: http://YOUR_SERVER_IP:9080/ready
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Keyword monitoring, enable it and enter: Ready
7. Click Save.

The keyword check Ready is critical: Promtail can return a 200 status code even when transitioning states. The keyword confirms the full readiness condition — Promtail has established its connection to Loki and is actively accepting log targets.

A 503 from /ready means Promtail lost its Loki connection and no logs are being shipped.

Step 3: Monitor Promtail Metrics for Active Targets and Sent Entries

The /metrics endpoint exposes Prometheus-format metrics that reveal whether Promtail is actually shipping logs versus just being "ready":

Key metrics to watch:

• promtail_targets_active_total — number of active log targets; should be > 0
• promtail_sent_entries_total — cumulative log entries sent to Loki; should be increasing
• promtail_sent_bytes_total — bytes sent; should be nonzero

Add a Vigilmon HTTP monitor for the metrics endpoint:

1. Add Monitor → HTTP / HTTPS
2. URL: http://YOUR_SERVER_IP:9080/metrics
3. Check interval: 2 minutes
4. Expected status: 200
5. Keyword: promtail_targets_active_total
6. Save

The keyword promtail_targets_active_total confirms the metrics plugin is active. To spot a stalled pipeline, also check manually:

# Check active targets
curl -s http://localhost:9080/metrics | grep promtail_targets_active_total

# Check sent entries (should increase between checks)
curl -s http://localhost:9080/metrics | grep promtail_sent_entries_total

A promtail_targets_active_total of 0 when you expect log files to be tailed means all targets have been dropped — often caused by a permission issue reading log files or a misconfigured scrape config.

Step 4: Monitor Promtail Target Discovery Status

Promtail's /targets endpoint lists discovered log targets and their state. Targets in dropped or error state indicate specific log sources that are not being scraped.

Check target status manually:

curl -s http://localhost:9080/targets

Look for targets with state: "error" or entries in the dropped_targets section. A target is dropped when its labels don't match the configured relabel_configs, or when the log file path doesn't exist.

Add a Vigilmon monitor to confirm the targets endpoint is healthy:

1. Add Monitor → HTTP / HTTPS
2. URL: http://YOUR_SERVER_IP:9080/targets
3. Check interval: 5 minutes
4. Expected status: 200
5. Click Save (no keyword needed — a 200 confirms the endpoint is serving the target list)

For proactive alerting on dropped targets, pair Vigilmon with a lightweight script that queries /targets and pings a Vigilmon heartbeat only when no targets are in error state. This inverts the heartbeat pattern: silence means something is wrong.

Step 5: SSL Certificate Alerts for HTTPS Promtail Deployments

If Promtail is deployed behind an nginx or Caddy TLS proxy, or if TLS is configured in the Promtail server block:

server:
  http_listen_port: 9080
  http_tls_config:
    cert_file: /etc/promtail/certs/server.crt
    key_file: /etc/promtail/certs/server.key

Enable SSL monitoring on the readiness monitor from Step 2:

1. Open the /ready monitor in Vigilmon.
2. Update the URL to https://YOUR_SERVER_IP:9080/ready.
3. Scroll to the SSL section and enable Monitor SSL certificate.
4. Set Alert when certificate expires in less than 21 days.
5. Click Save.

An expired certificate on Promtail's HTTP server means the /ready endpoint becomes unreachable, but more critically it may also affect Promtail's outbound TLS connections to Loki if the same certificate is used for client auth.

Step 6: Heartbeat Monitoring for the Promtail → Loki Pipeline

The /ready endpoint confirms Promtail has connected to Loki, but it doesn't prove logs are actually flowing. A Vigilmon heartbeat monitors the full pipeline — Promtail must be up, connected to Loki, and actively writing to confirm the heartbeat.

Create the heartbeat:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_TOKEN

Ping the heartbeat from a health-check script:

Create /usr/local/bin/promtail-heartbeat.sh:

#!/bin/bash
# Only ping if Promtail is ready AND has sent at least 1 entry
READY=$(curl -fs http://localhost:9080/ready 2>/dev/null)
SENT=$(curl -s http://localhost:9080/metrics | grep 'promtail_sent_entries_total' | grep -v '#' | awk '{print $2}' | head -1)

if [ "$READY" = "Ready" ] && [ "${SENT:-0}" != "0" ]; then
  curl -fsS https://vigilmon.online/heartbeat/YOUR_TOKEN
fi

chmod +x /usr/local/bin/promtail-heartbeat.sh

Add a cron job to run it every 5 minutes:

# crontab -e
*/5 * * * * /usr/local/bin/promtail-heartbeat.sh

This heartbeat only pings if Promtail is both ready AND has sent log entries — a more complete signal than the readiness endpoint alone. If Promtail connects to Loki but immediately loses the connection, the /ready endpoint may return 200 briefly while the heartbeat script catches the true state.

Step 7: Monitor Loki Write Error Rate

Promtail exposes metrics about its outbound write requests to Loki:

# Check for Loki write errors
curl -s http://localhost:9080/metrics | grep promtail_request_duration_seconds_count

If promtail_request_duration_seconds_count is not increasing, Promtail has stopped sending requests to Loki. Combine this check with the heartbeat from Step 6 for comprehensive coverage.

For Kubernetes deployments, also check Promtail DaemonSet pod status:

kubectl get pods -n monitoring -l app=promtail

Any pods in CrashLoopBackOff or Error state means those nodes are not shipping logs.

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. Set Consecutive failures before alert to 2 on the /ready monitor — Loki restarts or brief network interruptions cause transient not-ready states.
3. Set Consecutive failures before alert to 1 on the heartbeat monitor — a missed heartbeat is a strong signal of sustained log shipping failure.
4. Tag monitors with log-pipeline to group Promtail and Loki monitors under one view.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP + keyword Ready | :9080/ready | Promtail process down, Loki connection lost | | HTTP + keyword promtail_targets_active_total | :9080/metrics | Metrics endpoint unavailable, plugin stopped | | HTTP (targets endpoint) | :9080/targets | Target discovery endpoint unavailable | | SSL certificate | :9080 (if TLS) | Expiring TLS certificate on Promtail HTTP server | | Cron heartbeat | Heartbeat URL | Full pipeline stall: logs not flowing to Loki |

When Promtail fails, log data vanishes silently — applications keep writing to disk, but nothing reaches Loki. With Vigilmon monitoring Promtail's readiness, metrics activity, and end-to-end heartbeat, you detect a broken log pipeline before it becomes a missing audit trail during a security incident or an empty Grafana panel during an outage.