tutorial

Monitoring Filebeat with Vigilmon

Filebeat is the Elastic Stack's log shipping workhorse — but a failed Filebeat silently drains your ELK pipeline. Here's how to monitor Filebeat's HTTP monitoring endpoint, harvester status, and output ACK rate with Vigilmon.

Filebeat is the lightweight log shipper at the front of every Elastic Stack deployment. It tails log files, reads from Docker and Kubernetes, applies 80+ parsing modules for Nginx, Apache, MySQL, and more, then ships events to Elasticsearch, Logstash, or Kafka with at-least-once delivery guarantees. It maintains a registry file to resume exactly where it left off after a restart. When Filebeat is healthy, your ELK pipeline is fed. When Filebeat crashes or its output is rejected, logs accumulate on disk while your Elasticsearch indices go stale — silently. Vigilmon monitors Filebeat's own HTTP monitoring endpoint so you catch a broken log pipeline before it becomes a missing audit trail.

What You'll Set Up

  • HTTP uptime monitor for Filebeat's monitoring endpoint (port 5066)
  • Keyword check confirming the Filebeat process identity
  • Harvester status monitor checking active log tailers
  • Output ACK rate monitor detecting dropped or rejected events
  • SSL certificate alerts for HTTPS Filebeat monitoring endpoints
  • Cron heartbeat confirming end-to-end event delivery to Elasticsearch

Prerequisites

  • Filebeat 7.x or 8.x installed and running as a daemon or DaemonSet
  • Filebeat HTTP monitoring endpoint enabled (port 5066)
  • Elasticsearch or Logstash output configured and reachable
  • A free Vigilmon account

Step 1: Enable Filebeat's HTTP Monitoring Endpoint

Filebeat includes a built-in HTTP monitoring server that must be explicitly enabled in filebeat.yml:

http.enabled: true
http.host: localhost
http.port: 5066

After adding this configuration, restart Filebeat:

sudo systemctl restart filebeat
sudo systemctl status filebeat

Verify the endpoint is responding:

curl -s http://localhost:5066/

Expected output:

{"name":"filebeat","uuid":"a1b2c3d4-...","version":"8.x.x","beat":{"name":"filebeat","hostname":"...","version":"8.x.x"}}

Also check the stats endpoint:

curl -s http://localhost:5066/stats | python3 -m json.tool | head -40

If either endpoint returns an error, confirm http.enabled: true is in your filebeat.yml and there are no YAML indentation errors.

Step 2: Monitor the Filebeat Root Endpoint

The root endpoint at / returns basic Filebeat identity information and is the simplest availability check.

Add a Vigilmon monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the URL: http://YOUR_SERVER_IP:5066/
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Keyword monitoring, enable it and enter: filebeat
  7. Click Save.

The keyword filebeat matches the name field in the JSON response, confirming this is a live Filebeat process and not an unrelated HTTP server on the same port. A 200 without the keyword would indicate an unexpected response format.

Step 3: Monitor Filebeat Harvester Status

Harvesters are the goroutines that tail individual log files. Active harvesters confirm Filebeat is actively reading logs. Zero active harvesters when logs should be flowing indicates a configuration problem — wrong file paths, permission errors, or an empty glob pattern.

Add a Vigilmon monitor for the stats endpoint:

  1. Add Monitor → HTTP / HTTPS
  2. URL: http://YOUR_SERVER_IP:5066/stats
  3. Check interval: 2 minutes
  4. Expected status: 200
  5. Under Keyword monitoring, enter: "active":
  6. Click Save

The keyword "active": appears in the filebeat.harvester section of the stats response:

{
  "filebeat": {
    "harvester": {
      "open_files": 3,
      "running": 3,
      "skipped": 0,
      "started": 5,
      "closed": 2
    }
  }
}

To check active harvesters directly:

curl -s http://localhost:5066/stats | python3 -c "
import json, sys
stats = json.load(sys.stdin)
harvesters = stats.get('filebeat', {}).get('harvester', {})
print(f'Active harvesters: {harvesters.get(\"running\", 0)}')
"

If running is 0 when you expect Filebeat to be tailing files, check:

# Confirm Filebeat can read the target log files
ls -la /var/log/nginx/access.log
sudo -u filebeat cat /var/log/nginx/access.log | head -1

Step 4: Monitor Filebeat Output ACK Rate

Filebeat's libbeat layer tracks events through the pipeline: acknowledged (delivered to Elasticsearch), failed, and dropped. A rising dropped count means Elasticsearch is rejecting events — the logs are being read but not indexed.

Check the output stats:

curl -s http://localhost:5066/stats | python3 -c "
import json, sys
stats = json.load(sys.stdin)
events = stats.get('libbeat', {}).get('output', {}).get('events', {})
print(f'ACKed: {events.get(\"acked\", 0)}')
print(f'Failed: {events.get(\"failed\", 0)}')
print(f'Dropped: {events.get(\"dropped\", 0)}')
"

Add a Vigilmon keyword monitor to detect dropped events:

  1. Add Monitor → HTTP / HTTPS
  2. URL: http://YOUR_SERVER_IP:5066/stats
  3. Check interval: 5 minutes
  4. Under Keyword monitoring, enter: "dropped":0
  5. Click Save

This keyword matches "dropped":0 in the libbeat output stats. If Elasticsearch starts rejecting events (due to mapping conflicts, index policy issues, or authentication errors), the dropped count rises above zero and the keyword no longer matches, triggering a Vigilmon alert.

For the full pipeline view, also monitor Elasticsearch directly — a Filebeat alert paired with an Elasticsearch health alert narrows root-cause analysis immediately.

Step 5: SSL Certificate Alerts for HTTPS Filebeat Monitoring Endpoints

If Filebeat's HTTP monitoring endpoint is secured with TLS:

http.enabled: true
http.host: localhost
http.port: 5066
http.ssl.enabled: true
http.ssl.certificate: /etc/filebeat/certs/filebeat.crt
http.ssl.key: /etc/filebeat/certs/filebeat.key

Update the root endpoint monitor from Step 2:

  1. Open the monitor in Vigilmon.
  2. Update the URL to https://YOUR_SERVER_IP:5066/.
  3. Scroll to the SSL section.
  4. Enable Monitor SSL certificate.
  5. Set Alert when certificate expires in less than 21 days.
  6. Click Save.

Most Filebeat deployments restrict port 5066 to localhost or an internal network, but any TLS endpoint — including those behind an nginx reverse proxy — should have certificate expiry monitoring.

Step 6: Heartbeat Monitoring for End-to-End Event Delivery

Filebeat's /stats endpoint shows that events are being processed, but not that they are successfully reaching Elasticsearch. A Vigilmon heartbeat monitors the full delivery chain.

Create the heartbeat:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 5 minutes.
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_TOKEN

Option A: Ping from a health-check script (recommended)

Create /usr/local/bin/filebeat-heartbeat.sh:

#!/bin/bash
# Check Filebeat is running and ACKing events
ACKED=$(curl -s http://localhost:5066/stats | python3 -c "
import json, sys
try:
  stats = json.load(sys.stdin)
  print(stats.get('libbeat', {}).get('output', {}).get('events', {}).get('acked', 0))
except:
  print(0)
" 2>/dev/null)

DROPPED=$(curl -s http://localhost:5066/stats | python3 -c "
import json, sys
try:
  stats = json.load(sys.stdin)
  print(stats.get('libbeat', {}).get('output', {}).get('events', {}).get('dropped', 0))
except:
  print(1)
" 2>/dev/null)

# Only ping heartbeat if events are being ACKed and nothing is dropped
if [ "${ACKED:-0}" -gt "0" ] && [ "${DROPPED:-1}" -eq "0" ]; then
  curl -fsS https://vigilmon.online/heartbeat/YOUR_TOKEN
fi

chmod +x /usr/local/bin/filebeat-heartbeat.sh

Add to cron:

# crontab -e
*/5 * * * * /usr/local/bin/filebeat-heartbeat.sh

Option B: Use Filebeat's output processor

In filebeat.yml, add a script processor that pings Vigilmon after Elasticsearch ACKs a batch:

processors:
  - script:
      lang: javascript
      id: heartbeat_ping
      source: >
        function process(event) {
          // This runs per-event; use rate limiting in production
          return event;
        }

For most deployments, Option A (external cron script) is simpler and more reliable than an in-process approach.

Step 7: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
  2. Set Consecutive failures before alert to 2 on the root endpoint monitor — Filebeat reloads on config changes and has a brief startup period.
  3. Set Consecutive failures before alert to 1 on the dropped-events monitor — any dropped events warrant immediate investigation.
  4. Set Consecutive failures before alert to 1 on the heartbeat monitor — a missed heartbeat means events have stopped reaching Elasticsearch.

For Kubernetes DaemonSet deployments, create one Vigilmon monitor per node if per-node visibility is required, or rely on Kubernetes pod health checks for cluster-level detection and use Vigilmon for end-to-end pipeline verification.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP + keyword filebeat | :5066/ | Filebeat process down or unexpected process | | HTTP + keyword "active": | :5066/stats | Harvester status unavailable | | HTTP + keyword "dropped":0 | :5066/stats | Events rejected by Elasticsearch/Logstash | | SSL certificate | :5066 (if TLS) | Expiring certificate on monitoring endpoint | | Cron heartbeat | Heartbeat URL | Full pipeline stall: events not reaching Elasticsearch |

Filebeat's silent failure modes — wrong log path, permission denied on log files, Elasticsearch mapping conflicts, registry corruption — all produce the same visible symptom: zero new log entries in Kibana. With Vigilmon watching Filebeat's harvester count, ACK rate, dropped events, and end-to-end heartbeat, you catch log pipeline failures before they become investigation blockers during your most critical incidents.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles