Monitoring Logstash with Vigilmon

Logstash ingests, transforms, and ships data from hundreds of sources to dozens of outputs — Elasticsearch, Kafka, S3, and more. When a Logstash pipeline stalls, drops events, or the process dies, your entire observability stack goes blind. Vigilmon gives you a lightweight external safety net: HTTP health checks on Logstash's monitoring API, cron heartbeats to confirm pipeline activity, and SSL alerts for secured nodes.

What You'll Set Up

• HTTP monitor on Logstash's built-in monitoring API (_node/stats)
• Cron heartbeat to confirm pipeline event flow
• TCP port monitor for the beats/HTTP input listener
• Alert tuning for pipeline lag and JVM memory pressure

Prerequisites

• Logstash 7.x or 8.x running with the HTTP monitoring API enabled
• A free Vigilmon account

Step 1: Enable the Logstash HTTP Monitoring API

Logstash ships with a built-in HTTP API on port 9600. Verify it is enabled in logstash.yml:

# logstash.yml
http.host: "0.0.0.0"
http.port: 9600

Restart Logstash and confirm the endpoint is responding:

curl http://localhost:9600/_node/stats?pretty

You should see a JSON payload including pipeline stats, JVM heap usage, and event counters. This is the endpoint Vigilmon will probe.

Step 2: Add an HTTP Monitor for the Node Stats Endpoint

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: http://your-logstash-host:9600/_node/stats.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

If Logstash is behind a reverse proxy with TLS, use the HTTPS URL and enable Monitor SSL certificate with a 21-day expiry alert.

For a lighter check, use the root endpoint which just returns the node info without full stats:

http://your-logstash-host:9600/

This confirms the process is alive even if the pipeline is idle.

Step 3: Monitor a Specific Pipeline's Health

Logstash exposes per-pipeline statistics at /_node/stats/pipelines. You can add a second monitor pointed at a critical pipeline:

http://your-logstash-host:9600/_node/stats/pipelines/main

The response includes events.in, events.out, and events.filtered counters. A 200 response confirms the pipeline object exists in memory — meaning Logstash loaded and started it without a configuration error.

If you have multiple pipelines, add one monitor per critical pipeline using the pipeline ID in the path.

Step 4: Add a TCP Port Monitor for the Beats Input

If Logstash accepts Filebeat or Metricbeat connections, its beats input listens on TCP port 5044 by default. A dead listener means no logs arrive, but the Logstash process itself may still be healthy enough to return 200 on port 9600.

Add a TCP port monitor:

1. Click Add Monitor → TCP Port.
2. Enter your Logstash host and port 5044.
3. Set Check interval to 1 minute.
4. Click Save.

Do the same for any other inputs: Syslog (port 5000), HTTP input (port 8080), etc.

Step 5: Heartbeat Monitoring for Pipeline Event Flow

A 200 on the stats API confirms Logstash is running, but not that events are actually flowing. Use a cron heartbeat to verify end-to-end pipeline activity.

Add a Logstash filter that pings Vigilmon whenever events are processed:

# logstash pipeline config
filter {
  if [type] == "heartbeat" {
    ruby {
      code => '
        require "net/http"
        uri = URI("https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID")
        Net::HTTP.get(uri) rescue nil
      '
    }
  }
}

Then set up a scheduled input that emits a heartbeat event every 5 minutes:

input {
  heartbeat {
    interval => 300
    type => "heartbeat"
  }
}

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 5 minutes (300 seconds).
3. Copy the heartbeat URL into the config above.
4. Click Save.

If the pipeline stalls and stops processing events, Vigilmon alerts after 10 minutes (2× the expected interval).

Step 6: Configure Alert Thresholds

Logstash pipeline issues often manifest as gradual slowdowns rather than hard crashes. Tune your alerts:

1. On each monitor, set Consecutive failures before alert to 2 — a single slow response during GC pauses is normal.
2. Add a Slack or email alert channel under Alert Channels.
3. Use Maintenance windows during planned Logstash upgrades or pipeline reloads to suppress false positives.

For JVM memory alerts, watch for the stats API responding slowly (>5 seconds) before failing — that pattern usually precedes an out-of-memory crash. Set a Response time threshold on your Step 2 monitor to 5000ms to catch this early.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP node stats | :9600/_node/stats | Process crash, API failure | | HTTP pipeline stats | :9600/_node/stats/pipelines/main | Pipeline config error, reload failure | | TCP port | :5044 (beats input) | Input listener crash | | Cron heartbeat | Heartbeat URL | Pipeline stall, event processing halt |

Logstash is the quiet workhorse of many observability stacks — when it fails, everything downstream goes dark before anyone notices. With Vigilmon watching the monitoring API, input listeners, and pipeline heartbeat, you catch Logstash problems at the source rather than discovering them when dashboards stop updating.