tutorial

Monitoring Graylog with Vigilmon: HTTP, TCP Port, and Alerting for Log Management

How to monitor Graylog centralized log management with Vigilmon — HTTP uptime checks for the web interface, TCP port monitoring for GELF inputs, API health endpoints, and alerting for log ingestion gaps.

Graylog is an open-source centralized log management platform that ingests logs over GELF (Graylog Extended Log Format), syslog, and Beats inputs, then lets you search, analyze, and alert on them through a web dashboard. A Graylog outage is doubly dangerous: not only are your logs unavailable for incident investigation, but log shippers may drop messages silently if the GELF input goes down. Vigilmon catches both: HTTP monitors for the web interface, TCP port monitors for GELF inputs, API health checks, and heartbeat monitors for log ingestion pipelines.

What You'll Build

  • A Vigilmon HTTP monitor for the Graylog web interface on port 9000
  • A TCP port monitor for the GELF UDP/TCP input (port 12201)
  • An HTTP monitor against the Graylog REST API health endpoint
  • A heartbeat monitor that fires when your log shipper stops sending logs

Prerequisites

  • A running Graylog instance reachable at graylog.example.com (default ports: 9000 for HTTP, 12201 for GELF)
  • A free account at vigilmon.online
  • Graylog API credentials (username + password, or API token)

Step 1: Verify Graylog Is Reachable

Check that the web interface and GELF input are up before creating Vigilmon monitors:

# Check web interface
curl -s -o /dev/null -w "%{http_code}" http://graylog.example.com:9000/

# Check GELF TCP port
nc -zv graylog.example.com 12201

# Check REST API health
curl -s -u admin:yourpassword \
  http://graylog.example.com:9000/api/system/lbstatus

The load balancer status endpoint returns ALIVE when Graylog is healthy:

ALIVE

The system overview endpoint provides cluster details:

curl -s -u admin:yourpassword \
  -H "X-Requested-By: vigilmon" \
  http://graylog.example.com:9000/api/system | jq '{lifecycle, lb_status, is_processing}'

Expected healthy output:

{
  "lifecycle": "running",
  "lb_status": "alive",
  "is_processing": true
}

If is_processing is false, Graylog has paused message processing — logs are not being indexed even though the interface is up.

Step 2: Create an HTTP Monitor for the Web Interface

The simplest monitor checks that the Graylog login page is reachable:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://graylog.example.com:9000/.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Graylog (or login — text present on the login page).
  7. Click Save.

For TLS-terminated Graylog:

  • URL: https://graylog.example.com/
  • Enable SSL certificate expiry monitoring in Vigilmon for advance warning of cert issues.

Step 3: Create an HTTP Monitor for the Graylog API Health Endpoint

The REST API's lbstatus endpoint is designed for health probes and requires no authentication:

  1. Add Monitor → HTTP.
  2. URL: http://graylog.example.com:9000/api/system/lbstatus.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ALIVE.
  7. Click Save.

This endpoint returns DEAD (HTTP 503) when Graylog is in a degraded state, making it a reliable health signal.

Monitor the Processing Status

For deeper checks — specifically detecting when is_processing is false — build a thin proxy that reads the system API and returns 200/503 accordingly:

# graylog_health.py
import os
import requests
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

GRAYLOG_URL = os.environ.get("GRAYLOG_URL", "http://localhost:9000")
GRAYLOG_USER = os.environ.get("GRAYLOG_USER", "admin")
GRAYLOG_PASS = os.environ.get("GRAYLOG_PASS", "admin")


@app.get("/health/graylog")
def graylog_health():
    try:
        r = requests.get(
            f"{GRAYLOG_URL}/api/system",
            auth=(GRAYLOG_USER, GRAYLOG_PASS),
            headers={"X-Requested-By": "vigilmon-health-check"},
            timeout=10,
        )
        r.raise_for_status()
        data = r.json()
        lifecycle = data.get("lifecycle", "unknown")
        lb_status = data.get("lb_status", "unknown")
        is_processing = data.get("is_processing", False)

        if lifecycle != "running" or lb_status != "alive" or not is_processing:
            return JSONResponse(status_code=503, content={
                "status": "degraded",
                "lifecycle": lifecycle,
                "lb_status": lb_status,
                "is_processing": is_processing,
            })

        return {"status": "ok", "lifecycle": lifecycle, "lb_status": lb_status}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

# Start the health proxy
uvicorn graylog_health:app --host 0.0.0.0 --port 3012

# Test it
curl http://localhost:3012/health/graylog
# {"status":"ok","lifecycle":"running","lb_status":"alive"}

Add a Vigilmon HTTP monitor for http://your-server.example.com:3012/health/graylog with keyword "status":"ok".

Step 4: Create a TCP Port Monitor for the GELF Input

Graylog accepts logs over GELF on port 12201 (TCP and UDP). If this input goes down, log shippers start buffering or dropping messages.

TCP Monitor

  1. Add Monitor → TCP Port.
  2. Host: graylog.example.com.
  3. Port: 12201.
  4. Check interval: 60 seconds.
  5. Click Save.

Verify the GELF Input Yourself

Send a test GELF message to confirm the input is processing:

# Send a GELF message over TCP
echo '{"version":"1.1","host":"monitor-check","short_message":"vigilmon health test","level":6}' \
  | nc -w1 graylog.example.com 12201

Then search in Graylog for short_message:vigilmon health test to confirm it arrived.

For UDP:

echo '{"version":"1.1","host":"monitor-check","short_message":"vigilmon health test","level":6}' \
  | nc -u -w1 graylog.example.com 12201

Step 5: Heartbeat Monitor for Log Shippers

Log shippers (Filebeat, Fluentd, Logstash, Vector) run continuously. If a shipper crashes, logs stop arriving in Graylog but no alert fires — they just go missing. A Vigilmon heartbeat monitor catches shipper failures.

Create the Heartbeat Monitor

  1. Add Monitor → Heartbeat.
  2. Name: graylog-filebeat-shipper.
  3. Expected interval: 300 seconds (5 minutes — adjust to your heartbeat script cadence).
  4. Grace period: 10 minutes.
  5. Copy the unique heartbeat URL: https://vigilmon.online/heartbeat/your-unique-id.

Wire Into Filebeat

Add a custom processor to Filebeat's config that calls the heartbeat URL periodically:

# filebeat.yml
filebeat.inputs:
  - type: log
    paths:
      - /var/log/app/*.log
    fields:
      service: myapp

output.gelf:
  hosts:
    - "graylog.example.com:12201"

# Run a monitoring script every 5 minutes alongside Filebeat

Because Filebeat doesn't natively support HTTP callbacks, run a companion cron job:

# /etc/cron.d/filebeat-heartbeat
*/5 * * * * root pgrep -x filebeat > /dev/null && curl -fsS "${VIGILMON_HEARTBEAT_URL}" > /dev/null 2>&1

Wire Into Fluentd

Add a heartbeat ping to Fluentd's output chain:

# fluentd.conf
<match heartbeat.**>
  @type exec
  command curl -fsS ${VIGILMON_HEARTBEAT_URL}
  <format>
    @type json
  </format>
</match>

<source>
  @type timer
  tag heartbeat.vigilmon
  interval 300
</source>

Wire Into Vector

# vector.toml
[sources.internal_metrics]
type = "internal_metrics"

[sinks.vigilmon_heartbeat]
type = "http"
inputs = ["internal_metrics"]
uri = "https://vigilmon.online/heartbeat/your-unique-id"
method = "GET"
# Send every 5 minutes — adjust as needed
batch.timeout_secs = 300

Wire Into a Custom Shipper Script

#!/bin/bash
# log-shipper.sh — wrap your custom log forwarding

while true; do
  # Forward logs to Graylog
  send_logs_to_graylog.sh

  # Ping heartbeat after successful delivery
  curl -fsS "${VIGILMON_HEARTBEAT_URL}" || true

  sleep 300
done

Step 6: Monitor Graylog's Elasticsearch / OpenSearch Backend

Graylog stores indexed logs in Elasticsearch or OpenSearch. If the backend goes down, Graylog continues receiving logs but can't index or search them — it looks alive but is actually broken.

Check the cluster state via the Graylog API:

curl -s -u admin:yourpassword \
  -H "X-Requested-By: vigilmon" \
  "http://graylog.example.com:9000/api/system/indexer/cluster/health" \
  | jq '{status, number_of_nodes}'

Extend the health proxy to include this check:

@app.get("/health/graylog/indexer")
def indexer_health():
    try:
        r = requests.get(
            f"{GRAYLOG_URL}/api/system/indexer/cluster/health",
            auth=(GRAYLOG_USER, GRAYLOG_PASS),
            headers={"X-Requested-By": "vigilmon-health-check"},
            timeout=10,
        )
        r.raise_for_status()
        data = r.json()
        cluster_status = data.get("status", "unknown")  # green / yellow / red

        if cluster_status == "red":
            return JSONResponse(status_code=503, content={
                "status": "down",
                "indexer_cluster": cluster_status,
                "nodes": data.get("number_of_nodes", 0),
            })
        if cluster_status == "yellow":
            return JSONResponse(status_code=200, content={
                "status": "degraded",
                "indexer_cluster": cluster_status,
                "nodes": data.get("number_of_nodes", 0),
            })

        return {"status": "ok", "indexer_cluster": cluster_status}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Add a Vigilmon HTTP monitor for /health/graylog/indexer with keyword "status":"ok".

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert routing:

| Monitor | Trigger | Recommended Action | |---|---|---| | HTTP (web interface) | 503 or keyword absent | Check Graylog process; inspect system logs; verify MongoDB is running | | HTTP (lbstatus) | DEAD response or 503 | Graylog has paused — check disk space, MongoDB, Elasticsearch | | HTTP (processing status) | is_processing: false | Restart Graylog message processing via API or web UI | | TCP (GELF port 12201) | Port unreachable | Check GELF input is enabled in Graylog; verify firewall rules | | HTTP (indexer health) | 503 (cluster red) | Check Elasticsearch / OpenSearch cluster logs; restore shards | | Heartbeat (log shipper) | Ping not received | Confirm shipper process is running; check network to Graylog |

Recommended thresholds:

  • Confirmation period: 2 consecutive failures (brief Graylog restarts during upgrades are common)
  • Response time alert: 5000ms (Graylog's web interface can be slow on overloaded nodes)
  • Recovery notification: Enable so your team knows when logging is restored

Common Graylog Failure Modes

| Scenario | What Vigilmon Catches | |---|---| | Graylog process crash | HTTP web interface monitor and lbstatus both fail | | MongoDB unavailable | Graylog starts but lbstatus returns DEAD | | Elasticsearch cluster red | Indexer health endpoint returns 503 | | GELF input disabled in UI | TCP port may still respond; GELF monitor misses input errors | | Disk full (Graylog journal) | Processing pauses; is_processing: false in health proxy | | Log shipper process crash | Heartbeat goes silent | | Network partition from shippers to Graylog | Heartbeat goes silent from shipper host |

Graylog is a critical piece of infrastructure: when it fails, you lose both visibility into your systems and the evidence trail for incident investigation. Vigilmon's layered monitors — HTTP, TCP, API health proxy, and shipper heartbeats — catch Graylog failures from multiple angles before your team realizes logs are missing.

Get started free at vigilmon.online — your Graylog monitor is running in under 5 minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles