How to Monitor Fluent Bit: Plugin Health, Buffer Overflow, and Log Pipeline SLA with Vigilmon

Fluent Bit is the lightweight backbone of most cloud-native log pipelines — it runs on every node, collects from every container, and forwards to every backend. When it silently drops records because a buffer overflowed, retries exhaust for a failing output plugin, or memory limits cause the process to restart, you lose logs that may be critical for incident response or compliance audit trails.

Vigilmon gives Fluent Bit external visibility: HTTP monitors for the built-in HTTP server metrics, heartbeat monitors wired to your downstream log consumers, and response time alerting to catch buffer pressure before records start dropping.

How Fluent Bit Fails

Fluent Bit failures are insidious because the process stays running while data is lost:

• Output plugin failure with exhausted retries — Fluent Bit tries to deliver to the backend (Elasticsearch, Loki, CloudWatch), fails, retries 5× (configurable), then drops the chunk and logs a warning — but continues running
• Buffer overflow — input ingestion rate exceeds output throughput; when the filesystem or memory buffer fills, Fluent Bit begins dropping the oldest chunks
• Memory limit OOM — Mem_Buf_Limit exceeded on an input plugin causes it to pause or drop records silently
• Filter plugin crash — a bad Lua filter or record_modifier panics, dropping all records that pass through that filter
• Tail input losing track — the /fluent-bit/etc/tail_db/ state database gets corrupted; the tail plugin reprocesses from the beginning (log duplication) or from the end (log gap)

Step 1: Enable Fluent Bit's Built-In HTTP Server

Fluent Bit ships an HTTP server that exposes metrics and health. Enable it in your config:

# fluent-bit.conf
[SERVICE]
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Health_Check On
    HC_Errors_Count  5      # fail health check after 5 errors
    HC_Retry_Failure_Count 5 # fail after 5 retry failures
    HC_Period    10         # evaluate every 10 seconds

With this config, Fluent Bit exposes:

# Liveness
GET http://localhost:2020/

# Health check (200 = healthy, 500 = unhealthy based on HC_* thresholds)
GET http://localhost:2020/api/v1/health

# Prometheus-format metrics
GET http://localhost:2020/api/v1/metrics/prometheus

# JSON metrics (easier to parse)
GET http://localhost:2020/api/v1/metrics

You can probe /api/v1/health directly from Vigilmon — no sidecar needed for basic health checks.

Step 2: Build a Richer Health Endpoint

For retry rate, buffer overflow, and record drop detection, parse the JSON metrics endpoint:

// fluent-bit-health.js
const express = require('express');
const axios = require('axios');

const app = express();
const FB_URL = process.env.FLUENT_BIT_URL || 'http://localhost:2020';
const MAX_RETRY_RATE = parseFloat(process.env.MAX_RETRY_RATE || '0.05');  // 5% retry rate threshold
const MAX_ERROR_RATE = parseFloat(process.env.MAX_ERROR_RATE || '0.01');  // 1% drop rate threshold

app.get('/health/fluent-bit', async (req, res) => {
  try {
    // Check built-in health endpoint first
    const healthResp = await axios.get(`${FB_URL}/api/v1/health`, { timeout: 3000 });
    if (healthResp.status !== 200) {
      return res.status(503).json({
        status: 'unhealthy',
        reason: 'fluent_bit_health_check_failed',
        fb_status: healthResp.status,
      });
    }

    // Parse detailed metrics
    const metricsResp = await axios.get(`${FB_URL}/api/v1/metrics`, { timeout: 3000 });
    const metrics = metricsResp.data;

    const issues = [];

    // Check output plugins for retry and drop rates
    for (const [name, output] of Object.entries(metrics.output || {})) {
      const retries = output.retries || 0;
      const errors = output.errors || 0;
      const records = output.proc_records || 0;

      if (records > 0) {
        const retryRate = retries / records;
        const errorRate = errors / records;

        if (retryRate > MAX_RETRY_RATE) {
          issues.push(`output[${name}] retry rate ${(retryRate * 100).toFixed(1)}%`);
        }

        if (errorRate > MAX_ERROR_RATE) {
          issues.push(`output[${name}] drop rate ${(errorRate * 100).toFixed(1)}%`);
        }

        // Check for failed retries (records permanently dropped)
        if (output.retries_failed && output.retries_failed > 0) {
          issues.push(`output[${name}] ${output.retries_failed} records permanently dropped`);
        }
      }
    }

    // Check input plugins for dropped records (buffer overflow)
    for (const [name, input] of Object.entries(metrics.input || {})) {
      if (input.drop_records > 0) {
        issues.push(`input[${name}] dropped ${input.drop_records} records (buffer overflow)`);
      }
    }

    if (issues.length > 0) {
      return res.status(503).json({ status: 'degraded', issues });
    }

    const totalRecords = Object.values(metrics.output || {})
      .reduce((sum, o) => sum + (o.proc_records || 0), 0);

    return res.status(200).json({ status: 'ok', total_records_processed: totalRecords });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/fluent-bit/inputs', async (req, res) => {
  try {
    const resp = await axios.get(`${FB_URL}/api/v1/metrics`, { timeout: 3000 });
    const inputs = resp.data.input || {};

    const summary = Object.entries(inputs).map(([name, stats]) => ({
      name,
      records: stats.records || 0,
      bytes: stats.bytes || 0,
      dropped: stats.drop_records || 0,
    }));

    const hasDrops = summary.some(s => s.dropped > 0);
    return res.status(hasDrops ? 503 : 200).json({
      status: hasDrops ? 'degraded' : 'ok',
      inputs: summary,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3013, () => console.log('Fluent Bit health sidecar on :3013'));

Python Version

# fluent_bit_health.py
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)
FB_URL = os.environ.get('FLUENT_BIT_URL', 'http://localhost:2020')

@app.route('/health/fluent-bit')
def health():
    try:
        h = requests.get(f'{FB_URL}/api/v1/health', timeout=3)
        if h.status_code != 200:
            return jsonify(status='unhealthy'), 503

        m = requests.get(f'{FB_URL}/api/v1/metrics', timeout=3).json()
        issues = []

        for name, out in m.get('output', {}).items():
            records = out.get('proc_records', 0) or 1
            if out.get('retries_failed', 0) > 0:
                issues.append(f'output[{name}] has permanently dropped records')
            if out.get('errors', 0) / records > 0.01:
                issues.append(f'output[{name}] error rate >1%')

        for name, inp in m.get('input', {}).items():
            if inp.get('drop_records', 0) > 0:
                issues.append(f'input[{name}] buffer overflow drops')

        if issues:
            return jsonify(status='degraded', issues=issues), 503
        return jsonify(status='ok')
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

if __name__ == '__main__':
    app.run(port=3013)

Step 3: Configure Vigilmon HTTP Monitors

Monitor 1 — Direct Health Check

1. Log in to vigilmon.online → Monitors → New Monitor → HTTP / HTTPS
2. URL: http://your-fluent-bit-host:2020/api/v1/health
3. Interval: 1 minute
4. Expected status: 200
5. Response time threshold: 2000ms
6. Alert channel: PagerDuty (P1 — Fluent Bit health failure means log data loss)

Monitor 2 — Deep Health (Drop Rate + Retry Rate)

1. New monitor → HTTP / HTTPS
2. URL: https://your-fluent-bit-host.example.com/health/fluent-bit
3. Interval: 2 minutes
4. Expected: 200, body contains "status":"ok"
5. Response time threshold: 5000ms
6. Alert channel: Slack + PagerDuty for retries_failed drops

Step 4: Heartbeat Monitoring for Log Pipeline SLA

HTTP monitors tell you Fluent Bit is running. They don't tell you logs are reaching their destination on time. Use a Vigilmon heartbeat monitor driven by your log backend's ingestion acknowledgment:

# Add a synthetic log line that Fluent Bit routes to a special heartbeat output
# Configure a second OUTPUT in fluent-bit.conf:

[OUTPUT]
    Name  http
    Match vigilmon_heartbeat
    Host  vigilmon.online
    Port  443
    URI   /heartbeat/YOUR_HEARTBEAT_ID
    TLS   On
    tls.verify On

Or send it from application code alongside your logs:

# In your application's log shipping loop
import requests, os

def ship_logs_batch(records: list):
    # ... ship records via Fluent Bit / direct API ...
    # Then confirm delivery:
    hb_url = os.environ.get('VIGILMON_LOG_PIPELINE_HEARTBEAT')
    if hb_url and records:
        requests.get(hb_url, timeout=3)

Set up the heartbeat monitor:

1. Monitors → New Monitor → Heartbeat
2. Name: fluent-bit-log-pipeline
3. Expected interval: 5 minutes (adjust to your log volume — low-traffic systems may need longer)
4. Grace period: 10 minutes
5. Alert channel: PagerDuty for compliance-critical log streams

Step 5: Memory Limit and Buffer Overflow Detection

Add explicit memory monitoring to catch Mem_Buf_Limit exhaustion before records drop:

# fluent-bit.conf — set explicit limits and watch for overflows
[INPUT]
    Name  tail
    Path  /var/log/app/*.log
    Mem_Buf_Limit  50MB    # alert when this is consistently near-full
    storage.type  filesystem  # spill to disk instead of dropping on overflow

With storage.type filesystem, configure Vigilmon to monitor disk space on the buffer path as a secondary check — a full disk means buffer overflow is imminent.

Alert Routing Table

| Monitor | Trigger | Channel | Priority | |---|---|---|---| | HTTP: /api/v1/health | Fluent Bit process unhealthy | PagerDuty | P1 | | HTTP: deep health | Record drops, retry exhaustion | PagerDuty | P1 | | HTTP: inputs check | Buffer overflow on any input | Slack | P2 | | Heartbeat: log pipeline | No log delivery in 10 min | PagerDuty | P1 | | Heartbeat: low-traffic stream | No delivery in 30 min | Email | P2 |

Summary

Fluent Bit drops logs silently — buffer overflows, exhausted retries, and filter panics all keep the process running while your log data disappears. Cover every failure mode:

| Layer | What It Catches | |---|---| | HTTP: built-in health | Process unhealthy (HC threshold exceeded) | | HTTP: deep metrics | Record drops, retry rates, permanently failed records | | HTTP: input metrics | Buffer overflow per input plugin | | Heartbeat: pipeline SLA | No log delivery reaching destination |

Start protecting your log pipeline at vigilmon.online — free tier, no card required.