How to Monitor Apache NiFi: Processor Health, Queue Depth, and Uptime with Vigilmon

Apache NiFi orchestrates complex data flows between systems — but when a processor silently fails, a queue fills to back-pressure, or a cluster node drops out, your pipeline stalls without raising an obvious alarm. NiFi's internal bulletin board catches errors after the fact; you need external, proactive monitoring to catch degradation before it becomes an outage.

Vigilmon layers on top of NiFi's built-in metrics: HTTP probes for the REST API health endpoints, heartbeat monitors for your pipeline's downstream consumers, and response time alerting to detect JVM pressure before GC pauses start dropping flow files.

What Can Go Wrong in NiFi

Before setting up monitors, understand the failure modes:

• Processor failures — a processor transitions to INVALID or stops running after an upstream error, silently halting the flow
• Back-pressure triggers — queue depth exceeds the threshold, causing upstream processors to stop enqueuing; no explicit error is raised
• JVM heap pressure — GC pauses cause processor execution timeouts, manifesting as slow throughput before full stalls
• Cluster node loss — in clustered NiFi, a node disconnect causes partition of the flow across remaining nodes
• Port availability — Input/Output ports used by site-to-site connections become unavailable if their hosting component stops

Step 1: Expose NiFi Health via the REST API

NiFi ships a REST API that exposes cluster, processor, and system diagnostics. The key endpoints:

# Overall cluster status
GET http://localhost:8080/nifi-api/controller/cluster

# System diagnostics (JVM heap, GC, thread count)
GET http://localhost:8080/nifi-api/system-diagnostics

# Process group status (queue depth, active threads, bytes in/out)
GET http://localhost:8080/nifi-api/flow/process-groups/{processGroupId}/status

# Bulletin board — recent processor errors
GET http://localhost:8080/nifi-api/flow/bulletin-board

For basic liveness, NiFi also exposes a simple status endpoint:

curl -s http://localhost:8080/nifi-api/access/token-expiration
# Returns 200 if NiFi is up, 401 if auth required but service is alive

Build a Health Sidecar

For Vigilmon to probe, you need an HTTP endpoint that returns 200/503 based on NiFi's internal health. Here's a lightweight Node.js sidecar:

// nifi-health.js
const express = require('express');
const axios = require('axios');

const app = express();
const NIFI_BASE = process.env.NIFI_BASE_URL || 'http://localhost:8080/nifi-api';
const QUEUE_DEPTH_THRESHOLD = parseInt(process.env.QUEUE_DEPTH_THRESHOLD || '50000');
const HEAP_USED_THRESHOLD = parseFloat(process.env.HEAP_USED_THRESHOLD || '0.85');

app.get('/health/nifi', async (req, res) => {
  try {
    const [sysResp, clusterResp] = await Promise.all([
      axios.get(`${NIFI_BASE}/system-diagnostics`, { timeout: 5000 }),
      axios.get(`${NIFI_BASE}/controller/cluster`, { timeout: 5000 }),
    ]);

    const heap = sysResp.data.systemDiagnostics.aggregateSnapshot;
    const heapUsedRatio = heap.usedHeapBytes / heap.maxHeapBytes;

    const cluster = clusterResp.data.cluster;
    const disconnectedNodes = cluster.nodes.filter(n => n.status !== 'CONNECTED');

    const issues = [];

    if (heapUsedRatio > HEAP_USED_THRESHOLD) {
      issues.push(`heap at ${(heapUsedRatio * 100).toFixed(1)}%`);
    }

    if (disconnectedNodes.length > 0) {
      issues.push(`${disconnectedNodes.length} node(s) disconnected`);
    }

    if (issues.length > 0) {
      return res.status(503).json({ status: 'degraded', issues });
    }

    return res.status(200).json({
      status: 'ok',
      heap_used_pct: (heapUsedRatio * 100).toFixed(1),
      nodes: cluster.nodes.length,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/nifi/queues', async (req, res) => {
  const processGroupId = process.env.NIFI_ROOT_PG_ID || 'root';
  try {
    const resp = await axios.get(
      `${NIFI_BASE}/flow/process-groups/${processGroupId}/status`,
      { timeout: 5000 }
    );

    const stats = resp.data.processGroupStatus.aggregateSnapshot;
    const queuedCount = stats.queuedCount;  // total flow files queued across all connections

    if (queuedCount > QUEUE_DEPTH_THRESHOLD) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'queue_depth_exceeded',
        queued: queuedCount,
        threshold: QUEUE_DEPTH_THRESHOLD,
      });
    }

    return res.status(200).json({ status: 'ok', queued: queuedCount });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3010, () => console.log('NiFi health sidecar on :3010'));

Python Alternative (for NiFi with Kerberos/token auth)

# nifi_health.py
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)
NIFI_BASE = os.environ.get('NIFI_BASE_URL', 'http://localhost:8080/nifi-api')
HEADERS = {'Authorization': f"Bearer {os.environ.get('NIFI_TOKEN', '')}"}

@app.route('/health/nifi')
def health():
    try:
        r = requests.get(f'{NIFI_BASE}/system-diagnostics', headers=HEADERS, timeout=5)
        r.raise_for_status()
        snap = r.json()['systemDiagnostics']['aggregateSnapshot']
        ratio = snap['usedHeapBytes'] / snap['maxHeapBytes']
        if ratio > 0.85:
            return jsonify(status='degraded', heap_pct=f'{ratio:.1%}'), 503
        return jsonify(status='ok', heap_pct=f'{ratio:.1%}')
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

if __name__ == '__main__':
    app.run(port=3010)

Step 2: Configure Vigilmon Monitors for NiFi

HTTP Monitor — NiFi Liveness

1. Log in to vigilmon.online → Monitors → New Monitor → HTTP / HTTPS
2. URL: https://your-nifi-host.example.com/health/nifi
3. Interval: 1 minute
4. Expected status: 200
5. Body assertion: "status":"ok"
6. Response time threshold: 10000ms (NiFi REST API can be slow under load)
7. Alert channel: PagerDuty (P1)

HTTP Monitor — Queue Depth

1. New monitor → HTTP / HTTPS
2. URL: https://your-nifi-host.example.com/health/nifi/queues
3. Interval: 2 minutes
4. Expected: 200, body contains "status":"ok"
5. Alert channel: Slack (P2 — back-pressure doesn't always mean outage)

Step 3: Heartbeat Monitoring for Downstream Consumers

NiFi health endpoints tell you the pipeline is running — but not that data is actually flowing through to its destination. Add a Vigilmon heartbeat monitor to the system that consumes NiFi's output:

// In your downstream consumer (e.g., a service reading from NiFi output ports)
const axios = require('axios');

async function processNiFiOutput(records) {
  for (const record of records) {
    await handle(record);
  }

  // Ping Vigilmon to prove end-to-end flow is alive
  await axios.get(process.env.VIGILMON_HEARTBEAT_URL, { timeout: 3000 })
    .catch(() => {});
}

Set up the heartbeat in Vigilmon:

1. Monitors → New Monitor → Heartbeat
2. Name: nifi-pipeline-output
3. Expected interval: match your pipeline's SLA (e.g., 10 minutes for a pipeline that should deliver every 10 min)
4. Grace period: 5 minutes
5. Alert channel: PagerDuty

This catches the scenario where NiFi appears healthy internally but a misconfigured routing rule silently drops all records before they reach the destination.

Step 4: JVM and Back-Pressure Alert Routing

| Monitor | Trigger | Alert Channel | Priority | |---|---|---|---| | /health/nifi | NiFi down or heap >85% | Slack + PagerDuty | P1 | | /health/nifi/queues | Queue depth >50,000 | Slack | P2 | | Heartbeat: pipeline output | No delivery in 10 min | PagerDuty | P1 | | Heartbeat: bulletin board watcher | No check-in in 5 min | Email | P3 |

Response time alerting is particularly valuable for NiFi: a GC pause manifests as a slow API response (>5000ms) before full stall. Configure a response time threshold alert in Vigilmon to catch this early.

Summary

NiFi data pipelines fail silently — back-pressure, processor errors, and JVM pressure don't page you by default. Pair NiFi's REST API with Vigilmon's external probes and heartbeat monitors to build a complete alerting layer:

| Layer | What It Catches | |---|---| | HTTP monitor: /health/nifi | NiFi down, heap pressure, node disconnects | | HTTP monitor: /health/nifi/queues | Back-pressure triggers, queue buildup | | Heartbeat: downstream consumer | End-to-end pipeline stall, routing errors | | Response time threshold | JVM GC pressure before full stall |

Start monitoring your NiFi pipelines today at vigilmon.online — free tier included.