tutorial

How to Monitor Kafdrop with Vigilmon

Kafdrop is a lightweight Kafka web UI for browsing topics, viewing messages, and inspecting consumer groups. Learn how to monitor Kafdrop server uptime, Kafka broker connectivity, consumer group view health, and key endpoints with Vigilmon.

Kafdrop is the go-to open-source Kafka web UI for development and operations teams who need a fast, dependency-light way to inspect topics, browse messages, and view consumer group offsets without deploying a full commercial control plane. It's a single Spring Boot JAR that connects to your Kafka bootstrap servers and renders everything in a browser — no persistent storage, no agents, no plugins required.

That simplicity is also a monitoring gap: when Kafdrop's embedded Tomcat serves an error page instead of topic metadata, your on-call engineer doesn't find out until they open the browser. This tutorial shows you how to monitor Kafdrop server uptime, Kafka broker connectivity status, consumer group view health, and critical API endpoints with Vigilmon HTTP monitors — so you know before your team does.

Why Kafdrop Needs External Monitoring

Kafdrop is stateless, but it can fail in several ways that look fine from the outside:

  • JVM startup failure — Kafdrop's process exits silently if it can't resolve the Kafka bootstrap address on startup
  • Kafka connectivity lost mid-session — the Kafdrop process stays up, but all topic and consumer views return errors
  • TLS/authentication failures — if your Kafka cluster rotates credentials and Kafdrop isn't updated, every metadata call returns 401 Unauthorized
  • Memory pressure — Kafdrop holds topic metadata in memory; on large clusters it can OOM and restart, serving 503s during the restart window
  • Stale consumer lag display — if Kafdrop's Kafka admin client loses the cluster connection, its consumer group tables show stale or empty data without a visible error

Vigilmon catches all of these with targeted HTTP monitors.

Step 1: Verify Kafdrop's Built-In Health Endpoints

Kafdrop exposes Spring Boot Actuator endpoints by default. Confirm they're enabled:

# Check Kafdrop actuator health
curl -s http://localhost:9000/actuator/health | jq .

# Expected output:
# {
#   "status": "UP"
# }

If the actuator is disabled, enable it in your Kafdrop startup command:

# Docker run
docker run -d --rm -p 9000:9000 \
  -e KAFKA_BROKERCONNECT=kafka:9092 \
  -e JVM_OPTS="-Xms32M -Xmx64M" \
  -e SERVER_SERVLET_CONTEXTPATH="/" \
  -e MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE="health,info" \
  obsidiandynamics/kafdrop:latest

# Kubernetes deployment environment variable
env:
  - name: MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE
    value: "health,info"

Verify the main API responds:

# Topic list API — fails if Kafka connectivity is lost
curl -s http://localhost:9000/api/topic | jq '.[0].name'

# Consumer group API
curl -s http://localhost:9000/api/consumer | jq '.[0].groupId'

Step 2: Add a Kafka Connectivity Health Endpoint

Kafdrop's actuator /health only checks Spring Boot internals, not Kafka connectivity. Add a reverse-proxy or sidecar probe that validates the Kafka-backed API:

# kafdrop_probe.py — lightweight probe for Kafdrop's Kafka connectivity
import os
import requests
from http.server import HTTPServer, BaseHTTPRequestHandler
import json

KAFDROP_BASE = os.environ.get('KAFDROP_URL', 'http://localhost:9000')
PROBE_PORT = int(os.environ.get('PROBE_PORT', '9001'))

def check_kafdrop_up():
    """Verify Kafdrop process is alive via actuator."""
    try:
        r = requests.get(f'{KAFDROP_BASE}/actuator/health', timeout=5)
        if r.status_code == 200 and r.json().get('status') == 'UP':
            return True, None
        return False, f'actuator_status_{r.status_code}'
    except Exception as e:
        return False, str(e)

def check_kafka_connectivity():
    """Verify Kafdrop can reach Kafka by listing topics."""
    try:
        r = requests.get(f'{KAFDROP_BASE}/api/topic', timeout=10)
        if r.status_code == 200:
            topics = r.json()
            return True, len(topics)
        return False, f'topic_api_status_{r.status_code}'
    except Exception as e:
        return False, str(e)

def check_consumer_groups():
    """Verify consumer group metadata is accessible."""
    try:
        r = requests.get(f'{KAFDROP_BASE}/api/consumer', timeout=10)
        if r.status_code == 200:
            groups = r.json()
            return True, len(groups)
        return False, f'consumer_api_status_{r.status_code}'
    except Exception as e:
        return False, str(e)

class ProbeHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/health/kafdrop':
            self._check_all()
        elif self.path == '/health/kafdrop/kafka':
            self._check_kafka()
        elif self.path == '/health/kafdrop/consumers':
            self._check_consumers()
        else:
            self.send_response(404)
            self.end_headers()

    def _respond(self, code, body):
        self.send_response(code)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(body).encode())

    def _check_all(self):
        issues = []
        up, err = check_kafdrop_up()
        if not up:
            return self._respond(503, {'status': 'down', 'reason': err})

        kafka_ok, result = check_kafka_connectivity()
        if not kafka_ok:
            issues.append(f'kafka_unreachable: {result}')
        else:
            topic_count = result

        cg_ok, cg_result = check_consumer_groups()
        if not cg_ok:
            issues.append(f'consumer_groups_unavailable: {cg_result}')

        if issues:
            return self._respond(503, {'status': 'degraded', 'issues': issues})
        self._respond(200, {
            'status': 'ok',
            'topics': topic_count if kafka_ok else 0,
            'consumer_groups': cg_result if cg_ok else 0,
        })

    def _check_kafka(self):
        ok, result = check_kafka_connectivity()
        if not ok:
            return self._respond(503, {'status': 'down', 'reason': result})
        self._respond(200, {'status': 'ok', 'topic_count': result})

    def _check_consumers(self):
        ok, result = check_consumer_groups()
        if not ok:
            return self._respond(503, {'status': 'down', 'reason': result})
        self._respond(200, {'status': 'ok', 'consumer_group_count': result})

    def log_message(self, *args):
        pass

if __name__ == '__main__':
    print(f'Kafdrop probe listening on :{PROBE_PORT}')
    HTTPServer(('0.0.0.0', PROBE_PORT), ProbeHandler).serve_forever()

Run alongside Kafdrop:

pip install requests
KAFDROP_URL=http://localhost:9000 PROBE_PORT=9001 python kafdrop_probe.py

Or in Docker Compose:

# docker-compose.yml
version: '3.8'
services:
  kafka:
    image: bitnami/kafka:3.7
    environment:
      KAFKA_CFG_PROCESS_ROLES: "broker,controller"
      KAFKA_CFG_NODE_ID: "1"
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093"
      KAFKA_CFG_LISTENERS: "PLAINTEXT://:9092,CONTROLLER://:9093"
      KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: "CONTROLLER"

  kafdrop:
    image: obsidiandynamics/kafdrop:latest
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "kafka:9092"
      MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE: "health,info"
    depends_on:
      - kafka

  kafdrop-probe:
    image: python:3.12-slim
    ports:
      - "9001:9001"
    environment:
      KAFDROP_URL: "http://kafdrop:9000"
      PROBE_PORT: "9001"
    command: ["sh", "-c", "pip install requests -q && python /probe/kafdrop_probe.py"]
    volumes:
      - ./kafdrop_probe.py:/probe/kafdrop_probe.py
    depends_on:
      - kafdrop

Step 3: Monitor the Kafka Broker Connectivity Count

When Kafdrop loses connectivity to one or more brokers, it degrades gracefully but displays incomplete data. Add a broker-count check:

def check_broker_count(expected_brokers=3):
    """Check that Kafdrop sees the expected number of brokers."""
    try:
        # Kafdrop's /api/broker endpoint returns broker metadata
        r = requests.get(f'{KAFDROP_BASE}/api/broker', timeout=10)
        if r.status_code != 200:
            return False, f'broker_api_{r.status_code}'
        brokers = r.json()
        if len(brokers) < expected_brokers:
            return False, f'only_{len(brokers)}_of_{expected_brokers}_brokers_visible'
        return True, len(brokers)
    except Exception as e:
        return False, str(e)

Add the route to the probe:

elif self.path.startswith('/health/kafdrop/brokers'):
    from urllib.parse import urlparse, parse_qs
    params = parse_qs(urlparse(self.path).query)
    expected = int((params.get('expected') or ['1'])[0])
    ok, result = check_broker_count(expected)
    if not ok:
        return self._respond(503, {'status': 'degraded', 'reason': result})
    self._respond(200, {'status': 'ok', 'broker_count': result})

Step 4: Configure Vigilmon Monitors

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Kafdrop probe endpoint: https://your-host.example.com/health/kafdrop
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your Slack or PagerDuty integration

Monitor table:

| Monitor URL | Purpose | Interval | Alert | |---|---|---|---| | /health/kafdrop | Full Kafdrop + Kafka + consumer health | 1 min | P1 – PagerDuty | | http://kafdrop:9000/actuator/health | Spring Boot process health | 30 sec | P1 – PagerDuty | | /health/kafdrop/kafka | Kafka connectivity via topic API | 1 min | P1 – PagerDuty | | /health/kafdrop/consumers | Consumer group metadata API | 2 min | P2 – Slack | | /health/kafdrop/brokers?expected=3 | Broker count completeness | 2 min | P2 – Slack |

Step 5: Heartbeat Monitor for Long-Running Kafdrop

Kafdrop is typically deployed as a long-running service. Use a Vigilmon heartbeat to detect silent crashes between HTTP checks:

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name it kafdrop-liveness
  3. Expected interval: 3 minutes, grace period: 6 minutes
  4. Copy the heartbeat URL

Send the heartbeat from a cron or a sidecar script that runs alongside Kafdrop:

# heartbeat.sh — run every 2 minutes via cron
#!/bin/bash
KAFDROP_URL="${KAFDROP_URL:-http://localhost:9000}"
VIGILMON_HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"

# Only ping if Kafdrop is actually responding
if curl -sf "$KAFDROP_URL/actuator/health" | grep -q '"status":"UP"'; then
  curl -sf "$VIGILMON_HEARTBEAT_URL" || true
fi

Add to crontab:

*/2 * * * * /opt/kafdrop/heartbeat.sh >> /var/log/kafdrop-heartbeat.log 2>&1

Step 6: Alert Routing

| Monitor | Priority | Channel | Reason | |---|---|---|---| | Full health (/health/kafdrop) | P1 | PagerDuty | Kafdrop completely unavailable | | Kafka connectivity | P1 | PagerDuty | Team cannot inspect topics or consumers | | Spring Boot actuator | P1 | PagerDuty | JVM crash or OOM restart | | Consumer group API | P2 | Slack | Stale lag data but Kafdrop still functional | | Broker count | P2 | Slack | Partial cluster visible, metadata may be wrong | | Heartbeat | P2 | Slack + email | Silent crash between HTTP check intervals |

Summary

Kafdrop's simplicity means its monitoring setup is equally simple — but without it, Kafka connectivity failures and JVM crashes go unnoticed until a developer reports stale data in the browser. Two Vigilmon HTTP monitors (the built-in actuator and a lightweight connectivity probe) plus a heartbeat give complete coverage.

| Monitor | What It Catches | |---|---| | Spring Boot actuator | JVM health, OOM restarts, startup failures | | /health/kafdrop | End-to-end Kafka connectivity and consumer metadata | | /health/kafdrop/brokers | Partial cluster visibility | | Heartbeat: kafdrop-liveness | Silent crashes between HTTP check windows |

Get started free at vigilmon.online — Kafdrop monitoring configured in under three minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles