tutorial

How to Monitor Kafka UI (Provectus) with Vigilmon

Kafka UI by Provectus is a free open-source Kafka web console. Learn how to monitor UI server uptime, cluster connectivity health, consumer group monitoring, and broker metrics display health with Vigilmon.

Kafka UI (also known as kafka-ui by Provectus Labs) is the most-starred open-source Kafka web console on GitHub. Unlike Kafdrop, it's built as a full-featured management interface: multi-cluster support, ACL management, schema registry integration, Kafka Connect management, and consumer group lag visualization. It runs as a Spring Boot service and talks directly to Kafka brokers and, optionally, Schema Registry and Kafka Connect.

Because Kafka UI sits at the intersection of multiple infrastructure services, it has more failure surfaces than a simple read-only UI: it can be up but unable to reach one cluster, show correct topic metadata but stale consumer lag, or proxy Schema Registry requests while the registry itself is down. This tutorial shows you how to monitor Kafka UI's uptime, cluster connectivity, consumer group health, and broker metrics display with Vigilmon HTTP monitors.

Why Kafka UI Needs Its Own Monitoring Layer

Kafka UI's Spring Boot process staying alive does not mean it's working correctly:

  • Cluster disconnection — Kafka UI retains its process but returns error cards for all cluster-specific views if the Kafka bootstrap address is unreachable
  • Stale consumer group data — consumer lag tables cache data from the Kafka admin client; a client connection drop causes stale display without surfacing an error
  • Schema Registry proxy failures — if Schema Registry is down, Kafka UI schema browser returns errors while all other views remain functional
  • Multi-cluster partial failure — in a multi-cluster Kafka UI deployment, one cluster can be unreachable while others appear healthy
  • Metrics endpoint degradation — broker JMX metrics displayed in Kafka UI require a separate polling path; loss of that path silently zeros out the metrics charts

Vigilmon catches these conditions by probing the API paths that back each UI view.

Step 1: Verify Kafka UI's Built-In Health Endpoints

Kafka UI exposes Spring Boot Actuator health endpoints at /actuator:

# Spring Boot health endpoint
curl -s http://localhost:8080/actuator/health | jq .

# Expected output:
# { "status": "UP" }

# Full details (requires management.endpoint.health.show-details=always)
curl -s http://localhost:8080/actuator/health | jq '.components'

Verify the cluster-specific API endpoint that powers the cluster overview page:

# List configured clusters
curl -s http://localhost:8080/api/clusters | jq '[.[] | {name, status}]'

# Expected output:
# [
#   {"name": "local", "status": "online"},
#   {"name": "staging", "status": "online"}
# ]

The /api/clusters endpoint is the most important signal for Kafka UI health: a cluster shown as "status": "offline" means Kafka UI cannot reach the Kafka bootstrap servers for that cluster.

Step 2: Build a Kafka UI Health Probe

Create a probe that aggregates Kafka UI's health signals into Vigilmon-readable endpoints:

# kafka_ui_probe.py
import os
import json
import requests
from http.server import HTTPServer, BaseHTTPRequestHandler

KAFKA_UI_URL = os.environ.get('KAFKA_UI_URL', 'http://localhost:8080')
PROBE_PORT = int(os.environ.get('PROBE_PORT', '8090'))
EXPECTED_CLUSTERS = os.environ.get('EXPECTED_CLUSTERS', '').split(',')
EXPECTED_CLUSTERS = [c.strip() for c in EXPECTED_CLUSTERS if c.strip()]

def check_actuator():
    """Check Spring Boot health."""
    try:
        r = requests.get(f'{KAFKA_UI_URL}/actuator/health', timeout=5)
        if r.status_code == 200 and r.json().get('status') == 'UP':
            return True, None
        return False, f'actuator_{r.status_code}'
    except Exception as e:
        return False, str(e)

def check_clusters():
    """Verify all configured clusters are online."""
    try:
        r = requests.get(f'{KAFKA_UI_URL}/api/clusters', timeout=10)
        if r.status_code != 200:
            return False, [], f'clusters_api_{r.status_code}'
        clusters = r.json()
        offline = [c['name'] for c in clusters if c.get('status') != 'online']
        return len(offline) == 0, clusters, offline
    except Exception as e:
        return False, [], str(e)

def check_expected_clusters(clusters):
    """Verify that all expected clusters are present and online."""
    if not EXPECTED_CLUSTERS:
        return []
    names = {c['name']: c.get('status') for c in clusters}
    missing = []
    for expected in EXPECTED_CLUSTERS:
        if expected not in names:
            missing.append(f'{expected}_not_found')
        elif names[expected] != 'online':
            missing.append(f'{expected}_status_{names[expected]}')
    return missing

def check_consumer_groups(cluster_name):
    """Verify consumer group data is accessible for a cluster."""
    try:
        r = requests.get(
            f'{KAFKA_UI_URL}/api/clusters/{cluster_name}/consumer-groups',
            timeout=10,
        )
        if r.status_code == 200:
            groups = r.json()
            return True, len(groups), None
        return False, 0, f'consumer_groups_api_{r.status_code}'
    except Exception as e:
        return False, 0, str(e)

def check_broker_metrics(cluster_name):
    """Check that broker metric data is being returned."""
    try:
        r = requests.get(
            f'{KAFKA_UI_URL}/api/clusters/{cluster_name}/metrics',
            timeout=10,
        )
        if r.status_code == 200:
            metrics = r.json()
            # Verify at least some broker metrics are non-null
            broker_count = metrics.get('brokerCount', 0)
            active_controllers = metrics.get('activeControllers', None)
            return True, {
                'broker_count': broker_count,
                'active_controllers': active_controllers,
            }, None
        return False, {}, f'metrics_api_{r.status_code}'
    except Exception as e:
        return False, {}, str(e)

def check_topics(cluster_name):
    """Verify topic list API responds."""
    try:
        r = requests.get(
            f'{KAFKA_UI_URL}/api/clusters/{cluster_name}/topics',
            timeout=10,
        )
        if r.status_code == 200:
            data = r.json()
            topics = data.get('topics', data) if isinstance(data, dict) else data
            return True, len(topics) if isinstance(topics, list) else 0, None
        return False, 0, f'topics_api_{r.status_code}'
    except Exception as e:
        return False, 0, str(e)

class ProbeHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        from urllib.parse import urlparse, parse_qs
        parsed = urlparse(self.path)
        params = parse_qs(parsed.query)

        if parsed.path == '/health/kafka-ui':
            self._check_all()
        elif parsed.path == '/health/kafka-ui/clusters':
            self._check_clusters()
        elif parsed.path == '/health/kafka-ui/consumers':
            cluster = (params.get('cluster') or [''])[0]
            self._check_consumers(cluster)
        elif parsed.path == '/health/kafka-ui/metrics':
            cluster = (params.get('cluster') or [''])[0]
            self._check_metrics(cluster)
        elif parsed.path == '/health/kafka-ui/topics':
            cluster = (params.get('cluster') or [''])[0]
            self._check_topics(cluster)
        else:
            self.send_response(404)
            self.end_headers()

    def _respond(self, code, body):
        self.send_response(code)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps(body).encode())

    def _check_all(self):
        issues = []

        up, err = check_actuator()
        if not up:
            return self._respond(503, {'status': 'down', 'reason': err})

        clusters_ok, clusters, offline = check_clusters()
        if not clusters_ok and not clusters:
            return self._respond(503, {'status': 'down', 'reason': 'clusters_api_failed'})

        if offline:
            issues.append(f'offline_clusters: {",".join(offline)}')

        missing = check_expected_clusters(clusters)
        if missing:
            issues.append(f'expected_clusters_missing: {",".join(missing)}')

        # Spot-check the first online cluster
        online_clusters = [c['name'] for c in clusters if c.get('status') == 'online']
        if online_clusters:
            cluster = online_clusters[0]
            cg_ok, cg_count, cg_err = check_consumer_groups(cluster)
            if not cg_ok:
                issues.append(f'consumer_groups_unavailable: {cg_err}')

        if issues:
            return self._respond(503, {
                'status': 'degraded',
                'issues': issues,
                'total_clusters': len(clusters),
                'online_clusters': len([c for c in clusters if c.get('status') == 'online']),
            })
        self._respond(200, {
            'status': 'ok',
            'total_clusters': len(clusters),
            'online_clusters': len(online_clusters),
        })

    def _check_clusters(self):
        ok, clusters, offline = check_clusters()
        if not ok and not clusters:
            return self._respond(503, {'status': 'down', 'reason': 'clusters_api_failed'})
        if offline:
            return self._respond(503, {
                'status': 'degraded',
                'offline_clusters': offline,
                'total': len(clusters),
            })
        self._respond(200, {
            'status': 'ok',
            'clusters': [{'name': c['name'], 'status': c.get('status')} for c in clusters],
        })

    def _check_consumers(self, cluster):
        if not cluster:
            _, clusters, _ = check_clusters()
            online = [c['name'] for c in clusters if c.get('status') == 'online']
            cluster = online[0] if online else ''
        if not cluster:
            return self._respond(503, {'status': 'down', 'reason': 'no_online_cluster'})
        ok, count, err = check_consumer_groups(cluster)
        if not ok:
            return self._respond(503, {'status': 'down', 'reason': err, 'cluster': cluster})
        self._respond(200, {'status': 'ok', 'consumer_group_count': count, 'cluster': cluster})

    def _check_metrics(self, cluster):
        if not cluster:
            _, clusters, _ = check_clusters()
            online = [c['name'] for c in clusters if c.get('status') == 'online']
            cluster = online[0] if online else ''
        if not cluster:
            return self._respond(503, {'status': 'down', 'reason': 'no_online_cluster'})
        ok, metrics, err = check_broker_metrics(cluster)
        if not ok:
            return self._respond(503, {'status': 'down', 'reason': err, 'cluster': cluster})
        if metrics.get('broker_count', 0) == 0:
            return self._respond(503, {
                'status': 'degraded',
                'reason': 'zero_brokers_in_metrics',
                'cluster': cluster,
            })
        self._respond(200, {'status': 'ok', 'cluster': cluster, **metrics})

    def _check_topics(self, cluster):
        if not cluster:
            _, clusters, _ = check_clusters()
            online = [c['name'] for c in clusters if c.get('status') == 'online']
            cluster = online[0] if online else ''
        if not cluster:
            return self._respond(503, {'status': 'down', 'reason': 'no_online_cluster'})
        ok, count, err = check_topics(cluster)
        if not ok:
            return self._respond(503, {'status': 'down', 'reason': err, 'cluster': cluster})
        self._respond(200, {'status': 'ok', 'topic_count': count, 'cluster': cluster})

    def log_message(self, *args):
        pass

if __name__ == '__main__':
    print(f'Kafka UI probe listening on :{PROBE_PORT}')
    HTTPServer(('0.0.0.0', PROBE_PORT), ProbeHandler).serve_forever()

Run it alongside Kafka UI:

pip install requests
KAFKA_UI_URL=http://localhost:8080 \
PROBE_PORT=8090 \
EXPECTED_CLUSTERS=local,staging \
python kafka_ui_probe.py

Or in Docker Compose:

# docker-compose.yml
version: '3.8'
services:
  kafka:
    image: bitnami/kafka:3.7
    environment:
      KAFKA_CFG_PROCESS_ROLES: "broker,controller"
      KAFKA_CFG_NODE_ID: "1"
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093"
      KAFKA_CFG_LISTENERS: "PLAINTEXT://:9092,CONTROLLER://:9093"
      KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: "CONTROLLER"

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
      MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE: "health,info,prometheus"
    depends_on:
      - kafka

  kafka-ui-probe:
    image: python:3.12-slim
    ports:
      - "8090:8090"
    environment:
      KAFKA_UI_URL: "http://kafka-ui:8080"
      PROBE_PORT: "8090"
      EXPECTED_CLUSTERS: "local"
    command: ["sh", "-c", "pip install requests -q && python /probe/kafka_ui_probe.py"]
    volumes:
      - ./kafka_ui_probe.py:/probe/kafka_ui_probe.py
    depends_on:
      - kafka-ui

Step 3: Monitor Consumer Group Lag Display Health

Consumer group lag is one of Kafka UI's most-used views. Verify it returns data:

# Test consumer group API for a cluster named "local"
curl -s http://localhost:8080/api/clusters/local/consumer-groups | jq 'length'

# Inspect lag for a specific group
curl -s "http://localhost:8080/api/clusters/local/consumer-groups/my-consumer-group" \
  | jq '{groupId:.groupId, status:.state, partitionAssignmentStrategy:.partitionAssignmentStrategy}'

The Vigilmon monitor at /health/kafka-ui/consumers?cluster=local probes this path automatically.

Step 4: Configure Vigilmon Monitors

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your probe: https://your-host.example.com/health/kafka-ui
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your Slack or PagerDuty integration

Complete monitor table:

| Monitor URL | Purpose | Interval | Alert | |---|---|---|---| | /health/kafka-ui | Full health check | 1 min | P1 – PagerDuty | | http://kafka-ui:8080/actuator/health | Spring Boot process health | 30 sec | P1 – PagerDuty | | /health/kafka-ui/clusters | Cluster connectivity status | 1 min | P1 – PagerDuty | | /health/kafka-ui/consumers?cluster=local | Consumer group data availability | 2 min | P2 – Slack | | /health/kafka-ui/metrics?cluster=local | Broker metrics display health | 5 min | P2 – Slack | | /health/kafka-ui/topics?cluster=local | Topic list API health | 5 min | P2 – Slack |

Step 5: Heartbeat Monitor for Kafka UI

Use a Vigilmon heartbeat to detect silent JVM crashes between HTTP check intervals:

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name it kafka-ui-liveness
  3. Expected interval: 3 minutes, grace period: 6 minutes
  4. Copy the heartbeat URL

Send the heartbeat from a health-gated script:

# heartbeat.sh — run every 2 minutes via cron or Docker CMD wrapper
#!/bin/bash
KAFKA_UI_URL="${KAFKA_UI_URL:-http://localhost:8080}"
VIGILMON_URL="${VIGILMON_HEARTBEAT_URL}"

health=$(curl -sf --max-time 5 "$KAFKA_UI_URL/actuator/health" 2>/dev/null)
if echo "$health" | grep -q '"status":"UP"'; then
  curl -sf "$VIGILMON_URL" || true
fi

For Kubernetes:

# kafka-ui-heartbeat-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kafka-ui-heartbeat
  namespace: kafka
spec:
  schedule: "*/2 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: heartbeat
              image: curlimages/curl:8.5.0
              command:
                - sh
                - -c
                - |
                  health=$(curl -sf --max-time 5 "$KAFKA_UI_URL/actuator/health")
                  if echo "$health" | grep -q '"status":"UP"'; then
                    curl -sf "$VIGILMON_HEARTBEAT_URL"
                  fi
              env:
                - name: KAFKA_UI_URL
                  value: "http://kafka-ui.kafka.svc:8080"
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: kafka-ui-heartbeat-url

Step 6: Multi-Cluster Alert Strategy

For Kafka UI deployments managing multiple clusters, configure per-cluster monitors:

# Check each cluster individually
for cluster in production staging dev; do
  echo "Cluster: $cluster"
  curl -s "http://localhost:8080/api/clusters/$cluster/metrics" \
    | jq '{brokerCount:.brokerCount, activeControllers:.activeControllers}'
done

Vigilmon alert routing by cluster criticality:

| Cluster | Monitor | Alert Channel | Priority | |---|---|---|---| | production | /health/kafka-ui/clusters + all sub-checks | PagerDuty | P1 | | staging | /health/kafka-ui/clusters | Slack | P2 | | dev | Heartbeat only | Slack | P3 |

Summary

Kafka UI by Provectus is the most capable open-source Kafka management console, but its multi-service dependencies (Kafka, Schema Registry, Kafka Connect) mean a single downstream failure can degrade the UI in ways that aren't obvious from the outside. The Vigilmon monitors in this tutorial give you full coverage: Spring Boot process health, cluster connectivity, consumer group data availability, broker metrics, and a heartbeat liveness check.

| Monitor | What It Catches | |---|---| | Spring Boot actuator | JVM crashes, OOM restarts, startup failures | | /health/kafka-ui | Cluster offline, consumer data unavailable | | /health/kafka-ui/clusters | Any cluster disconnected from Kafka UI | | /health/kafka-ui/consumers | Consumer lag view serving errors | | /health/kafka-ui/metrics | Broker JMX metrics zeroed or unavailable | | Heartbeat: kafka-ui-liveness | Silent crashes between HTTP check windows |

Get started free at vigilmon.online — Kafka UI monitoring deployed and running in under five minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles