tutorial

Monitoring Kafka UI (Provectus) with Vigilmon

Kafka UI is the primary operational dashboard for self-hosted Kafka clusters. Learn how to monitor its application health, Kafka broker connectivity, consumer lag display, and connector status with Vigilmon.

Kafka UI by Provectus is the go-to open source management interface for self-hosted Apache Kafka clusters. Engineering teams rely on it daily for browsing messages, monitoring consumer group lag, managing Kafka Connect connectors, and browsing Schema Registry schemas. When Kafka UI goes down, operators lose their primary visibility tool — and problems in the underlying Kafka cluster go undetected longer.

Vigilmon gives you external health monitoring for Kafka UI itself: application uptime, Kafka broker connectivity, consumer lag display health, and connector status — so you know immediately when your observability tooling is down.

Why Kafka UI Needs Its Own Monitoring

It's easy to monitor Kafka and forget to monitor the tool you use to monitor Kafka. But Kafka UI failures have real operational impact:

  • Blind spots accumulate — consumer group lag climbs undetected while operators assume the dashboard would have shown it
  • Connector failures go unnoticed — a Kafka Connect connector in FAILED state may not trigger your alerting if Kafka UI is the only place it's visible
  • Schema Registry issues stay hidden — schema evolution errors are only surfaced through Kafka UI's schema browser
  • Multi-cluster failures cascade — if Kafka UI loses connectivity to one of several monitored clusters, the loss is invisible until someone opens that cluster's tab

External monitoring from Vigilmon covers Kafka UI application health independently of the Kafka cluster itself — and monitors Kafka cluster connectivity through Kafka UI's actuator endpoints.

What You'll Set Up

  • HTTP uptime monitor for Kafka UI's Spring Boot actuator health endpoint (port 8080)
  • Kafka broker connectivity health check via Kafka UI's own health probe
  • Consumer lag display health monitoring
  • Kafka Connect connector status integration
  • Memory usage alert for heap exhaustion prevention
  • Alert routing by severity

Step 1: Monitor the Kafka UI Application Health

Kafka UI exposes Spring Boot Actuator endpoints. The /actuator/health endpoint returns the application status including Kafka connectivity:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your Kafka UI URL: http://your-kafkaui-host:8080/actuator/health
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Response body contains, enter "status":"UP".
  7. Set Response time threshold to 10000ms (Spring Boot startup and Kafka connectivity checks can take several seconds).
  8. Click Save.

If your Kafka UI deployment uses a custom management port (set via management.server.port in application configuration), replace 8080 with that port.

For Kafka UI instances behind an nginx reverse proxy or authentication middleware, point the monitor at the exposed health path and configure any required headers under Vigilmon's Request headers setting.

Step 2: Build a Kafka Broker Connectivity Health Endpoint

Kafka UI's actuator health endpoint includes Kafka broker connectivity status, but you may want a more granular check. Add a lightweight sidecar that queries Kafka UI's internal cluster status API:

Node.js Broker Connectivity Sidecar

// kafkaui-health.js
const express = require('express');
const axios = require('axios');

const app = express();

const KAFKAUI_BASE = process.env.KAFKAUI_URL || 'http://localhost:8080';
const KAFKAUI_CLUSTERS = (process.env.KAFKAUI_CLUSTERS || 'local').split(',');

app.get('/health/kafkaui/brokers', async (req, res) => {
  const results = [];
  let allHealthy = true;

  for (const cluster of KAFKAUI_CLUSTERS) {
    try {
      const resp = await axios.get(
        `${KAFKAUI_BASE}/api/clusters/${cluster}/brokers`,
        { timeout: 8000 }
      );
      const brokers = resp.data;
      const brokerCount = Array.isArray(brokers) ? brokers.length : 0;

      if (brokerCount === 0) {
        allHealthy = false;
        results.push({ cluster, status: 'no_brokers' });
      } else {
        results.push({ cluster, status: 'ok', broker_count: brokerCount });
      }
    } catch (err) {
      allHealthy = false;
      results.push({ cluster, status: 'unreachable', error: err.message });
    }
  }

  const statusCode = allHealthy ? 200 : 503;
  res.status(statusCode).json({ status: allHealthy ? 'ok' : 'degraded', clusters: results });
});

app.get('/health/kafkaui/consumer-lag', async (req, res) => {
  const cluster = process.env.KAFKAUI_PRIMARY_CLUSTER || 'local';
  const lagThreshold = parseInt(process.env.CONSUMER_LAG_THRESHOLD || '10000');

  try {
    const resp = await axios.get(
      `${KAFKAUI_BASE}/api/clusters/${cluster}/consumer-groups`,
      { timeout: 10000 }
    );
    const groups = resp.data;

    const laggingGroups = groups.filter(g =>
      g.messagesBehind && g.messagesBehind > lagThreshold
    );

    if (laggingGroups.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'consumer_lag_threshold_exceeded',
        lagging_groups: laggingGroups.map(g => ({
          name: g.groupId,
          lag: g.messagesBehind,
        })),
      });
    }

    return res.json({ status: 'ok', groups_monitored: groups.length });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/kafkaui/connectors', async (req, res) => {
  const cluster = process.env.KAFKAUI_PRIMARY_CLUSTER || 'local';

  try {
    const resp = await axios.get(
      `${KAFKAUI_BASE}/api/clusters/${cluster}/connects/connectors`,
      { timeout: 8000 }
    );
    const connectors = resp.data || [];

    const failedConnectors = connectors.filter(c =>
      c.status && c.status.connector && c.status.connector.state === 'FAILED'
    );

    if (failedConnectors.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'connectors_in_failed_state',
        failed: failedConnectors.map(c => c.name),
      });
    }

    return res.json({ status: 'ok', connector_count: connectors.length });
  } catch (err) {
    // If Kafka Connect is not configured, this endpoint returns 404 — treat as healthy
    if (err.response && err.response.status === 404) {
      return res.json({ status: 'ok', kafka_connect: 'not_configured' });
    }
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(8099, () => console.log('Kafka UI health sidecar on :8099'));

Run this sidecar on the same host as Kafka UI:

node kafkaui-health.js

Add Vigilmon monitors for each endpoint:

  • http://your-kafkaui-host:8099/health/kafkaui/brokers — broker connectivity across all clusters
  • http://your-kafkaui-host:8099/health/kafkaui/consumer-lag — consumer lag threshold breaches
  • http://your-kafkaui-host:8099/health/kafkaui/connectors — connector FAILED state detection

Step 3: Monitor Kafka UI Memory Usage

Kafka UI is a Spring Boot application that caches Kafka metadata, message browse results, and consumer group lag data in JVM heap. Heap exhaustion causes OOM crashes and silent restarts — particularly in multi-cluster configurations with many topics and consumer groups.

Add a memory health check:

app.get('/health/kafkaui/memory', async (req, res) => {
  try {
    const resp = await axios.get(
      `${KAFKAUI_BASE}/actuator/metrics/jvm.memory.used`,
      { timeout: 5000 }
    );
    const used = resp.data.measurements.find(m => m.statistic === 'VALUE')?.value || 0;

    const maxResp = await axios.get(
      `${KAFKAUI_BASE}/actuator/metrics/jvm.memory.max`,
      { timeout: 5000 }
    );
    const max = maxResp.data.measurements.find(m => m.statistic === 'VALUE')?.value || 1;

    const usagePct = Math.round((used / max) * 100);
    const threshold = parseInt(process.env.HEAP_THRESHOLD_PCT || '80');

    if (usagePct > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'heap_pressure',
        heap_used_pct: usagePct,
        threshold_pct: threshold,
      });
    }

    return res.json({ status: 'ok', heap_used_pct: usagePct });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

This check requires Spring Boot Actuator metrics to be enabled (the default in Kafka UI). Add a Vigilmon monitor for http://your-kafkaui-host:8099/health/kafkaui/memory.

Step 4: Heartbeat for Schema Registry Integration

If Kafka UI is configured to display Schema Registry schemas, a Schema Registry outage causes Kafka UI's schema browser to fail silently. Add a schema registry health check:

app.get('/health/kafkaui/schema-registry', async (req, res) => {
  const schemaRegistryUrl = process.env.SCHEMA_REGISTRY_URL || '';
  if (!schemaRegistryUrl) {
    return res.json({ status: 'ok', schema_registry: 'not_configured' });
  }

  try {
    const resp = await axios.get(`${schemaRegistryUrl}/subjects`, { timeout: 5000 });
    return res.json({ status: 'ok', schema_count: Array.isArray(resp.data) ? resp.data.length : 0 });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

Step 5: Configure Alert Channels and Routing

Open Vigilmon's Alert Channels and configure routing for each monitor:

| Monitor | Alert Channel | Suggested Priority | |---|---|---| | Kafka UI actuator /actuator/health | Slack + PagerDuty | P1 — dashboard down | | Broker connectivity /health/kafkaui/brokers | Slack + PagerDuty | P1 — Kafka cluster unreachable | | Consumer lag /health/kafkaui/consumer-lag | Slack | P2 — lag threshold exceeded | | Connectors /health/kafkaui/connectors | Slack | P2 — connector in FAILED state | | Memory /health/kafkaui/memory | Slack | P2 — heap pressure before OOM | | Schema Registry /health/kafkaui/schema-registry | Email | P3 — schema browser broken |

Recommended settings:

  • Consecutive failures before alert: 2 — Spring Boot health checks can be slow on first probe after GC pause
  • Response time threshold: 10000ms for actuator health (cluster metadata fetch can be slow)
  • Maintenance windows: suppress alerts during Kafka UI version upgrades and Kafka cluster maintenance

Step 6: TCP Port Monitor for Fast-Path Detection

Add a TCP monitor to detect Kafka UI process crashes before the HTTP health probe can fire:

  1. In Vigilmon, click Add Monitor → TCP Port.
  2. Host: your-kafkaui-host, Port: 8080 (or your management port).
  3. Interval: 1 minute.
  4. Alert channel: same P1 channel as the actuator health monitor.

TCP monitors fire within one check interval of a JVM crash — faster than waiting for an HTTP timeout.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP — actuator health | :8080/actuator/health | Kafka UI application down | | HTTP — broker connectivity | :8099/health/kafkaui/brokers | Kafka cluster unreachable from Kafka UI | | HTTP — consumer lag | :8099/health/kafkaui/consumer-lag | Consumer lag exceeding threshold | | HTTP — connectors | :8099/health/kafkaui/connectors | Kafka Connect connector in FAILED state | | HTTP — memory | :8099/health/kafkaui/memory | JVM heap pressure before OOM crash | | HTTP — Schema Registry | :8099/health/kafkaui/schema-registry | Schema Registry connectivity loss | | TCP — port | :8080 | JVM crash, fast-path detection |

Kafka UI is the visibility layer your Kafka operations depend on — which makes monitoring the monitoring tool essential. With Vigilmon watching Kafka UI's application health, broker connectivity, consumer lag display, and connector status, you catch infrastructure issues before they become invisible problems.

Get started free at vigilmon.online — your first Kafka UI monitor is up in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles