tutorial

Monitoring Kuma with Vigilmon: Control Plane Health, Data Plane Proxy Status, mTLS Policy & Multi-Zone Connectivity

How to monitor Kuma with Vigilmon — control plane health checks, data plane proxy status, mTLS policy enforcement verification, traffic metrics collection, and zone connectivity for multi-cluster deployments.

Kuma is the CNCF service mesh by Kong that brings mTLS, traffic policies, observability, and multi-zone networking to both Kubernetes and VM workloads through an Envoy-based data plane and a unified control plane. When Kuma's control plane becomes unavailable, data plane proxies (kumalets and kuma-dp) stop receiving configuration updates — existing connections continue with cached policy, but new policy changes stop applying and new services cannot join the mesh. When a zone control plane loses connectivity to the global control plane in a multi-zone deployment, cross-zone traffic silently fails. When mTLS policy enforcement breaks, services that should be rejecting unauthenticated connections start accepting them. Vigilmon gives you external visibility into every layer of your Kuma deployment: control plane health, data plane proxy connectivity, mTLS policy outcomes, and cross-zone reachability.

What You'll Build

  • HTTP monitors on Kuma's control plane API and GUI endpoints
  • Data plane proxy status checks to detect unhealthy or disconnected Envoy sidecars
  • mTLS connectivity verification through service-to-service endpoint monitoring
  • Zone connectivity monitoring for multi-zone Kuma deployments
  • Traffic metrics monitoring via Kuma's Prometheus integration
  • An alerting runbook mapping Kuma failure modes to the right control plane resources

Prerequisites

  • A Kubernetes cluster with Kuma 2.x installed (via kumactl or Helm chart)
  • Kuma control plane running in the kuma-system namespace
  • At least one mesh with data plane proxies (kuma-dp sidecars) injected
  • Kuma GUI and API accessible externally (via Kubernetes Service or Ingress)
  • A free account at vigilmon.online

Step 1: Understand Kuma's Health Endpoints

Kuma's control plane exposes several health and status endpoints that reflect its operational state:

# Check control plane health
kubectl -n kuma-system port-forward svc/kuma-control-plane 5681:5681 &

# Main health endpoint
curl http://localhost:5681/healthz
# Returns: HTTP 200 with JSON health status

# Control plane info and version
curl http://localhost:5681/
# Returns: Kuma version and feature flags

# List all data plane proxies and their status
curl http://localhost:5681/meshes/default/dataplanes+insights | jq .
# Each data plane shows subscriptions with lastConnectedTime

# Check zone control plane status (multi-zone only)
curl http://localhost:5681/zones | jq '.items[].status'

The control plane API at port 5681 (HTTP) and 5682 (HTTPS) is the primary observability surface. Data plane proxy status is particularly important — a proxy with total: 0 subscriptions or a lastConnectedTime far in the past indicates that the Envoy sidecar has lost contact with the control plane.

For external monitoring, Kuma's GUI (served on port 5681 at /gui) and API are the right targets:

# GUI health (if exposed externally)
curl https://kuma.example.com/gui
# Returns HTML with Kuma GUI

# API health
curl https://kuma.example.com/healthz

Step 2: Monitor the Kuma Control Plane API

The control plane API is the authoritative source of Kuma's operational status. When the control plane API is down, new policy changes cannot be applied, new services cannot join the mesh, and the GUI is unavailable:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://kuma.example.com/healthz (your externally exposed Kuma control plane health endpoint).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Label: Kuma control plane API.
  7. Click Save.

Also monitor the control plane's version endpoint, which indicates the API is fully initialized and serving requests (not just passing a shallow health check):

  1. Add Monitor → HTTP.
  2. URL: https://kuma.example.com/.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: version (appears in the JSON version response from a healthy control plane).
  7. Label: Kuma control plane version API.
  8. Click Save.

Alert after: 1 consecutive failure for the health endpoint. Data plane proxies cache policy and can continue operating for a period after control plane loss, but new deployments and policy changes are blocked immediately.

Step 3: Monitor the Kuma GUI

The Kuma GUI provides the dashboard that platform and operations teams use to inspect mesh health, traffic policies, and data plane proxy status. A GUI failure is distinct from an API failure and indicates a rendering or serving issue:

  1. Add Monitor → HTTP.
  2. URL: https://kuma.example.com/gui.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Kuma (appears in the GUI page title and content).
  7. Label: Kuma GUI dashboard.
  8. Click Save.

When the GUI monitor fires but the /healthz API monitor stays green, the issue is in the static asset serving layer or a GUI-specific configuration — not the control plane itself. This distinction helps you avoid unnecessary control plane restarts when the fix is simpler.

Step 4: Monitor Services Running Inside the Mesh

Services inside a Kuma mesh communicate through Envoy sidecar proxies (kuma-dp). The most effective way to monitor mesh health from the outside is to monitor the external-facing endpoints of services in the mesh — the same endpoints your users or API consumers reach. When mTLS, traffic policies, or proxy issues affect service-to-service communication, the external endpoint response degrades:

# Verify a mesh service is reachable externally
curl https://api.example.com/health
# This request enters the mesh through a Gateway/Ingress and reaches the target service via kuma-dp

# Check data plane proxy status for a specific deployment
kubectl -n production get pods -l app=order-service -o jsonpath='{.items[0].metadata.name}'
kubectl -n production exec myapp-pod -c kuma-sidecar -- \
  curl http://localhost:19901/ready
# Envoy admin API — returns "LIVE" when proxy is healthy

For each mesh-internal service exposed externally:

  1. Add Monitor → HTTP.
  2. URL: https://api.example.com/health (your application's externally reachable health endpoint).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: healthy (or your app's health response).
  7. Label: order-service (Kuma mesh health).
  8. Click Save.

When this monitor fires, the failure could be in the application, the kuma-dp sidecar, an mTLS policy, a traffic permission policy, or the upstream service the application calls. The Vigilmon alert is your starting point; kumactl inspect dataplanes --mesh default gives you the proxy-level detail.

Step 5: Verify mTLS Policy Enforcement

Kuma's mTLS feature ensures that all service-to-service traffic within a mesh is encrypted and mutually authenticated. When mTLS is enabled, services should reject unauthenticated connections. Verifying mTLS from outside the mesh requires checking that plain HTTP connections to mesh services fail as expected:

# This should FAIL when mTLS is enforced (connection refused or TLS error)
curl http://order-service.production.svc.cluster.local:8080/health
# Expected: connection refused, TLS handshake failure, or 404

# This should SUCCEED (via the mesh proxy with valid mTLS certificate)
curl https://api.example.com/health

Use Vigilmon to verify that your external HTTPS endpoint requires TLS (confirming the mesh is encrypting edge traffic):

  1. Add Monitor → SSL Certificate.
  2. Domain: api.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days.
  5. Click Save.

Certificate rotation: Kuma's mTLS certificates are automatically rotated by the control plane. Rotation failures are rare but can occur when the control plane is under resource pressure or when etcd is slow. If your mesh services start returning TLS errors while the control plane appears healthy, check kumactl get secrets --mesh default for certificate expiry times and review control plane logs for rotation errors.

Step 6: Monitor Multi-Zone Connectivity

In multi-zone Kuma deployments, a global control plane coordinates multiple zone control planes (each running in a separate Kubernetes cluster or datacenter). Cross-zone service communication depends on zone ingress/egress gateways and the global–zone control plane connection:

# Check zone status from the global control plane
kumactl get zones
# STATUS column should show "Online" for all zones

# Inspect zone control plane connectivity
curl https://kuma-global.example.com/zones | jq '.items[] | {name: .name, online: .status.zones[0].active}'

For each zone's ingress gateway (which handles cross-zone traffic entry):

  1. Add Monitor → HTTP.
  2. URL: https://zone-us-east.kuma.example.com/healthz (your zone ingress gateway health endpoint, if exposed).
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Label: Kuma zone-us-east ingress gateway.
  7. Click Save.

For the zone control plane's connectivity to the global control plane, monitor cross-zone service reachability directly:

# Test a cross-zone service call
curl https://api.example.com/cross-zone-test
# If zone-us-east loses contact with global, this fails even if the service runs in zone-us-east

Add a Vigilmon monitor on a cross-zone endpoint that exercises inter-zone routing:

  1. Add Monitor → HTTP.
  2. URL: https://api.example.com/cross-zone-status.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Label: Cross-zone service routing (Kuma multi-zone).
  6. Click Save.

Step 7: Monitor Traffic Metrics via Prometheus

Kuma integrates with Prometheus to expose traffic metrics for every data plane proxy in the mesh. These metrics reveal request success rates, latency percentiles, and connection counts for each service pair:

# Kuma data plane proxy Prometheus metrics
# Each pod exposes metrics at port 5670 by default
kubectl -n production exec myapp-pod -c kuma-sidecar -- \
  curl http://localhost:5670/metrics | grep -E 'envoy_cluster_upstream_rq_total|envoy_http_downstream_rq_5xx'

Key Kuma-specific Prometheus queries for your dashboards:

# Cross-service request error rate
sum(rate(envoy_cluster_upstream_rq_xx{envoy_response_code_class="5"}[5m]))
  by (kuma_io_service)
/
sum(rate(envoy_cluster_upstream_rq_total[5m]))
  by (kuma_io_service)

# P99 latency between services
histogram_quantile(0.99,
  sum(rate(envoy_cluster_upstream_rq_time_bucket[5m]))
    by (kuma_io_service, le)
)

# Data plane connection count to control plane
sum(envoy_control_plane_connected_state) by (kuma_io_service)
# Should be 1.0 for all proxies; 0 means proxy lost control plane connection

Monitor Prometheus itself (the dependency for Kuma metric evaluation):

  1. Add Monitor → HTTP.
  2. URL: https://prometheus.example.com/-/healthy.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: Prometheus Server is Healthy.
  6. Label: Prometheus (Kuma metrics backend).
  7. Click Save.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels with clear runbook steps:

| Monitor | Trigger | Immediate action | |---|---|---| | Control plane /healthz | Non-200 | Check kubectl -n kuma-system get pods; control plane pod may be crash-looping; data planes continue on cached policy | | Control plane version API | Non-200 or version missing | Control plane degraded; new policy changes and service joins blocked | | Kuma GUI | Non-200 | GUI serving issue; investigate static asset serving; API may still be healthy | | Mesh service endpoint | Non-200 | Run kumactl inspect dataplanes --mesh default; check proxy subscription status | | Zone ingress gateway | Non-200 | Cross-zone traffic to this zone affected; check global–zone control plane sync | | Cross-zone routing | Non-200 | Global–zone connectivity failure; run kumactl get zones to identify disconnected zone | | SSL certificate | < 30 days | Check cert-manager or Kuma's mTLS certificate rotation; verify kumactl get secrets | | Prometheus | Non-200 | Traffic policy metric evaluation may be affected; restore Prometheus for full observability |

Common Kuma Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Control plane pod OOM killed | Health API fires; existing proxies continue on cached policy; new services cannot join | | Zone loses connection to global control plane | Cross-zone routing monitor fires; zone-local services may still work | | mTLS certificate rotation failure | Mesh services start returning TLS errors; external endpoint monitor fires | | Data plane proxy crash-loop in a pod | Mesh service endpoint fires; other services in mesh unaffected | | Kuma operator missed policy update (stale Envoy config) | Traffic behaves according to old policy; external endpoint may show intermittent errors | | Zone ingress gateway pod evicted | Zone ingress monitor fires; cross-zone routing to that zone fails | | Node replacement removes kuma-dp with high connection count | Service endpoint may show brief errors during proxy re-registration | | Prometheus unavailable | Metrics observability lost; Kuma traffic policies still enforce; external endpoint unaffected | | DNS misconfiguration for mesh gateway hostname | All external monitors for that hostname fire simultaneously | | Global control plane database (PostgreSQL/etcd) slow | Control plane API responds slowly; health check may timeout |

Kuma's multi-zone, multi-mesh architecture creates a rich operational surface — the global and zone control planes, data plane proxies in every pod, mTLS certificate management, and cross-zone ingress/egress gateways all need independent health visibility. Vigilmon gives you external eyes on every layer: the control plane API that policy changes flow through, the mesh-internal service endpoints that mTLS and traffic permissions protect, zone ingress gateways that enable cross-cluster communication, and the SSL certificates that secure external access to your mesh. When Kuma's internal observability doesn't surface a failure in time, Vigilmon's external checks will.

Start monitoring your Kuma service mesh in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles