tutorial

How to Monitor Kiali with Vigilmon (HTTP + TCP + Alerts)

Kiali is the observability and management console for Istio service meshes. It provides service topology graphs, traffic flow visualization, Istio configurat...

Kiali is the observability and management console for Istio service meshes. It provides service topology graphs, traffic flow visualization, Istio configuration validation, and health dashboards by scraping Prometheus metrics and querying the Istio control plane and Kubernetes API. When Kiali's deployment crashes or its Prometheus connection breaks, platform teams lose their primary window into mesh health — traffic flow graphs go blank, configuration validation stops, and mesh misconfiguration accumulates silently because nobody has visibility into what Istio is actually doing. Kiali's own pod status tells you only whether the process is alive; it doesn't tell you whether its external connections to Prometheus, Istio, and Jaeger are functioning.

In this tutorial you'll set up comprehensive uptime monitoring for your Kiali deployment using Vigilmon — free tier, no credit card required.

Why Kiali needs external monitoring

Kiali's architecture depends on several external integrations that can fail independently of the Kiali pod itself:

  • Kiali pod crash or OOMKill — the main Kiali server process exits; the mesh continues to function but all observability access is gone; developers cannot validate Istio configuration, view service graphs, or diagnose traffic anomalies until someone notices the pod is down
  • Prometheus connection failure — Kiali queries a Prometheus instance for all its metrics data; if Prometheus is unreachable, restarts, or its storage fills up, Kiali's UI returns empty graphs and degraded health indicators while showing "connected" on its status page
  • Kubernetes API access revoked — Kiali's ServiceAccount requires read access to Istio CRDs, Services, and Pods; an RBAC policy change or namespace restriction silently breaks Kiali's namespace list, making entire namespaces disappear from the topology view
  • Jaeger/Zipkin trace integration failure — distributed tracing stops appearing in Kiali when the tracing backend is unreachable; teams lose the ability to correlate mesh errors with application traces, increasing mean-time-to-diagnose for production incidents
  • Kiali Operator upgrade failure — the Kiali Operator that reconciles the KialiCR can get stuck during an upgrade; the Kiali deployment is left in an inconsistent state with some pods running old and some running new versions, causing UI instability

These failures share a pattern: the mesh itself keeps working, but the team's ability to observe and manage it goes dark. External monitoring ensures Kiali's own health is continuously validated.

What you'll need

  • An Istio service mesh (Istio upstream, OSSM, or similar) with Kiali installed
  • A Kiali instance accessible from outside the cluster via a Route, Ingress, or NodePort
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose Kiali's health endpoint

Kiali exposes a /kiali/healthz endpoint on its HTTP port (default 20001) that returns its own health status and the status of its external integrations.

First, verify your Kiali deployment is running:

# Check the Kiali pod
kubectl get pods -n istio-system -l app=kiali
# NAME                     READY   STATUS    RESTARTS   AGE
# kiali-7b4c8f9d6-tn2wp   1/1     Running   0          4d

# Check the Kiali service
kubectl get svc -n istio-system kiali
# NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)
# kiali   ClusterIP   10.96.55.130   <none>        20001/TCP

# Check Kiali health directly from inside the cluster
kubectl exec -n istio-system deploy/kiali -- curl -s http://localhost:20001/kiali/healthz
# {"status":"ok"}

Expose Kiali externally through an Ingress so Vigilmon can reach it:

# kiali-external-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kiali-external
  namespace: istio-system
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /kiali/healthz
spec:
  ingressClassName: nginx
  rules:
    - host: kiali-health.your-cluster.example.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: kiali
                port:
                  number: 20001
  tls:
    - hosts:
        - kiali-health.your-cluster.example.com
      secretName: kiali-health-tls

kubectl apply -f kiali-external-ingress.yaml

# Confirm the health endpoint responds
curl https://kiali-health.your-cluster.example.com/healthz
# {"status":"ok"}

If you are running on OpenShift with OSSM, Kiali is already exposed via a Route:

# Find the existing Kiali route
oc get route -n istio-system kiali
# NAME    HOST/PORT                                    SERVICES   PORT   TERMINATION
# kiali   kiali-istio-system.apps.mycluster.example.com   kiali      http   edge/Redirect

# Test the Kiali health endpoint through the route
curl https://kiali-istio-system.apps.mycluster.example.com/kiali/healthz
# {"status":"ok"}

Use https://kiali-istio-system.apps.mycluster.example.com/kiali/healthz as your monitor URL in this case.

Step 2: Set up HTTP monitoring in Vigilmon

With the Kiali health endpoint accessible, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to your Kiali health endpoint: https://kiali-health.your-cluster.example.com/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

This monitor validates that Kiali's web server is up and responding. If the pod crashes, OOMKills, or if the Ingress/Route routing breaks, Vigilmon catches it within a minute.

Add a monitor for Kiali's integration health

Kiali also exposes a /kiali/api/istio/status endpoint that reports the status of all its external integrations — Prometheus, Istio, Jaeger, and Grafana. Add a second monitor targeting this endpoint:

  1. Create another HTTP / HTTPS monitor
  2. Set the URL to: https://kiali-health.your-cluster.example.com/api/istio/status
  3. Set check interval to 5 minutes
  4. Under Expected response:
    • Status code: 200
  5. Save the monitor

Update the Ingress to forward this path too:

# Add to kiali-external-ingress.yaml paths:
- path: /api/istio/status
  pathType: Exact
  backend:
    service:
      name: kiali
      port:
        number: 20001

Why external monitoring catches what Kubernetes misses

Kiali's pod readiness probe only checks whether the HTTP server can accept connections on port 20001. It does not check whether Prometheus is reachable, whether Istio API calls succeed, or whether the Kubernetes RBAC permissions are intact. Vigilmon's external probe goes beyond pod-level health to validate the observable state of the service. Its multi-region consensus prevents false positives from transient DNS or network blips.

Step 3: Monitor Kiali's TCP port

Add a TCP monitor to detect network-level failures before HTTP-layer validation:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host <ingress-external-ip>, Port 443
  4. Save the monitor

Also add a direct TCP check for Kiali's native port if the pod is accessible via NodePort:

# Expose Kiali as a NodePort for direct TCP monitoring (optional)
kubectl patch svc kiali -n istio-system -p '{"spec":{"type":"NodePort","ports":[{"port":20001,"targetPort":20001,"nodePort":32001}]}}'

Add a TCP monitor: Host <node-ip>, Port 32001.

Step 4: Configure alert channels

Kiali is the primary observability window for your mesh. When it goes down, your team is flying blind during any incident that involves mesh traffic.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call email
  3. Assign the channel to all Kiali monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload when Kiali goes down:

{
  "monitor_name": "Kiali Health Check",
  "status": "down",
  "url": "https://kiali-health.your-cluster.example.com/healthz",
  "started_at": "2024-01-15T09:15:00Z",
  "duration_seconds": 180
}

Run these diagnostics immediately when an alert fires:

# Check Kiali pod status and recent restarts
kubectl get pods -n istio-system -l app=kiali
kubectl describe pod -n istio-system -l app=kiali

# Check Kiali logs for errors
kubectl logs -n istio-system deploy/kiali --tail=200 | grep -E "error|failed|prometheus|istio"

# Check Kiali ConfigMap for integration config
kubectl get configmap -n istio-system kiali -o yaml | grep -A10 external_services

# Check Prometheus connectivity from within the cluster
kubectl exec -n istio-system deploy/kiali -- curl -s http://prometheus:9090/-/healthy

# Check Kiali ServiceAccount RBAC
kubectl auth can-i get pods --as=system:serviceaccount:istio-system:kiali -n istio-system

# Check Kiali Operator status (if using Kiali Operator)
kubectl get kiali -n istio-system -o yaml | grep -A20 status

# Check events in istio-system
kubectl get events -n istio-system --sort-by='.lastTimestamp' | grep -i kiali | tail -20

# Verify Kiali custom resource
kubectl get kiali -n istio-system

Step 5: Create a public status page

Kiali is the shared observability tool for all teams running workloads in the mesh. A status page communicates its availability to all stakeholders.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Kiali Service Mesh Console Status"
  3. Add your monitors: Kiali Health HTTP, Kiali Integrations HTTP, Kiali TCP
  4. Group them: Web Console, Integrations, Network
  5. Publish the page

Putting it all together

Here's a summary of the monitors to create for a Kiali deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | https://kiali-health.your-cluster.example.com/healthz | HTTP | Kiali pod health, Ingress/Route routing | | https://kiali-health.your-cluster.example.com/api/istio/status | HTTP | Prometheus, Istio, Jaeger integration connectivity | | <ingress-ip>:443 | TCP | HTTPS ingress port reachability | | <node-ip>:32001 | TCP | Kiali native port direct reachability |

And the Kiali diagnostics to run when an alert fires:

# Quick triage: pod crash or integration failure?
kubectl get pods -n istio-system -l app=kiali
kubectl logs -n istio-system deploy/kiali --tail=50

# Check Prometheus integration
kubectl exec -n istio-system deploy/kiali -- wget -qO- http://prometheus.istio-system:9090/-/ready

# Check Kiali's view of Istio API health
kubectl exec -n istio-system deploy/kiali -- curl -s http://localhost:20001/kiali/api/istio/status | python3 -m json.tool

# Verify RBAC is intact
kubectl auth can-i list virtualservices.networking.istio.io --as=system:serviceaccount:istio-system:kiali

# Check for Kiali Operator reconciliation issues
kubectl logs -n kiali-operator deploy/kiali-operator --tail=100 | grep -E "error|reconcil"

# Verify Kiali ConfigMap is valid
kubectl get cm -n istio-system kiali -o yaml | python3 -c "import sys,yaml; yaml.safe_load(sys.stdin)" && echo "ConfigMap valid"

What's next

  • SSL certificate monitoring — add an SSL monitor for Kiali's HTTPS endpoint to catch certificate expiry before users start seeing browser warnings
  • Prometheus monitoring — since Kiali depends entirely on Prometheus data, add a dedicated Vigilmon monitor for your Prometheus instance to catch failures before they propagate to Kiali
  • Heartbeat monitoring — deploy a script that authenticates to Kiali's API, fetches a service graph, and sends a Vigilmon heartbeat on success, validating that the full Kiali UI path including Istio and Prometheus integration is working end-to-end

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles