tutorial

How to Monitor OpenShift Service Mesh with Vigilmon (HTTP + TCP + Alerts)

OpenShift Service Mesh (OSSM) is Red Hat's enterprise distribution of Istio, bundled with Kiali, Jaeger, and Prometheus and integrated into the OpenShift pla...

OpenShift Service Mesh (OSSM) is Red Hat's enterprise distribution of Istio, bundled with Kiali, Jaeger, and Prometheus and integrated into the OpenShift platform through the Service Mesh Operator. OSSM runs as a set of control-plane pods — istiod, the Ingress Gateway, and the Egress Gateway — alongside a per-namespace Kiali and Jaeger instance. When any of these control-plane components fail, sidecar proxies lose configuration updates, traffic policies stop being enforced, mTLS breaks silently, and workloads that rely on virtual services and destination rules continue receiving traffic with no mesh-level circuit breaking or retries. Neither OpenShift's cluster health checks nor Kubernetes node conditions surface these failures; they appear as subtle latency increases or authentication errors rather than pod crashes.

In this tutorial you'll set up comprehensive uptime monitoring for your OpenShift Service Mesh deployment using Vigilmon — free tier, no credit card required.

Why OSSM deployments need external monitoring

OpenShift Service Mesh introduces a class of failure that OpenShift's built-in monitoring cannot reliably surface:

  • istiod crashes or OOMKills — the Pilot component that distributes xDS configuration to all Envoy sidecars exits; existing connections continue on stale configuration but new pods receive no mesh policy; circuit breakers and retries silently stop working across all namespaces in the mesh
  • Ingress Gateway pod failures — the OSSM Ingress Gateway that terminates external HTTPS traffic and routes it into the mesh goes CrashLoopBackOff; all traffic to mesh-managed services returns 503 while individual workload pods remain Running and show healthy readiness probes
  • mTLS policy misconfiguration — a PeerAuthentication or DestinationRule update sets STRICT mTLS on a namespace containing a legacy non-mesh workload; traffic is silently dropped at the proxy layer with no application-level error surfaced in standard pod logs
  • Kiali or Jaeger operator degradation — the observability stack that feeds service topology and trace data stops collecting; teams lose visibility into mesh health without knowing the collection pipeline is broken
  • Service Mesh Member Roll misconfiguration — a namespace is removed from or never added to the ServiceMeshMemberRoll; workloads in that namespace stop getting sidecar injection and traffic policy without any warning

These failures are particularly dangerous because all workload pods continue to report Running and pass their readiness probes. External monitoring from Vigilmon probes the actual traffic path from outside the cluster and surfaces failures before users do.

What you'll need

  • An OpenShift cluster with OpenShift Service Mesh Operator installed
  • A mesh-managed workload exposed through the OSSM Ingress Gateway via a Route or LoadBalancer
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose a health endpoint through the OSSM Ingress Gateway

OSSM routes external traffic through an Ingress Gateway. Deploy a probe service that returns mesh health status and expose it via a VirtualService and Gateway resource.

First, verify your OSSM control plane is running:

# Check OSSM control plane pods
oc get pods -n istio-system
# NAME                                    READY   STATUS    RESTARTS   AGE
# istiod-basic-7d9f6b8c4-xk2mn           1/1     Running   0          5d
# istio-ingressgateway-5b7c9d6f8-vr3lp   1/1     Running   0          5d
# istio-egressgateway-6c8d4e7b2-mn9qr    1/1     Running   0          5d
# kiali-7b4c8f9d6-tn2wp                  1/1     Running   0          5d

# Check ServiceMeshControlPlane status
oc get smcp -n istio-system
# NAME    READY   STATUS            PROFILES      VERSION   AGE
# basic   10/10   ComponentsReady   ["default"]   2.5.1     5d

# Check the Ingress Gateway route
oc get route -n istio-system istio-ingressgateway
# NAME                    HOST/PORT                                        PATH   SERVICES               PORT    TERMINATION   WILDCARD
# istio-ingressgateway    istio-ingressgateway-istio-system.apps.mycluster.example.com   ...

Deploy a probe service into a mesh-managed namespace:

# ossm-probe.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ossm-probe
  namespace: my-mesh-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ossm-probe
  template:
    metadata:
      labels:
        app: ossm-probe
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
        - name: probe
          image: nginx:alpine
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
      volumes:
        - name: config
          configMap:
            name: ossm-probe-nginx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ossm-probe-nginx
  namespace: my-mesh-app
data:
  default.conf: |
    server {
      listen 8080;
      location /healthz {
        return 200 '{"status":"ok","component":"ossm-probe","mesh":"basic"}';
        add_header Content-Type application/json;
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: ossm-probe
  namespace: my-mesh-app
spec:
  selector:
    app: ossm-probe
  ports:
    - port: 8080
      targetPort: 8080
      name: http

Expose it through the OSSM Gateway and VirtualService:

# ossm-probe-gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: ossm-probe-gateway
  namespace: my-mesh-app
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "ossm-probe.apps.mycluster.example.com"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ossm-probe
  namespace: my-mesh-app
spec:
  hosts:
    - "ossm-probe.apps.mycluster.example.com"
  gateways:
    - ossm-probe-gateway
  http:
    - route:
        - destination:
            host: ossm-probe
            port:
              number: 8080

oc apply -f ossm-probe.yaml
oc apply -f ossm-probe-gateway.yaml

# Create the OpenShift Route for the Ingress Gateway
oc create route edge ossm-probe \
  --service=istio-ingressgateway \
  --hostname=ossm-probe.apps.mycluster.example.com \
  -n istio-system

# Confirm the health endpoint responds through the mesh
curl https://ossm-probe.apps.mycluster.example.com/healthz
# {"status":"ok","component":"ossm-probe","mesh":"basic"}

Step 2: Set up HTTP monitoring in Vigilmon

With the probe endpoint accessible through the OSSM Ingress Gateway, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to your probe endpoint: https://ossm-probe.apps.mycluster.example.com/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

This single monitor validates the entire OSSM data path: the Ingress Gateway pod must be running, the VirtualService route must be active, the sidecar proxy must be accepting traffic, and the workload pod must be healthy. If any of these fail, Vigilmon catches it within a minute.

Why external monitoring catches what OpenShift misses

OpenShift's built-in alerting fires on pod crashes and resource thresholds, but OSSM failures often don't crash pods — they silently break traffic policy. A misconfigured DestinationRule blocks traffic at the Envoy proxy layer while the application pod remains Running and its readiness probe continues to return 200. Vigilmon's external probe traverses the full traffic path — DNS, TLS, Ingress Gateway routing, VirtualService rules, and sidecar proxy — and detects failures that OpenShift's internal probes cannot see.

Step 3: Monitor the istiod control plane and Ingress Gateway TCP ports

Add monitors to detect control-plane and gateway-level failures independently:

istiod health endpoint:

Expose the istiod health port through a Service and Route:

# istiod listens on port 15021 for health checks
oc expose svc/istiod -n istio-system --port=15021 --name=istiod-health \
  --hostname=istiod-health.apps.mycluster.example.com

Add an HTTP monitor in Vigilmon for https://istiod-health.apps.mycluster.example.com/healthz/ready expecting status 200.

Ingress Gateway TCP check:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host <ingress-gateway-external-ip>, Port 443
  4. Save the monitor

Get the external IP:

oc get svc istio-ingressgateway -n istio-system
# NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)
# istio-ingressgateway   LoadBalancer   172.30.44.120   203.0.113.50    80/TCP,443/TCP

Also add a TCP monitor for port 80 to catch routing failures before TLS termination.

Step 4: Configure alert channels

An OSSM failure silently degrades all traffic for every mesh-managed workload. Alerts must reach the platform team immediately.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call email
  3. Assign the channel to all OSSM monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload when the OSSM probe goes down:

{
  "monitor_name": "OSSM Ingress Gateway Health",
  "status": "down",
  "url": "https://ossm-probe.apps.mycluster.example.com/healthz",
  "started_at": "2024-01-15T14:30:00Z",
  "duration_seconds": 120
}

Run these diagnostics immediately when an alert fires:

# Check control plane pod status
oc get pods -n istio-system

# Check istiod logs for xDS errors
oc logs -n istio-system deploy/istiod-basic --tail=200 | grep -E "error|failed|warn"

# Check Ingress Gateway logs for routing errors
oc logs -n istio-system deploy/istio-ingressgateway --tail=200 | grep -E "error|503|404"

# Check ServiceMeshControlPlane status
oc describe smcp -n istio-system basic

# Check ServiceMeshMemberRoll for namespace membership
oc get smmr -n istio-system default -o yaml | grep -A20 status

# Check VirtualService routing rules
oc get virtualservice -n my-mesh-app ossm-probe -o yaml

# Check Envoy proxy config sync status
oc exec -n my-mesh-app deploy/ossm-probe -c istio-proxy -- pilot-agent request GET /debug/syncz | head -20

# Check recent events in mesh namespace
oc get events -n istio-system --sort-by='.lastTimestamp' | tail -20

Step 5: Create a public status page

Service Mesh platforms power multiple teams' workloads. A shared status page gives all dependent teams visibility into mesh health.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "OpenShift Service Mesh Status"
  3. Add your monitors: OSSM Probe HTTP, istiod Health, Ingress Gateway TCP 443, Ingress Gateway TCP 80
  4. Group them: Data Path, Control Plane, Gateway Ports
  5. Publish the page

Share this URL with application teams whose workloads run inside the mesh.

Putting it all together

Here's a summary of the monitors to create for an OSSM deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | https://ossm-probe.apps.mycluster.example.com/healthz | HTTP | Ingress Gateway, VirtualService routing, sidecar proxy, workload health | | https://istiod-health.apps.mycluster.example.com/healthz/ready | HTTP | istiod control plane readiness | | <ingress-ip>:443 | TCP | HTTPS gateway port reachability | | <ingress-ip>:80 | TCP | HTTP gateway port reachability |

And the OSSM diagnostics to run when an alert fires:

# Quick triage: control plane or data plane?
oc get pods -n istio-system
oc get smcp -n istio-system

# Check istiod xDS distribution errors
oc logs -n istio-system deploy/istiod-basic | grep -E "error|LDS|RDS|CDS|EDS"

# Check Ingress Gateway Envoy access log for 503s
oc logs -n istio-system deploy/istio-ingressgateway | grep " 503 "

# Verify sidecar injection is active in target namespace
oc get namespace my-mesh-app -o yaml | grep injection

# Check DestinationRule mTLS settings
oc get destinationrule -n my-mesh-app -o yaml | grep -A5 tls

# Verify PeerAuthentication policy
oc get peerauthentication -n my-mesh-app -o yaml

# Check member roll status
oc get smmr -n istio-system

What's next

  • SSL certificate monitoring — add SSL monitors for the OSSM Ingress Gateway's TLS certificate to catch expiry before it breaks all HTTPS traffic
  • Heartbeat monitoring — deploy a mesh-internal job that validates east-west service-to-service calls on a schedule and sends a Vigilmon heartbeat on success, catching mTLS breakage between services
  • Per-namespace coverage — create a dedicated probe deployment per mesh namespace to isolate which namespace's traffic policy is broken when an alert fires

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles