tutorial

How to Monitor Open Service Mesh with Vigilmon (HTTP + TCP + Alerts)

Open Service Mesh (OSM) is a CNCF lightweight service mesh that implements the Service Mesh Interface (SMI) specification. It runs an osm-controller pod that...

Open Service Mesh (OSM) is a CNCF lightweight service mesh that implements the Service Mesh Interface (SMI) specification. It runs an osm-controller pod that serves as the central control plane, uses Envoy as the sidecar proxy, and manages traffic policies through SMI-compliant TrafficTarget, TrafficSplit, and HTTPRouteGroup resources. When the osm-controller crashes, all proxy configuration updates stop — sidecar proxies freeze on stale policy, new pods receive no configuration, and traffic access control enforced through SMI TrafficTargets is no longer updated. Because OSM enforces a default-deny traffic policy, a stale controller means legitimate service-to-service calls start failing as soon as any policy change is missed, yet every workload pod continues to report Running and pass its health probes.

In this tutorial you'll set up comprehensive uptime monitoring for your Open Service Mesh deployment using Vigilmon — free tier, no credit card required.

Why OSM deployments need external monitoring

Open Service Mesh introduces failure modes that standard Kubernetes monitoring cannot surface:

  • osm-controller crash or OOMKill — the control plane pod that pushes xDS configuration to all Envoy sidecars exits; existing connections continue on frozen policy but new pods get no SMI policy, and default-deny rules block their traffic immediately; all workload pods stay Running while legitimate requests begin failing
  • osm-injector failure — the webhook that injects Envoy sidecar containers into new pods stops responding; newly deployed pods launch without a sidecar and bypass all mesh traffic policies, including mTLS and access control
  • SMI TrafficTarget misconfiguration — a TrafficTarget resource change blocks a service-to-service path at the proxy layer; the source and destination pods are both Running and healthy, but all requests return 403 with no error in application logs
  • Ingress controller / OSM Ingress Backend failure — the IngressBackend resource that allows the ingress controller to route traffic into the mesh is misconfigured or deleted; external traffic reaches the ingress pod but is blocked at the Envoy sidecar
  • Certificate rotation failure — OSM uses its own CA to issue mTLS certificates for each service; if the osm-bootstrap certificate or the osm-controller's cert-manager integration fails, certificates expire silently and mTLS connections begin failing across the mesh

These failures share a pattern: pod-level health is green while traffic is broken. External monitoring is the only way to validate the actual traffic path end-to-end.

What you'll need

  • A Kubernetes cluster with OSM installed (osm install or Helm)
  • A mesh-managed workload accessible from outside the cluster via an Ingress or LoadBalancer
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose a health endpoint through the OSM mesh

OSM uses SMI resources to control traffic. Deploy a probe service inside a mesh-managed namespace and expose it through an Ingress with an OSM IngressBackend resource.

First, verify your OSM control plane is running:

# Check OSM control plane pods
kubectl get pods -n osm-system
# NAME                              READY   STATUS    RESTARTS   AGE
# osm-controller-7d8f9b6c4-vr2mn   1/1     Running   0          3d
# osm-injector-5c7d8e9f2-tn4pk     1/1     Running   0          3d
# osm-bootstrap-6b4c9d7f8-wx1qr    1/1     Running   0          3d

# Check OSM mesh config
kubectl get meshconfig -n osm-system osm-mesh-config -o yaml | grep -A5 sidecarLogLevel

# Check which namespaces are part of the mesh
osm namespace list
# NAMESPACE       MESH     SIDECAR-INJECTION
# my-mesh-app     osm      enabled

Deploy a probe service into a mesh-managed namespace:

# osm-probe.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: osm-probe
  namespace: my-mesh-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: osm-probe
  template:
    metadata:
      labels:
        app: osm-probe
    spec:
      containers:
        - name: probe
          image: nginx:alpine
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
      volumes:
        - name: config
          configMap:
            name: osm-probe-nginx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: osm-probe-nginx
  namespace: my-mesh-app
data:
  default.conf: |
    server {
      listen 8080;
      location /healthz {
        return 200 '{"status":"ok","component":"osm-probe","mesh":"osm"}';
        add_header Content-Type application/json;
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: osm-probe
  namespace: my-mesh-app
spec:
  selector:
    app: osm-probe
  ports:
    - port: 8080
      targetPort: 8080
      name: http

Create the SMI TrafficTarget to allow ingress traffic to reach the probe:

# osm-probe-policy.yaml
apiVersion: access.smi-spec.io/v1alpha3
kind: TrafficTarget
metadata:
  name: osm-probe-ingress-access
  namespace: my-mesh-app
spec:
  destination:
    kind: ServiceAccount
    name: osm-probe
    namespace: my-mesh-app
  rules:
    - kind: HTTPRouteGroup
      name: osm-probe-routes
      matches:
        - healthz
  sources:
    - kind: ServiceAccount
      name: ingress-nginx
      namespace: ingress-nginx
---
apiVersion: specs.smi-spec.io/v1alpha4
kind: HTTPRouteGroup
metadata:
  name: osm-probe-routes
  namespace: my-mesh-app
spec:
  matches:
    - name: healthz
      pathRegex: "/healthz"
      methods:
        - GET

Expose the probe through nginx-ingress with an OSM IngressBackend:

# osm-probe-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: osm-probe
  namespace: my-mesh-app
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  rules:
    - host: osm-probe.your-cluster.example.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: osm-probe
                port:
                  number: 8080
---
apiVersion: policy.openservicemesh.io/v1alpha1
kind: IngressBackend
metadata:
  name: osm-probe
  namespace: my-mesh-app
spec:
  backends:
    - name: osm-probe
      port:
        number: 8080
        protocol: http
  sources:
    - kind: Service
      namespace: ingress-nginx
      name: ingress-nginx-controller

kubectl apply -f osm-probe.yaml
kubectl apply -f osm-probe-policy.yaml
kubectl apply -f osm-probe-ingress.yaml

# Verify the probe pod has the Envoy sidecar injected
kubectl get pod -n my-mesh-app -l app=osm-probe -o jsonpath='{.items[0].spec.containers[*].name}'
# nginx osm-proxy-init osm-envoy-sidecar

# Confirm the health endpoint responds
curl http://osm-probe.your-cluster.example.com/healthz
# {"status":"ok","component":"osm-probe","mesh":"osm"}

Step 2: Set up HTTP monitoring in Vigilmon

With the probe accessible through the OSM mesh, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to your probe endpoint: https://osm-probe.your-cluster.example.com/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

This monitor validates the full OSM data path: the osm-controller must be distributing xDS config, the Envoy sidecar must be active, the SMI TrafficTarget must permit ingress access, the IngressBackend must be allowing the ingress controller through, and the workload pod must be healthy.

Why external monitoring catches what Kubernetes misses

OSM enforces a default-deny traffic model through Envoy proxy rules, not through Kubernetes NetworkPolicies. A pod crash or a TrafficTarget deletion blocks traffic at the proxy layer with no reflection in pod status. Vigilmon probes from outside the cluster over the public network, so it sees the actual reachable state of the service. Its multi-region consensus prevents false positives from transient network blips.

Step 3: Monitor the osm-controller and injector TCP health

Add monitors to detect control-plane failures independently of the data path:

osm-controller health port:

# osm-controller exposes a health endpoint on port 9091
kubectl port-forward -n osm-system deploy/osm-controller 9091:9091 &
curl http://localhost:9091/health/ready
# OK

Expose it externally through a NodePort Service and add an HTTP monitor in Vigilmon:

# osm-controller-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: osm-controller-health
  namespace: osm-system
spec:
  type: NodePort
  selector:
    app: osm-controller
  ports:
    - port: 9091
      targetPort: 9091
      nodePort: 30091
      name: health

kubectl apply -f osm-controller-health-svc.yaml

Add an HTTP monitor in Vigilmon for http://<node-ip>:30091/health/ready expecting status 200 and body OK.

Ingress controller TCP check:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host <ingress-external-ip>, Port 443
  4. Save the monitor
kubectl get svc -n ingress-nginx ingress-nginx-controller
# NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)
# ingress-nginx-controller   LoadBalancer   10.96.44.120    203.0.113.60    80:30080/TCP,443:30443/TCP

Step 4: Configure alert channels

An OSM control plane failure silently breaks traffic for every mesh-managed workload. Alerts need to reach your platform team in real time.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call email
  3. Assign the channel to all OSM monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload when the OSM probe goes down:

{
  "monitor_name": "OSM Probe Health",
  "status": "down",
  "url": "https://osm-probe.your-cluster.example.com/healthz",
  "started_at": "2024-01-15T16:45:00Z",
  "duration_seconds": 60
}

Run these diagnostics immediately when an alert fires:

# Check OSM control plane pod status
kubectl get pods -n osm-system

# Check osm-controller logs for xDS errors
kubectl logs -n osm-system deploy/osm-controller --tail=200 | grep -E "error|failed|warn"

# Check osm-injector logs for webhook failures
kubectl logs -n osm-system deploy/osm-injector --tail=100 | grep -E "error|inject"

# Check probe pod sidecar status
kubectl get pod -n my-mesh-app -l app=osm-probe -o wide

# Check SMI TrafficTarget for the probe
kubectl get traffictarget -n my-mesh-app osm-probe-ingress-access -o yaml

# Check IngressBackend status
kubectl get ingressbackend -n my-mesh-app osm-probe -o yaml | grep -A10 status

# Check Envoy proxy config sync status
osm proxy get config_dump -n my-mesh-app <pod-name> | python3 -m json.tool | grep -A3 cluster_name

# Check SMI policy for the mesh namespace
kubectl get traffictarget,httproutegroup -n my-mesh-app

# Check recent events
kubectl get events -n osm-system --sort-by='.lastTimestamp' | tail -20

Step 5: Create a public status page

OSM manages the connectivity layer for all mesh-enrolled workloads. A shared status page communicates mesh health to all dependent teams.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Open Service Mesh Status"
  3. Add your monitors: OSM Probe HTTP, osm-controller Health, Ingress TCP 443, Ingress TCP 80
  4. Group them: Data Path, Control Plane, Ingress Ports
  5. Publish the page

Putting it all together

Here's a summary of the monitors to create for an OSM deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | https://osm-probe.your-cluster.example.com/healthz | HTTP | Envoy sidecar, SMI TrafficTarget, IngressBackend, workload health | | http://<node-ip>:30091/health/ready | HTTP | osm-controller control plane readiness | | <ingress-ip>:443 | TCP | HTTPS ingress port reachability | | <ingress-ip>:80 | TCP | HTTP ingress port reachability |

And the OSM diagnostics to run when an alert fires:

# Quick triage: control plane or data plane?
kubectl get pods -n osm-system
osm namespace list

# Check xDS config distribution from osm-controller
kubectl logs -n osm-system deploy/osm-controller | grep -E "error|xds|LDS|RDS|CDS"

# Verify sidecar injection is enabled in target namespace
kubectl get namespace my-mesh-app --show-labels | grep openservicemesh.io/sidecar-injection

# Check TrafficTarget and access policy
kubectl get traffictarget -n my-mesh-app -o yaml

# Check certificate status
kubectl get secret -n my-mesh-app | grep osm-envoy-client-cert

# Check OSM mesh config for global policy
kubectl get meshconfig -n osm-system osm-mesh-config -o yaml | grep -A5 traffic

What's next

  • SSL certificate monitoring — add SSL monitors for any HTTPS endpoints exposed through OSM IngressBackend resources
  • Heartbeat monitoring — deploy a mesh-internal cronjob that makes a service-to-service call through the mesh on a schedule and sends a Vigilmon heartbeat on success, validating SMI TrafficTarget enforcement end-to-end
  • Certificate expiry monitoring — OSM-issued mTLS certificates have a configurable TTL; monitor each service's cert expiry date to catch renewal failures before traffic breaks

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles