tutorial

Monitoring Envoy Proxy with Vigilmon

Envoy is the data plane powering Istio and cloud-native service meshes — when it goes down or loses xDS sync, your entire microservices routing layer breaks. Here's how to monitor Envoy readiness, cluster health, and xDS configuration sync with Vigilmon.

Envoy Proxy is the CNCF-graduated edge and service mesh proxy that powers Istio, Ambassador, and AWS App Mesh — and increasingly, standalone edge deployments that need HTTP/2, gRPC, circuit breaking, and rich observability out of the box. Originally built at Lyft, Envoy has become the standard data plane for cloud-native microservices architectures. When Envoy is healthy, it transparently routes traffic, enforces retries and circuit breakers, and exports detailed metrics. When it's not — whether due to a failed xDS configuration sync, a crashed sidecar, or an expired TLS certificate — downstream services stop receiving traffic and the failure can be opaque. Vigilmon monitors Envoy's readiness endpoint, admin API health, cluster states, and xDS configuration freshness so you catch proxy failures before they cascade through your service mesh.

What You'll Set Up

  • HTTP monitor for Envoy readiness endpoint (/ready)
  • HTTP monitor for Envoy server info and state (/server_info)
  • Heartbeat for cluster health and xDS configuration freshness
  • SSL certificate alerts for HTTPS listener certificates

Prerequisites

  • Envoy running with the admin interface enabled (port 9901 by default)
  • Access to the Envoy admin endpoint from your monitoring infrastructure
  • A free Vigilmon account

Step 1: Monitor Envoy Readiness

Envoy's /ready endpoint is the canonical liveness signal. It returns HTTP 200 with the body LIVE when Envoy has fully initialized and is ready to serve traffic. During startup (while waiting for xDS configuration from the control plane), it returns HTTP 503 with PRE_INITIALIZING or INITIALIZING.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the Envoy admin URL: http://your-envoy-host:9901/ready
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Keyword monitoring, enable it and set the keyword to LIVE.
  7. Click Save.

The body keyword LIVE distinguishes a fully initialized Envoy from one still waiting for xDS resources:

# Healthy Envoy
curl http://localhost:9901/ready
# LIVE

# Envoy still initializing
curl http://localhost:9901/ready
# PRE_INITIALIZING
# (HTTP 503)

If your Envoy is behind a firewall that blocks direct access to port 9901, use an SSH tunnel or expose the admin port on a restricted internal interface only — the admin API has no authentication.

Security note: The Envoy admin interface (/ready, /config_dump, /clusters, etc.) has no built-in authentication. Restrict access to 127.0.0.1 or a trusted management network. Never expose it on a public interface.

Step 2: Monitor Envoy Server State

The /server_info endpoint provides a JSON view of Envoy's operational state, including whether it has fully loaded all xDS resources and how long it has been running. This catches a broader set of issues than /ready alone — including partial initialization states and configuration load errors.

  1. Click Add Monitor → set Type to HTTP / HTTPS.
  2. Enter: http://your-envoy-host:9901/server_info
  3. Set Check interval to 2 minutes.
  4. Enable Keyword monitoring and set the keyword to "state": "LIVE".
  5. Click Save.

The /server_info response includes:

{
  "version": "1.29.0/...",
  "state": "LIVE",
  "uptime_current_epoch": 86400,
  "uptime_all_epochs": 172800,
  "hot_restart_version": "...",
  "command_line_options": {}
}

The "state" field transitions from "PRE_INITIALIZING" → "INITIALIZING" → "LIVE" as Envoy loads its configuration. Monitoring for "state": "LIVE" (with the quotes, to match the JSON field exactly) ensures you only pass the check when Envoy is fully operational.

The uptime_current_epoch vs uptime_all_epochs difference reveals hot restarts: if uptime_current_epoch is much lower than uptime_all_epochs, Envoy has recently performed a hot restart. This is normal for configuration reloads but unexpected restarts during normal operation indicate process crashes.

Step 3: Heartbeat for Cluster Health

Envoy's /clusters endpoint shows the health state of all upstream cluster backends. When all hosts in a cluster are UNHEALTHY (or healthy_weight == 0), Envoy can't route traffic to that cluster and will return 503 errors or apply circuit breaker responses.

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 5 minutes.
  3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Create /usr/local/bin/envoy-cluster-check.sh:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
ADMIN_URL="http://localhost:9901"

# Check if Envoy is ready
READY=$(curl -s -o /dev/null -w "%{http_code}" "$ADMIN_URL/ready")
if [ "$READY" != "200" ]; then
  echo "Envoy not ready: HTTP $READY"
  exit 1
fi

# Get cluster health summary
CLUSTERS=$(curl -s "$ADMIN_URL/clusters")
if [ $? -ne 0 ]; then
  echo "Failed to fetch cluster info"
  exit 1
fi

# Look for clusters where all hosts are unhealthy
# The /clusters endpoint returns text format; check for healthy host counts
UNHEALTHY_CLUSTERS=$(echo "$CLUSTERS" | grep -E "^[^:]+::[^:]+::health_flags::/failed_active_hc" | wc -l)

if [ "$UNHEALTHY_CLUSTERS" -eq 0 ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
else
  echo "Unhealthy cluster hosts detected: $UNHEALTHY_CLUSTERS"
fi

chmod +x /usr/local/bin/envoy-cluster-check.sh

Add to cron:

*/5 * * * * root /usr/local/bin/envoy-cluster-check.sh

For Envoy deployments with the admin API returning JSON format (using ?format=json), parse the structured output:

# Get clusters in JSON format
curl -s "http://localhost:9901/clusters?format=json" | \
  jq '[.cluster_statuses[] | select(.host_statuses[]?.health_status.eds_health_status != "HEALTHY")] | length'

Step 4: Heartbeat for xDS Configuration Freshness

In Istio or other xDS-based deployments, Envoy must stay synchronized with the control plane. If the xDS connection breaks (due to network partition, control plane restart, or certificate rotation), Envoy continues serving traffic with stale configuration — but new routes, service registrations, and policy changes stop propagating. In long-running stale-config scenarios, Envoy's routing state diverges from the actual service topology.

  1. Add a second Cron Heartbeat monitor with a 15 minute interval.
  2. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/def456).

Create /usr/local/bin/envoy-xds-check.sh:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
ADMIN_URL="http://localhost:9901"
MAX_STALE_SECONDS=300  # Alert if config hasn't updated in 5 minutes

# Get config dump and check last_updated timestamp
CONFIG_DUMP=$(curl -s "$ADMIN_URL/config_dump?include_eds" 2>/dev/null)
if [ $? -ne 0 ]; then
  echo "Failed to fetch config dump"
  exit 1
fi

# Extract the most recent last_updated timestamp from dynamic resources
LAST_UPDATED=$(echo "$CONFIG_DUMP" | jq -r '
  [
    .configs[]
    | .dynamic_listeners[]?.active_state.last_updated // empty,
      .dynamic_route_configs[]?.last_updated // empty,
      .dynamic_clusters[]?.last_updated // empty
  ]
  | sort
  | last
' 2>/dev/null)

if [ -z "$LAST_UPDATED" ]; then
  # No dynamic resources — static config only, always fresh
  curl -s "$HEARTBEAT_URL" > /dev/null
  exit 0
fi

# Convert last_updated to epoch (format: "2024-01-15T10:30:00.000Z")
LAST_EPOCH=$(date -d "$LAST_UPDATED" +%s 2>/dev/null)
NOW_EPOCH=$(date +%s)
STALE_SECONDS=$((NOW_EPOCH - LAST_EPOCH))

if [ "$STALE_SECONDS" -le "$MAX_STALE_SECONDS" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
else
  echo "xDS config stale: last updated ${STALE_SECONDS}s ago"
fi

chmod +x /usr/local/bin/envoy-xds-check.sh

Add to cron:

*/15 * * * * root /usr/local/bin/envoy-xds-check.sh

For standalone Envoy deployments without xDS (static configuration only), skip this check — static configs are always "current" by definition. Instead, monitor uptime_current_epoch from /server_info to detect unexpected restarts:

# Alert if Envoy has restarted in the last 15 minutes (epoch < 900s)
UPTIME=$(curl -s "http://localhost:9901/server_info" | jq -r '.uptime_current_epoch')
if [ "$UPTIME" -gt 900 ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
else
  echo "Envoy restarted recently: uptime=${UPTIME}s"
fi

Step 5: SSL Certificate Alerts for HTTPS Listeners

Envoy handles TLS termination for downstream clients and can serve certificates statically (from files) or dynamically via SDS (Secret Discovery Service). While SDS enables hot certificate rotation, a misconfigured SDS push or a static certificate that reaches expiry leaves Envoy serving an expired certificate to all downstream clients.

For each HTTPS listener port, add a certificate monitor:

  1. Click Add Monitor → set Type to HTTP / HTTPS.
  2. Enter the URL for your HTTPS listener: https://your-envoy-host:443/
  3. Enable Monitor SSL certificate under the SSL section.
  4. Set Alert when certificate expires in less than 21 days.
  5. Click Save.

For Envoy sidecars in Istio, the certificate rotation is managed by Istio's CA (istiod/Citadel) with a default 24-hour TTL. These short-lived certificates are automatically rotated, but if the rotation fails, they expire quickly. Monitor the workload certificate on each sidecar port:

# Check Envoy sidecar certificate expiry
openssl s_client -connect pod-ip:8080 </dev/null 2>/dev/null | openssl x509 -noout -enddate

# Or via the Envoy admin cert info
curl -s http://localhost:9901/certs | jq '.certificates[] | {path: .ca_cert[].path, days_until_expiry: ((.ca_cert[].expiration_time | fromdateiso8601) - now | . / 86400 | floor)}'

Step 6: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, PagerDuty, or a webhook.
  2. For the /ready monitor, set Consecutive failures before alert to 1 — a non-LIVE Envoy means traffic routing is broken.
  3. For /server_info, set consecutive failures to 2 — allow one miss for hot restarts.
  4. For heartbeat monitors (cluster health, xDS freshness), alerts fire automatically when the ping interval passes.
  5. For SSL certificates, set the alert window to 21 days minimum (shorter for Istio short-lived certs — set to 1 day and alert on any rotation failure).

For Kubernetes deployments, integrate with your alerting stack:

# Check Envoy sidecar status across all pods in a namespace
kubectl get pods -n production -o json | jq -r '
  .items[]
  | select(.status.containerStatuses[]?.name == "istio-proxy")
  | select(.status.containerStatuses[]? | select(.name == "istio-proxy") | .ready == false)
  | .metadata.name
'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP keyword | host:9901/ready → LIVE | Initialization failure, crash | | HTTP keyword | host:9901/server_info → "state": "LIVE" | Partial init, config load failure | | Cluster heartbeat | /clusters unhealthy host flags | All backends for a cluster down | | xDS heartbeat | /config_dump last_updated age | Control plane disconnect, stale config | | SSL certificate | HTTPS listener ports | TLS cert expiry on load-balanced traffic |

Envoy's power comes from its deep integration with the control plane — xDS dynamic routing, circuit breaking, and distributed tracing all depend on the proxy staying in sync with its configuration source. With Vigilmon monitoring readiness, server state, cluster health, xDS freshness, and certificate expiry, you get the observability layer that your service mesh's data plane deserves.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles