Monitoring Envoy Gateway on Kubernetes with Vigilmon

Envoy Gateway is a Kubernetes-native implementation of the Gateway API spec built on top of Envoy Proxy. It replaces the Ingress resource with expressive Gateway and HTTPRoute objects while keeping Envoy's battle-tested data plane underneath. But Kubernetes-native doesn't mean self-monitoring — when the Envoy data plane fails, when the control plane stops reconciling, or when a certificate expires, your users experience it before your team does. Vigilmon adds external uptime monitoring that catches what cluster-internal tooling misses: the view from outside your cluster.

What You'll Set Up

• External HTTP uptime monitors for services routed through Envoy Gateway
• Gateway admin API monitoring for control plane health
• SSL certificate expiry alerts for Gateway TLS termination
• Heartbeat monitoring for cert-manager and Gateway reconciliation jobs

Prerequisites

• Envoy Gateway 1.x installed on a Kubernetes cluster (kubectl get gateways)
• At least one HTTPRoute routing traffic to a backend service
• A free Vigilmon account

Step 1: Monitor Your Routed Application Endpoints

Envoy Gateway's job is to route external traffic to backend services. Monitor the external endpoint of each routed service to confirm the full path — DNS, LoadBalancer, Envoy data plane, and backend — is working end-to-end:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the external URL of your service: https://api.example.com/health.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

This external check catches failures that cluster-internal probes (readiness/liveness) miss: a misconfigured HTTPRoute, a broken TLS certificate, a LoadBalancer IP that stopped responding, or an Envoy worker crash.

Repeat for each HTTPRoute that serves production traffic.

Step 2: Expose and Monitor the Envoy Admin Interface

Envoy exposes an admin interface at port 9901 (by default) that surfaces the proxy's internal health, statistics, and configuration. Envoy Gateway deploys this on each Envoy Pod. While you shouldn't expose this publicly, you can monitor it from within the cluster or via a port-forwarded probe:

Check the admin endpoint is healthy:

# Confirm admin interface is responding on an Envoy Gateway pod
kubectl port-forward -n envoy-gateway-system \
  deployment/envoy-gateway 9901:9901 &

curl http://localhost:9901/ready
# Returns "LIVE" when the server is ready to accept connections

For continuous monitoring, add a Vigilmon HTTP monitor that probes a known healthy route end-to-end rather than the admin port directly. The admin port should stay cluster-internal; use Vigilmon's external check as your proxy health signal.

Step 3: Add Health Endpoints to Backend Services

The external Vigilmon check only tells you whether Envoy Gateway can serve a response. Add a /health endpoint to each backend service so the check verifies the backend's internal state too:

// Go backend service
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)
    w.Write([]byte(`{"status":"ok"}`))
})

# HTTPRoute pointing Vigilmon-compatible health check to the backend
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-health-route
spec:
  parentRefs:
    - name: prod-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /health
      backendRefs:
        - name: api-service
          port: 8080

Now the Vigilmon probe at https://api.example.com/health traverses Envoy Gateway's full data path and verifies the backend in one check.

Step 4: SSL Certificate Expiry Alerts for Gateway TLS

Envoy Gateway terminates TLS using certificates stored in Kubernetes Secrets and referenced from Gateway resources. If cert-manager fails to renew a certificate, or if a manually-provisioned certificate expires, Envoy will start returning TLS errors. Add SSL monitoring before that happens:

1. Open the HTTP / HTTPS monitor for your Gateway endpoint (from Step 1).
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

For Gateways that serve multiple hostnames (via listeners), add a separate monitor for each hostname — each may have a different certificate:

# Gateway with multiple TLS listeners
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-gateway
spec:
  gatewayClassName: eg
  listeners:
    - name: api
      hostname: api.example.com
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - name: api-tls-cert
    - name: dashboard
      hostname: dashboard.example.com
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - name: dashboard-tls-cert

Create one Vigilmon SSL monitor for api.example.com and another for dashboard.example.com.

Step 5: Heartbeat Monitor for cert-manager Renewals

cert-manager renews certificates automatically, but the renewal process can fail silently: an ACME challenge that gets blocked, a DNS provider API key that rotates, or a webhook that stops responding. Monitor the renewal pipeline with a heartbeat by adding a post-renewal hook:

# cert-manager Certificate with a renewal hook via a CronJob
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls-cert
spec:
  secretName: api-tls-cert
  dnsNames:
    - api.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Add a Kubernetes CronJob that checks certificate freshness and pings Vigilmon:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cert-renewal-heartbeat
spec:
  schedule: "0 */6 * * *"  # every 6 hours
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: check-certs
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  # Verify the certificate was renewed in the last 80 days
                  EXPIRY=$(kubectl get secret api-tls-cert -o jsonpath='{.data.tls\.crt}' \
                    | base64 -d | openssl x509 -noout -enddate | cut -d= -f2)
                  echo "Certificate expires: $EXPIRY"
                  # Ping Vigilmon heartbeat
                  wget -q -O- https://vigilmon.online/heartbeat/abc123
          restartPolicy: OnFailure

Set the Vigilmon heartbeat interval to 7 hours to alert if two consecutive runs are missed.

Step 6: Monitor Gateway Reconciliation Health

Envoy Gateway's control plane continuously reconciles Gateway and HTTPRoute resources. If the control plane crashes or the leader election stalls, existing traffic continues (Envoy is already configured) but new routes stop working. Monitor the control plane with an HTTP check against its metrics endpoint:

# Expose the Envoy Gateway controller metrics
kubectl port-forward -n envoy-gateway-system \
  deployment/envoy-gateway 8080:8080 &

curl http://localhost:8080/healthz/ready
# Returns 200 when the controller is ready

For external monitoring, deploy a small probe service inside your cluster that polls the controller's /healthz/ready endpoint and exposes a public health URL. Then add a Vigilmon HTTP monitor for that public URL.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. For external endpoint monitors, set Consecutive failures before alert to 2 — Envoy rolling restarts can cause a single probe to fail.
3. For SSL monitors, set it to 1 — a failed TLS handshake is always actionable.
4. For heartbeat monitors (cert-manager), set it to 1 — a missed renewal check warrants immediate investigation.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Application endpoint | https://api.example.com/health | Data plane failure, routing misconfiguration | | SSL certificate | Each Gateway hostname | cert-manager failure, manual cert expiry | | cert-manager heartbeat | CronJob ping | Renewal pipeline failure, webhook outage | | Backend health | /health on each service | Service crash behind Gateway |

Envoy Gateway gives you expressive, Kubernetes-native traffic management — Vigilmon gives you the external perspective that confirms it's working from your users' point of view. Together they close the gap between "the Gateway resource is Accepted" and "traffic is actually flowing."