tutorial

How to Monitor Higress with Vigilmon (HTTP + TCP + Alerts)

Higress is a cloud-native API gateway and service mesh gateway open-sourced by Alibaba, built on Envoy and designed to unify Kubernetes Ingress, service mesh...

Higress is a cloud-native API gateway and service mesh gateway open-sourced by Alibaba, built on Envoy and designed to unify Kubernetes Ingress, service mesh traffic management, and API gateway functionality in a single control plane. It supports WASM plugins for custom routing logic, integrates with Istio for mesh traffic, and provides HTTP route management through Higress Ingress resources and McpBridge configurations. When the Higress controller crashes or its gateway pods fail, all inbound API traffic is silently dropped — not with error responses, but with connection refused or timeouts — while the workload pods behind the gateway continue to show Running and pass their readiness probes. Because Higress unifies multiple traffic layers, a single control-plane failure can affect both external API traffic and internal mesh routing simultaneously.

In this tutorial you'll set up comprehensive uptime monitoring for your Higress deployment using Vigilmon — free tier, no credit card required.

Why Higress deployments need external monitoring

Higress introduces failure modes that Kubernetes pod health cannot surface:

  • Higress controller crash — the controller that reconciles Higress Ingress and McpBridge resources and pushes configuration to Envoy gateway pods exits; existing connections continue on stale routing config but new Ingress rules are never applied; the gateway pods stay Running while routing for newly deployed services silently fails
  • Gateway pod OOMKill or crash — the Higress gateway pod that actually terminates inbound connections exits or gets stuck; all API traffic returns connection refused; backend pods are healthy but unreachable; Kubernetes reschedules the pod but there is a gap that users experience as downtime
  • WASM plugin load failure — a custom WASM plugin built into a Higress route fails to load or panics during request processing; affected routes return 500 errors while the gateway pod continues running; standard health checks never catch plugin-level failures
  • McpBridge configuration error — a misconfigured McpBridge resource prevents Higress from discovering backend services in a service registry (Nacos, Consul, Eureka, or Kubernetes); routes that target those services return 503 while the Higress controller and gateway pods report healthy
  • Certificate rotation failure — Higress manages TLS certificates for HTTPS routes; if cert-manager or a manual certificate secret rotation fails, HTTPS routes begin returning TLS handshake errors while HTTP routes continue to work, making the failure appear intermittent

These failures share a pattern: the Higress process is running and passes pod-level checks, but actual traffic is broken. External monitoring from Vigilmon probes the real traffic path and surfaces these failures before users do.

What you'll need

  • A Kubernetes cluster with Higress installed (via Helm or the Higress CLI)
  • A workload service exposed through a Higress Ingress resource
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose a health endpoint through Higress

Deploy a probe service and expose it through a Higress Ingress resource so the full Higress data path is validated.

First, verify your Higress deployment is running:

# Check Higress pods
kubectl get pods -n higress-system
# NAME                                    READY   STATUS    RESTARTS   AGE
# higress-controller-6d7f8b9c4-vr2mn     2/2     Running   0          2d
# higress-gateway-5b8c9d6f4-tn3pk        1/1     Running   0          2d

# Check Higress gateway service
kubectl get svc -n higress-system higress-gateway
# NAME              TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)
# higress-gateway   LoadBalancer   10.96.44.200    203.0.113.80    80:30080/TCP,443:30443/TCP

# Check Higress ingress resources
kubectl get ingress -A

Deploy a probe service in a target namespace:

# higress-probe.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: higress-probe
  namespace: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: higress-probe
  template:
    metadata:
      labels:
        app: higress-probe
    spec:
      containers:
        - name: probe
          image: nginx:alpine
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
      volumes:
        - name: config
          configMap:
            name: higress-probe-nginx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: higress-probe-nginx
  namespace: my-app
data:
  default.conf: |
    server {
      listen 8080;
      location /healthz {
        return 200 '{"status":"ok","component":"higress-probe","gateway":"higress"}';
        add_header Content-Type application/json;
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: higress-probe
  namespace: my-app
spec:
  selector:
    app: higress-probe
  ports:
    - port: 8080
      targetPort: 8080
      name: http

Create a Higress Ingress resource to route traffic through the gateway:

# higress-probe-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: higress-probe
  namespace: my-app
  annotations:
    higress.io/enable-cors: "false"
spec:
  ingressClassName: higress
  rules:
    - host: higress-probe.your-cluster.example.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: higress-probe
                port:
                  number: 8080
  tls:
    - hosts:
        - higress-probe.your-cluster.example.com
      secretName: higress-probe-tls

kubectl apply -f higress-probe.yaml
kubectl apply -f higress-probe-ingress.yaml

# Get the gateway external IP
kubectl get svc -n higress-system higress-gateway
# NAME              TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)
# higress-gateway   LoadBalancer   10.96.44.200    203.0.113.80   80:30080/TCP,443:30443/TCP

# Add DNS: higress-probe.your-cluster.example.com -> 203.0.113.80
# Then confirm the health endpoint responds
curl https://higress-probe.your-cluster.example.com/healthz
# {"status":"ok","component":"higress-probe","gateway":"higress"}

# Or test by IP using Host header
curl -H "Host: higress-probe.your-cluster.example.com" http://203.0.113.80/healthz
# {"status":"ok","component":"higress-probe","gateway":"higress"}

Step 2: Set up HTTP monitoring in Vigilmon

With the probe endpoint reachable through Higress, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to your probe endpoint: https://higress-probe.your-cluster.example.com/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

This monitor validates the complete Higress data path: the Higress controller must be reconciling Ingress resources, the gateway pod must be running and forwarding traffic, the Envoy routing config must be correct, and the backend service must be healthy. Any failure in this chain causes the monitor to alert.

Why external monitoring catches what Kubernetes misses

Higress gateway pods pass their readiness probes by checking that the Envoy process is listening on port 80, not that it's correctly routing requests. A gateway pod can show Ready: 1/1 while returning 503 for all routes because the Higress controller failed to push updated xDS config. Vigilmon's external HTTP probe validates the actual routing result — the response body and status code — not just the port-open check. Its multi-region consensus prevents false positives from transient network blips.

Step 3: Monitor the Higress gateway TCP ports

Add TCP monitors to detect network-level failures before HTTP routing is even attempted:

Gateway HTTPS port:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host 203.0.113.80, Port 443
  4. Save the monitor

Gateway HTTP port:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host 203.0.113.80, Port 80
  4. Save the monitor

Higress controller health endpoint:

Higress controller exposes a readiness endpoint. Expose it externally for monitoring:

# higress-controller-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: higress-controller-health
  namespace: higress-system
spec:
  type: NodePort
  selector:
    app: higress-controller
  ports:
    - port: 8888
      targetPort: 8888
      nodePort: 30888
      name: health

kubectl apply -f higress-controller-health-svc.yaml

# Verify the controller health endpoint
curl http://<node-ip>:30888/ready
# OK

Add an HTTP monitor in Vigilmon for http://<node-ip>:30888/ready expecting status 200.

Step 4: Configure alert channels

Higress is the entry point for all inbound API traffic. A gateway failure means every external API call fails immediately.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call email
  3. Assign the channel to all Higress monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload when the Higress gateway goes down:

{
  "monitor_name": "Higress Gateway Health",
  "status": "down",
  "url": "https://higress-probe.your-cluster.example.com/healthz",
  "started_at": "2024-01-15T18:00:00Z",
  "duration_seconds": 90
}

Run these diagnostics immediately when an alert fires:

# Check Higress pod status
kubectl get pods -n higress-system

# Check gateway pod logs for routing errors
kubectl logs -n higress-system deploy/higress-gateway --tail=200 | grep -E "error|503|404|upstream"

# Check controller logs for reconciliation errors
kubectl logs -n higress-system deploy/higress-controller -c higress-core --tail=200 | grep -E "error|failed|reconcil"

# Check if Ingress resource is registered
kubectl get ingress -n my-app higress-probe -o yaml | grep -A10 status

# Check xDS config sync from Higress controller
kubectl logs -n higress-system deploy/higress-controller -c discovery --tail=100 | grep -E "error|xds|LDS|RDS"

# Check McpBridge status (if using service registries)
kubectl get mcpbridge -n higress-system -o yaml | grep -A5 status

# Check Higress gateway Envoy admin interface
kubectl port-forward -n higress-system deploy/higress-gateway 15000:15000 &
curl http://localhost:15000/clusters | grep -i probe

# Check certificate secrets
kubectl get secret -n my-app higress-probe-tls -o yaml | grep -A2 tls.crt

# Check recent events
kubectl get events -n higress-system --sort-by='.lastTimestamp' | tail -20

Step 5: Create a public status page

Higress serves as the API gateway for multiple downstream consumers. A public status page gives them visibility into gateway health.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Higress API Gateway Status"
  3. Add your monitors: Higress Probe HTTP, Controller Health HTTP, Gateway TCP 443, Gateway TCP 80
  4. Group them: API Gateway, Control Plane, Network
  5. Publish the page

Share this URL with API consumers and internal teams whose services depend on inbound routing through Higress.

Putting it all together

Here's a summary of the monitors to create for a Higress deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | https://higress-probe.your-cluster.example.com/healthz | HTTP | Gateway routing, Ingress config, Envoy xDS sync, backend health | | http://<node-ip>:30888/ready | HTTP | Higress controller readiness | | 203.0.113.80:443 | TCP | HTTPS gateway port reachability | | 203.0.113.80:80 | TCP | HTTP gateway port reachability |

And the Higress diagnostics to run when an alert fires:

# Quick triage: gateway pod or controller issue?
kubectl get pods -n higress-system
kubectl get ingress -A

# Check Envoy routing config on gateway pod
kubectl exec -n higress-system deploy/higress-gateway -- curl -s localhost:15000/routes | python3 -m json.tool | grep -A5 higress-probe

# Check gateway access log for upstream errors
kubectl logs -n higress-system deploy/higress-gateway | grep "NR\|UH\|UF\|UC" | tail -20

# Check controller for failed Ingress reconciliation
kubectl logs -n higress-system deploy/higress-controller -c higress-core | grep -E "error|Ingress|reconcil"

# Check WASM plugin status if plugins are configured
kubectl get wasmplugin -n higress-system -o wide

# Verify TLS certificate is valid
echo | openssl s_client -connect higress-probe.your-cluster.example.com:443 -servername higress-probe.your-cluster.example.com 2>/dev/null | openssl x509 -noout -dates

# Check McpBridge for service registry failures
kubectl describe mcpbridge -n higress-system

What's next

  • SSL certificate monitoring — add SSL monitors for each HTTPS route managed by Higress to catch certificate expiry before users see TLS errors
  • Heartbeat monitoring — deploy a periodic job that calls a key API endpoint through Higress and sends a Vigilmon heartbeat on success, validating the full routing chain including WASM plugin execution
  • Per-route coverage — for production APIs with strict SLAs, create a dedicated Vigilmon monitor for each critical route rather than relying on a single probe, so alerts pinpoint which route is degraded rather than just that the gateway is partially broken

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles