tutorial

How to Monitor Consul Connect Service Mesh with Vigilmon

HashiCorp Consul Connect is the service mesh layer built into Consul, enabling automatic mTLS encryption and authorization between microservices. It uses sid...

HashiCorp Consul Connect is the service mesh layer built into Consul, enabling automatic mTLS encryption and authorization between microservices. It uses sidecar proxies (typically Envoy) to intercept and secure all service-to-service traffic without requiring application code changes. For teams running microservices on-premise or across multi-cloud deployments, Consul Connect provides a production-grade alternative to cloud-native service meshes like AWS App Mesh or Istio.

But a service mesh adds layers — Consul servers, sidecar proxies, certificate authorities, intentions (authorization policies), and the services themselves — and each layer can fail independently. This tutorial shows you how to monitor your entire Consul Connect deployment end-to-end with Vigilmon.

Why Consul Connect needs external monitoring

Consul Connect's production failure modes span multiple layers:

  • Consul server quorum loss — Consul uses Raft consensus. If a majority of servers are unavailable, the cluster stops accepting writes. Service registration, health checks, and intention updates all fail silently from the service's perspective.
  • sidecar proxy crashes — if an Envoy sidecar crashes without the pod restarting (especially in non-Kubernetes deployments), the service is still running but all mTLS-gated traffic to it is blocked. No Consul alert fires.
  • Certificate rotation failure — Consul's built-in CA (or an external Vault-backed CA) issues leaf certificates to sidecars with short TTLs (72 hours by default). If the rotation job fails or the CA is unreachable, expired certs block all mTLS connections mesh-wide.
  • Intention misconfigurations — a policy push that creates an unintended deny rule between critical services silently breaks communication. Consul applies intentions immediately; there's no rollout and no automatic revert.
  • DNS resolution degradation — services discover each other through Consul DNS (service.consul addresses). If the Consul DNS server becomes overloaded or partitioned, service discovery fails even when the services themselves are healthy.

Vigilmon monitors cover the Consul control plane, sidecar health, certificate validity, and end-to-end service connectivity.

What you'll need

  • A running Consul cluster (3+ servers recommended for production)
  • Services registered in Consul with Connect enabled
  • A free Vigilmon account

Step 1: Expose health endpoints on Consul-registered services

Every service in the mesh should have a health endpoint registered with Consul. Add a /health route to each service:

Go (net/http):

package main

import (
    "encoding/json"
    "net/http"
)

func main() {
    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]interface{}{
            "status":  "ok",
            "service": "api",
        })
    })
    http.ListenAndServe(":8080", nil)
}

Register the service and its Connect sidecar in Consul with the health check:

# consul-service.hcl
service {
  name = "api"
  port = 8080

  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "database"
            local_bind_port  = 5432
          }
        ]
      }
    }
  }

  check {
    http     = "http://localhost:8080/health"
    interval = "10s"
    timeout  = "3s"
  }
}

Step 2: Monitor Consul server cluster health

The Consul HTTP API provides a /v1/status/leader endpoint that returns the current Raft leader. No leader = quorum failure. Monitor it externally:

In Vigilmon: Monitors → New Monitor → HTTP

  • URL: http://consul.yourdomain.com:8500/v1/status/leader
  • Expected status: 200
  • Expected body contains: " (a non-empty response = a leader exists; empty string = no leader)
  • Name: Consul Leader Health
  • Interval: 1 minute

For the peer list (all servers must be reachable):

#!/bin/bash
# consul-cluster-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_CLUSTER_HEARTBEAT"
CONSUL_ADDR="http://localhost:8500"
EXPECTED_PEERS=3  # adjust to your cluster size

# Check leader exists
leader=$(curl -s --max-time 5 "$CONSUL_ADDR/v1/status/leader")
if [ -z "$leader" ] || [ "$leader" = '""' ]; then
  echo "$(date): No Consul leader" >> /var/log/consul-probe.log
  exit 1
fi

# Check all peers are present
peer_count=$(curl -s --max-time 5 "$CONSUL_ADDR/v1/status/peers" | \
  python3 -c "import json,sys; print(len(json.load(sys.stdin)))" 2>/dev/null)

if [ "${peer_count:-0}" -lt "$EXPECTED_PEERS" ]; then
  echo "$(date): Only $peer_count/$EXPECTED_PEERS peers in Raft" >> /var/log/consul-probe.log
  exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

Create a Heartbeat monitor with 2-minute interval, name Consul Cluster Health.

Step 3: Monitor service health via the Consul catalog

Use the Consul health API to check that critical services have passing checks. This catches sidecar crashes and failed application health checks:

#!/bin/bash
# consul-service-health-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_SERVICE_HEALTH_HEARTBEAT"
CONSUL_ADDR="http://localhost:8500"

# Critical services to verify
SERVICES=("api" "database" "worker" "gateway")

all_ok=true

for svc in "${SERVICES[@]}"; do
  # Query service health — returns instances with their check status
  critical=$(curl -s --max-time 5 \
    "$CONSUL_ADDR/v1/health/service/$svc?passing=false" | \
    python3 -c "
import json, sys
data = json.load(sys.stdin)
critical = [
    i['Service']['ID']
    for i in data
    for check in i['Checks']
    if check['Status'] == 'critical'
]
print(len(critical))
" 2>/dev/null)

  if [ "${critical:-0}" -gt 0 ]; then
    echo "$(date): Service $svc has $critical critical instances" >> /var/log/consul-service-health.log
    all_ok=false
  fi
done

$all_ok && curl -s "$HEARTBEAT_URL" --max-time 10

Schedule every 2 minutes. Create a Heartbeat monitor with 5-minute interval.

Step 4: Monitor Connect mTLS certificate expiry

Consul's Connect CA issues leaf certificates with short TTLs. Monitor the CA itself and certificate rotation:

#!/bin/bash
# consul-cert-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_CERT_HEARTBEAT"
CONSUL_ADDR="http://localhost:8500"
WARN_DAYS=7  # Alert if root CA expires within 7 days

# Check the Connect CA status
ca_status=$(curl -s --max-time 5 "$CONSUL_ADDR/v1/connect/ca/roots")

if [ -z "$ca_status" ]; then
  echo "$(date): Could not retrieve CA roots" >> /var/log/consul-cert.log
  exit 1
fi

# Extract expiry of the active root certificate
expiry_date=$(echo "$ca_status" | python3 -c "
import json, sys
from datetime import datetime
data = json.load(sys.stdin)
roots = data.get('Roots', [])
active = [r for r in roots if r.get('Active')]
if not active:
    print('')
else:
    # NotAfter is in RFC3339 format
    print(active[0].get('NotAfter', ''))
" 2>/dev/null)

if [ -z "$expiry_date" ]; then
  echo "$(date): No active CA root found" >> /var/log/consul-cert.log
  exit 1
fi

# Parse and compare dates
days_left=$(python3 -c "
from datetime import datetime, timezone
expiry = datetime.fromisoformat('$expiry_date'.replace('Z', '+00:00'))
now = datetime.now(timezone.utc)
print((expiry - now).days)
" 2>/dev/null)

if [ "${days_left:-0}" -lt "$WARN_DAYS" ]; then
  echo "$(date): CA root expires in ${days_left} days" >> /var/log/consul-cert.log
  exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10

Run daily. Create a Heartbeat monitor with 25-hour interval, name Consul Connect CA Health.

Step 5: Monitor end-to-end Connect proxy connectivity

Verify that services can actually talk to each other through the Connect sidecar proxies — not just that Consul thinks they're healthy:

#!/bin/bash
# consul-connect-e2e-probe.sh
# Run from a service that has Connect upstreams configured

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_E2E_HEARTBEAT"

# Test connectivity through Connect sidecar upstreams
# These ports are the local_bind_port values in upstream config
UPSTREAM_CHECKS=(
  "http://127.0.0.1:5432"    # database upstream through Connect
  "http://127.0.0.1:6379"    # cache upstream through Connect
  "http://127.0.0.1:9090/health"  # internal API upstream
)

all_ok=true

for upstream in "${UPSTREAM_CHECKS[@]}"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$upstream")
  if [ "$code" = "000" ]; then
    echo "$(date): Cannot reach upstream $upstream (connection refused — sidecar down?)" >> /var/log/consul-e2e.log
    all_ok=false
  fi
done

$all_ok && curl -s "$HEARTBEAT_URL" --max-time 10

A connection refused on a local upstream port means the Envoy sidecar for that service has crashed. Create a Heartbeat monitor with 2-minute interval.

Step 6: Monitor Consul DNS resolution

Service discovery depends on Consul DNS. Test that .consul addresses resolve correctly:

#!/bin/bash
# consul-dns-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_DNS_HEARTBEAT"

# Critical services to resolve via Consul DNS
SERVICES=(
  "api.service.consul"
  "database.service.consul"
  "worker.service.consul"
)

all_ok=true

for svc in "${SERVICES[@]}"; do
  if ! dig @127.0.0.1 -p 8600 "$svc" +short +time=3 | grep -q '^[0-9]'; then
    echo "$(date): DNS resolution failed for $svc" >> /var/log/consul-dns.log
    all_ok=false
  fi
done

$all_ok && curl -s "$HEARTBEAT_URL" --max-time 10

Schedule every 2 minutes. Name the heartbeat monitor Consul DNS Health.

Step 7: Configure alerting and escalation

Structure Consul Connect alerts by layer:

  1. Consul Leader (HTTP monitor) — immediate; no leader blocks all control-plane operations
  2. Cluster Health (Heartbeat) — immediate; peer loss risks quorum failure
  3. Service Health (Heartbeat) — 2-minute delay; transient check failures are normal during deploys
  4. CA Health (Heartbeat) — 1-hour delay; cert expiry warning has lead time
  5. E2E Connect Proxy — immediate; sidecar down = mTLS broken for that service
  6. DNS Health — 5-minute delay; brief DNS hiccups are tolerable

In Vigilmon: Alert Channels → Add Channel → PagerDuty for Consul Leader and E2E Proxy monitors. Slack (#consul-ops) for service health and DNS.

Step 8: Build a service mesh status page

Give your platform team a consolidated view of mesh health:

  1. Status Pages → New Status Page
  2. Name: Consul Connect Service Mesh
  3. Add: Consul Leader, Cluster Health, Service Health, CA Health, E2E Connect Proxy, DNS Health
  4. Organize into component groups: Control Plane, Services, mTLS
  5. Share with your on-call rotation

Complete monitoring reference

| Monitor | Type | What it catches | |---------|------|-----------------| | Consul Leader | HTTP | No Raft leader, quorum failure | | Cluster Health | Heartbeat | Peer count below threshold | | Service Health | Heartbeat | Critical Consul health checks | | CA Health | Heartbeat | Root CA expiry, rotation failure | | E2E Connect Proxy | Heartbeat | Envoy sidecar crashes | | DNS Health | Heartbeat | Consul DNS resolution failure |

Troubleshooting common Consul Connect failures

  • No Raft leader — run consul operator raft list-peers on each server; identify which servers can't reach the others. Check network partitioning and firewall rules between server nodes. Common fix: restart the minority partition's Consul servers.
  • Service health check critical — run consul health service <name> to see which instances and which checks are failing; then inspect the specific service's logs and the Envoy sidecar logs (journalctl -u consul-envoy-<service> or docker logs).
  • mTLS connection refused on upstream port — the Envoy sidecar for that upstream crashed; restart it: systemctl restart consul-envoy-<service>. If it keeps crashing, check the xDS configuration: consul connect proxy -service <name> -log-level debug.
  • DNS resolution failing — verify Consul DNS is listening: dig @127.0.0.1 -p 8600 consul.service.consul; if that fails, the Consul agent on that host is down or the DNS port is blocked. Check consul members to verify the local agent is in the cluster.
  • CA rotation failure — if leaf cert TTLs expire before rotation, all mTLS traffic fails. Force rotation via the API: curl -X PUT http://localhost:8500/v1/connect/ca/configuration -d '{"Provider":"consul","Config":{}}'. This triggers immediate re-issuance of all leaf certs.

Next steps

  • Prometheus integration — Consul exposes metrics at /v1/agent/metrics?format=prometheus; pair with Grafana for trend analysis alongside your Vigilmon uptime checks
  • Intention audit — script a periodic export of consul intention list and diff against a known-good baseline to detect unauthorized policy changes

Start monitoring your Consul Connect service mesh today at vigilmon.online — free, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles