tutorial

Monitoring HAProxy with Vigilmon

HAProxy is the load balancer in front of your most critical services — so you need to know the moment a backend server goes DOWN or the proxy itself stops responding. Here's how to monitor HAProxy stats, backend health, and SSL certificates with Vigilmon.

HAProxy is the battle-tested load balancer and proxy trusted by major websites, CDNs, and Fortune 500 infrastructure teams. When HAProxy is healthy, it silently distributes traffic across your backend servers and nobody thinks about it. When a backend server fails its health checks, HAProxy marks it DOWN and routes around it — but you still need to know. And when HAProxy itself goes down, every service behind it becomes unreachable simultaneously. Vigilmon monitors the HAProxy stats page, backend server health, session queue depth, and SSL certificates on your load-balanced frontends so you stay ahead of capacity problems and catch backend failures before they escalate.

What You'll Set Up

  • HTTP monitor for HAProxy stats page availability
  • Keyword monitor for backend servers in DOWN state
  • TCP probe for HAProxy process health
  • Heartbeat for backend server count and queue depth
  • SSL certificate alerts for HTTPS frontends

Prerequisites

  • HAProxy installed with the stats endpoint enabled (port 8404 or 1936 typical)
  • A free Vigilmon account

Step 1: Enable the HAProxy Stats Page

If you haven't already enabled the stats endpoint, add it to haproxy.cfg:

frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:your-password-here

Reload HAProxy to apply the change:

systemctl reload haproxy
# or: haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf $(cat /run/haproxy.pid)

Verify the stats page is accessible:

curl -u admin:your-password-here http://localhost:8404/stats | grep -o "HAProxy"
# HAProxy

Step 2: Monitor HAProxy Stats Page Availability

The stats page being up confirms that HAProxy is running and responding. A 200 response with the right content means the process is alive and serving management traffic.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the stats URL: http://your-haproxy-host:8404/stats
  4. If you set a stats password, add HTTP basic auth credentials in the monitor settings.
  5. Set Check interval to 1 minute.
  6. Under Keyword monitoring, enable it and set the keyword to HAProxy.
  7. Click Save.

The stats page HTML includes "HAProxy" in the page title. A keyword match confirms you're getting the real stats page, not an error response or a redirect to a login page you didn't configure.

Step 3: Monitor Backend Server DOWN State

This is the most operationally critical monitor. When HAProxy marks a backend server DOWN (after failing its configured health checks), traffic to that server stops and the remaining servers absorb the load. If too many backends go DOWN, HAProxy starts returning 503 errors to clients.

The CSV stats endpoint provides machine-parseable backend status:

  1. Click Add Monitor → set Type to HTTP / HTTPS.
  2. Enter: http://your-haproxy-host:8404/stats;csv
  3. Add HTTP basic auth if configured.
  4. Set Check interval to 1 minute.
  5. Enable Keyword monitoring and set the keyword to ,DOWN,.
  6. Set Alert on keyword found (invert the match — alert when the keyword IS present).
  7. Click Save.

The CSV format has a status column. When a backend server is unhealthy, that column contains DOWN. Monitoring for ,DOWN, (with commas to avoid false matches on column headers) fires an alert the moment any backend fails its health check.

# See current backend status in CSV
curl -s -u admin:your-password-here "http://localhost:8404/stats;csv" \
  | awk -F',' '{print $2, $18}' | column -t
# BACKEND        status
# web-backend-1  UP
# web-backend-2  DOWN
# web-backend-3  UP

Step 4: TCP Probe for HAProxy Process Health

A TCP probe directly to the HAProxy frontend port (typically 80 or 443 for HTTP traffic) confirms the process is running and accepting connections, independent of the stats endpoint configuration.

  1. Click Add Monitor → set Type to TCP.
  2. Enter: your-haproxy-host:80 (or your primary frontend port)
  3. Set Check interval to 1 minute.
  4. Click Save.

If HAProxy crashes, TCP connections to all frontend ports are refused simultaneously. This probe catches process crashes faster than the HTTP stats check because it requires no HTTP processing — just a TCP accept.

# Verify HAProxy is listening
ss -tlnp | grep haproxy
# LISTEN  0  128  0.0.0.0:80  0.0.0.0:*  users:(("haproxy",pid=1234,fd=6))
# LISTEN  0  128  0.0.0.0:443  0.0.0.0:*  users:(("haproxy",pid=1234,fd=7))
# LISTEN  0  128  0.0.0.0:8404  0.0.0.0:*  users:(("haproxy",pid=1234,fd=8))

Step 5: Heartbeat for Backend Count and Queue Depth

Beyond individual server DOWN states, you should track two additional capacity signals: the total number of servers available and the session queue depth. These catch subtle degradation — a backend pool slowly draining of healthy servers and requests starting to queue because the remaining servers can't keep up.

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 5 minutes.
  3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Create /usr/local/bin/haproxy-health-check.sh:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
STATS_URL="http://localhost:8404/stats;csv"
STATS_AUTH="admin:your-password-here"
MAX_DOWN_SERVERS=1
MAX_QUEUE_DEPTH=10

CSV=$(curl -s -u "$STATS_AUTH" "$STATS_URL")
if [ $? -ne 0 ]; then
  echo "Failed to fetch HAProxy stats"
  exit 1
fi

# Count DOWN backend servers (column 18 = status, column 1 = pxname, column 2 = svname)
DOWN_COUNT=$(echo "$CSV" | awk -F',' '$18 == "DOWN" && $2 != "BACKEND" {count++} END {print count+0}')

# Check current queue depth across all backends (column 3 = qcur)
MAX_QCUR=$(echo "$CSV" | awk -F',' '$2 == "BACKEND" {if ($3+0 > max) max = $3+0} END {print max+0}')

if [ "$DOWN_COUNT" -le "$MAX_DOWN_SERVERS" ] && [ "$MAX_QCUR" -le "$MAX_QUEUE_DEPTH" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
else
  echo "HAProxy degraded: DOWN=$DOWN_COUNT, max queue=$MAX_QCUR"
fi

chmod +x /usr/local/bin/haproxy-health-check.sh

Add to cron:

*/5 * * * * root /usr/local/bin/haproxy-health-check.sh

Tune MAX_DOWN_SERVERS to match your tolerance — a pool of 5 servers can absorb 1 being DOWN, but not 3. Tune MAX_QUEUE_DEPTH based on your SLAs — a queue depth above 0 for more than one check interval means your backends are overloaded.

Step 6: SSL Certificate Alerts for HTTPS Frontends

HAProxy frequently terminates SSL for multiple backend services at once. A single expired certificate on a HAProxy frontend takes down all the backends behind that listener from every client's perspective — even though the backends themselves are perfectly healthy.

For each HTTPS frontend, add a certificate monitor:

  1. Open the HTTP stats monitor (Step 2) — or add a new HTTP/HTTPS monitor targeting each frontend domain.
  2. Enable Monitor SSL certificate under the SSL section.
  3. Set Alert when certificate expires in less than 21 days.
  4. Click Save.

If you have multiple HTTPS frontends (e.g. app.example.com, api.example.com, admin.example.com), add a separate SSL monitor for each domain:

# Check all certificates HAProxy is serving
echo | openssl s_client -connect app.example.com:443 2>/dev/null | openssl x509 -noout -dates
echo | openssl s_client -connect api.example.com:443 2>/dev/null | openssl x509 -noout -dates

HAProxy can serve certificates from files (crt directive in the bind configuration) or from a certificate store. If you use Let's Encrypt with certbot and a deploy hook to reload HAProxy, verify the reload hook is working:

# /etc/letsencrypt/renewal-hooks/deploy/haproxy-reload.sh
#!/bin/bash
cat /etc/letsencrypt/live/your-domain/fullchain.pem \
    /etc/letsencrypt/live/your-domain/privkey.pem \
    > /etc/haproxy/certs/your-domain.pem
systemctl reload haproxy

Step 7: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, PagerDuty, or a webhook.
  2. For the stats page HTTP monitor, set Consecutive failures before alert to 2 — HAProxy hot-reloads create a brief gap.
  3. For the TCP probe on frontend ports, set consecutive failures to 1.
  4. For the backend DOWN keyword monitor, set consecutive failures to 1 — a backend going DOWN is never transient.
  5. For heartbeat monitors, alerts fire automatically when the ping interval passes.

Use Maintenance windows during planned HAProxy reloads or certificate rotations:

# Before a planned config reload
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 2}'

systemctl reload haproxy

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP keyword | host:8404/stats → "HAProxy" | Process crash, stats endpoint down | | HTTP keyword (inverted) | /stats;csv → ,DOWN, | Backend server health check failure | | TCP probe | Frontend port (80/443) | HAProxy process crash | | Heartbeat script | CSV stats: DOWN count + qcur | Backend pool drain, queue overload | | SSL certificate | Each HTTPS frontend domain | Certificate expiry on load-balanced frontends |

HAProxy's reliability is legendary — but that reputation depends on monitoring what happens behind it. With Vigilmon watching the stats endpoint, backend health states, session queue depth, and frontend certificates, you get the operational visibility to keep your load-balanced services running smoothly.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles