Monitoring cloudflared with Vigilmon

cloudflared is the daemon that powers Cloudflare Tunnel — a reverse proxy that routes traffic from Cloudflare's edge to your origin server without requiring any open inbound ports or a public IP. It's elegant infrastructure: no firewall rules, no port forwarding, no DDoS exposure. But when the cloudflared process crashes, your publicly accessible services vanish without warning. Vigilmon gives you the outside-in visibility to catch tunnel failures before your users do.

What You'll Set Up

• HTTP monitor for tunnel connectivity (the public-facing URL)
• Origin server reachability check through the tunnel
• Metrics endpoint health check for cloudflared daemon status
• Connector status monitoring
• Alert channels for immediate notification

Prerequisites

• cloudflared v2023.x+ running as a systemd service or Docker container
• At least one active Cloudflare Tunnel routing traffic to an origin service
• Metrics server enabled (--metrics 0.0.0.0:2000 or equivalent in config)
• A free Vigilmon account

Step 1: Monitor Tunnel Connectivity via the Public URL

The most important health signal for a Cloudflare Tunnel is whether the public-facing hostname resolves and returns a valid response through the tunnel. This check validates the entire path: DNS → Cloudflare edge → tunnel connector → origin.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the public URL routed through your tunnel (e.g. https://app.example.com).
4. Set Check interval to 2 minutes.
5. Set Expected HTTP status to 200 (or 301/302 if your origin redirects).
6. Click Save.

When cloudflared loses its connection to Cloudflare's edge, requests to this URL return a Cloudflare error page (530 or 502) rather than your origin content. Vigilmon catches this within the check interval, regardless of where in the stack the failure occurred.

Step 2: Verify Origin Reachability Through the Tunnel

A tunnel check at the root path confirms the route is live, but you should also verify that specific backend endpoints are accessible through the tunnel. A misconfigured ingress rule in cloudflared's config can send traffic to the wrong origin or return a 404 from cloudflared's own router.

For a web application:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://app.example.com/health (your app's health endpoint).
3. Set Expected body contains to a string from your health response (e.g. ok or healthy).
4. Set Check interval to 2 minutes.
5. Click Save.

For API services:

• Monitor https://api.example.com/v1/status or equivalent.
• Assert on the expected JSON response field.

This two-layer approach separates cloudflared routing failures (the tunnel is up but the ingress rule is wrong) from origin failures (the tunnel is fine but the backend is down) — two different failure modes with different remediation paths.

Step 3: Monitor the cloudflared Metrics Endpoint

cloudflared exposes a Prometheus-compatible metrics endpoint that reports internal connector state. This is your window into the daemon's health independent of external traffic.

Enable the metrics server in your cloudflared config:

# config.yml
metrics: 0.0.0.0:2000

Or pass it as a flag:

cloudflared tunnel --metrics 0.0.0.0:2000 run <tunnel-name>

The metrics endpoint is at:

http://localhost:2000/metrics

Key metrics to watch in the output:

• cloudflared_tunnel_active_streams — active HTTP/2 streams to Cloudflare edge
• cloudflared_tunnel_request_errors_total — cumulative request errors
• cloudflared_tunnel_ha_connections — number of HA (high-availability) connections to edge

To monitor this with Vigilmon from outside the host, expose the metrics port behind a local reverse proxy or check it via a Vigilmon TCP monitor:

1. Add Monitor → TCP Port.
2. Host: <your-cloudflared-host>, Port: 2000.
3. Set Check interval to 2 minutes.
4. Click Save.

Alternatively, if you expose the metrics endpoint behind nginx with HTTPS:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://<host>/metrics.
3. Set Expected body contains to cloudflared_tunnel_active_streams.
4. Click Save.

A response that omits cloudflared_tunnel_active_streams or returns no metrics indicates the cloudflared process has crashed or the metrics server failed to bind.

Step 4: Check Connector Status via the Cloudflare API

Cloudflare's dashboard API exposes tunnel connector status — whether each cloudflared instance is healthy, degraded, or inactive. You can poll this from Vigilmon using a keyword check or integrate it into your alerting pipeline.

GET https://api.cloudflare.com/client/v4/accounts/{account_id}/cfd_tunnel/{tunnel_id}/connectors
Authorization: Bearer <CF_API_TOKEN>

A healthy connector returns:

{
  "result": [
    {
      "id": "...",
      "arch": "linux_amd64",
      "config_version": 1,
      "conns": [{"colo": "JNB", "id": "...", "is_pending_reconnect": false}],
      "is_online": true,
      "version": "2024.2.1"
    }
  ]
}

For a simple health signal, watch is_online: true and is_pending_reconnect: false on your connector entries. A connector that is pending_reconnect has lost its edge connections and is actively trying to re-establish them — a signal that warrants investigation even if the public URL is still partially available.

To monitor this endpoint with Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://api.cloudflare.com/client/v4/accounts/{account_id}/cfd_tunnel/{tunnel_id}/connectors.
3. Add custom header: Authorization: Bearer <your-cf-api-token>.
4. Set Expected body contains to "is_online":true.
5. Set Check interval to 5 minutes.
6. Click Save.

Step 5: Monitor SSL Certificate Expiry

Cloudflare manages the edge certificate between clients and Cloudflare's network. But the certificate between cloudflared and your origin (if you use originServerName or TLS to origin) needs separate attention.

For the public-facing certificate (Cloudflare-managed):

1. Add Monitor → SSL Certificate.
2. URL: https://app.example.com.
3. Set Alert when certificate expires in less than 30 days.
4. Click Save.

While Cloudflare auto-renews its edge certificates, this check catches cases where a DNS misconfiguration has caused traffic to bypass Cloudflare and hit your origin directly with an expired cert.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and configure your preferred channel: email, Slack, PagerDuty, or webhook.
2. For tunnel connectivity monitors, set Consecutive failures before alert to 1 — cloudflared reconnects within seconds after most transient blips, so a second consecutive failure is genuinely alarming.
3. For the metrics endpoint TCP check, use 2 consecutive failures to absorb brief daemon restarts.
4. Set up a Status page if you host public-facing services through the tunnel — your users deserve transparency when external services are degraded.

Step 7: Multi-Tunnel and Redundant Connector Setup

Cloudflare recommends running at least two cloudflared instances per tunnel for high availability. If you do:

• Monitor each connector host independently via its metrics endpoint.
• Add a separate TCP port monitor per cloudflared instance.
• Keep the public-URL check as the authoritative signal — it represents end-user impact regardless of which connector is serving traffic.

When one connector fails and Cloudflare routes traffic to the redundant one, the public-URL check stays green. But the connector-level metrics check alerts you to the partial degradation — useful for catching single-node failures before they cascade.

Going Further

• Tunnel restart alerting: cloudflared logs reconnect events. If you run cloudflared via systemd, pair Vigilmon alerts with journalctl -u cloudflared -f log shipping to catch rapid reconnect loops that don't fail the TCP check but indicate instability.
• Ingress rule coverage: For tunnels routing multiple hostnames, add a Vigilmon check for each public hostname. A single-root check won't catch a broken ingress rule for api.example.com if app.example.com is still healthy.
• Latency tracking: Vigilmon's response time history can surface latency regressions in your tunnel — an early signal of cloudflared connector degradation before the tunnel fully drops.

With Vigilmon watching your cloudflared tunnel's public URL, metrics endpoint, and connector status, you'll know within minutes when tunnel infrastructure fails — before users start reporting that your site is unreachable.