Monitoring Inlets Tunnels with Vigilmon

Inlets is a self-hosted cloud-native tunnel service created by Alex Ellis (creator of OpenFaaS) that lets you expose services running on private networks to the internet via a cloud exit server you control. The architecture is simple: an Inlets server runs on a public VPS (exit server), and an Inlets client runs in your private network, connecting outbound via WebSocket on a control port (typically 8090). Inbound traffic arrives at your exit server domain, flows through the tunnel, and reaches the private service. Unlike SaaS tunnels (ngrok, Cloudflare Tunnel), you own the entire stack — which means you own the monitoring too. Vigilmon covers exit server health, tunnel connectivity, TLS certificate validity, and end-to-end data plane probes.

What You'll Set Up

• Exit server process and TCP port health
• Tunnel control connection (WebSocket on port 8090) liveness
• End-to-end data plane HTTP probe through the tunnel
• Tunnel reconnect rate monitoring via cron heartbeat
• Exit server public IP reachability
• HTTPS certificate expiry alerts (14-day window)
• Per-tunnel health for multi-tunnel deployments
• Client-side process health in the private network

Prerequisites

• Inlets OSS or Inlets Pro exit server running on a public VPS
• Inlets client running in your private network and connected
• Public domain pointed at the exit server IP
• A free Vigilmon account

Step 1: Monitor the Exit Server Port

The Inlets server listens on a data plane port (default 8080 for HTTP traffic) that proxies inbound requests to the private service. A TCP failure here means all inbound traffic is failing:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to TCP.
3. Enter your-exit-server-ip:8080.
4. Set Check interval to 30 seconds.
5. Set Alert threshold to 2 consecutive failures.
6. Click Save.

Also add a TCP monitor for the control port (8090), which is where the Inlets client establishes its WebSocket connection:

1. Click Add Monitor → TCP.
2. Enter your-exit-server-ip:8090.
3. Set Check interval to 30 seconds.
4. Set Alert threshold to 2 consecutive failures.
5. Click Save.

If the control port goes down, the Inlets client will lose its WebSocket connection and the tunnel collapses.

Step 2: End-to-End Data Plane Probe

A TCP check on the exit server port confirms the Inlets server is running, but not that the tunnel is actually forwarding traffic to your private service. Add an HTTP monitor that sends a real request through the tunnel:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter https://your-tunnel-domain.com/health — this is the URL exposed by your exit server that routes through the tunnel to your private service's /health endpoint.
3. Set Expected HTTP status to 200.
4. Set Check interval to 1 minute.
5. Enable Keyword check and enter a string your /health endpoint reliably returns (e.g., "status":"ok").
6. Click Save.

This end-to-end probe validates the entire path: DNS → exit server → WebSocket tunnel → private network service. If the probe fails but the TCP checks pass, the tunnel data path is broken even though the Inlets process is running.

For services without a /health endpoint, probe any URL that returns a known status code:

# Test the tunnel manually first
curl -v https://your-tunnel-domain.com/ 2>&1 | head -20

Step 3: HTTPS Certificate Expiry Alert

If you use Inlets Pro with Caddy or cert-manager for automatic TLS, certificates must renew before expiry. Auto-renewal can fail silently if the DNS challenge fails or the exit server loses outbound internet access:

1. Open the HTTP monitor you created in Step 2 for your tunnel domain.
2. Enable Monitor SSL certificate under the SSL section.
3. Set Alert when certificate expires in less than 14 days.
4. Click Save.

A 14-day window gives you two renewal cycles to investigate and fix the issue before expiry. For Inlets Pro with cert-manager:

# Check certificate status on the exit server
kubectl get certificate -n inlets
# or check Caddy's certificate store
caddy list-certificates

If you run multiple tunnel domains, add an SSL monitor for each one separately.

Step 4: Tunnel Reconnect Rate Monitoring

Inlets clients auto-reconnect when the WebSocket control connection drops. A high reconnect rate indicates network instability between the client and exit server. Monitor this via the Inlets client logs on the private network host:

#!/bin/bash
# /usr/local/bin/inlets-reconnect-check.sh
# Run on the private network host where the Inlets client runs
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_RECONNECT_HEARTBEAT_ID"
MAX_RECONNECTS_PER_HOUR=5  # alert if more than 5 reconnects in the past hour
LOG_FILE="/var/log/inlets/client.log"

if [ ! -f "$LOG_FILE" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
  exit 0
fi

# Count reconnect events in the last hour
ONE_HOUR_AGO=$(date -d "1 hour ago" "+%Y-%m-%d %H:%M:%S")
RECONNECT_COUNT=$(awk -v since="$ONE_HOUR_AGO" '$0 >= since && /reconnect|Reconnecting/' "$LOG_FILE" | wc -l)

if [ "$RECONNECT_COUNT" -le "$MAX_RECONNECTS_PER_HOUR" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

If you run Inlets as a systemd service, use journalctl instead:

RECONNECT_COUNT=$(journalctl -u inlets-client --since "1 hour ago" 2>/dev/null | grep -c "reconnect\|Reconnecting")

Schedule this every 15 minutes with a 20-minute heartbeat interval in Vigilmon.

Step 5: Exit Server Public IP Reachability

If the exit server's public IP becomes unreachable (VPS suspended, IP block, firewall change), all tunnels fail. Add a Vigilmon TCP monitor for the exit server IP directly to catch this before DNS is involved:

1. Click Add Monitor → TCP.
2. Enter YOUR_EXIT_SERVER_PUBLIC_IP:22 (SSH port — almost always open if the VPS is alive).
3. Set Check interval to 1 minute.
4. Set Alert threshold to 2 consecutive failures.
5. Click Save.

Using port 22 (SSH) avoids false positives from Inlets-specific configuration issues and confirms the VPS itself is reachable at the network level.

Step 6: Exit Server Resource Usage

A single exit server VPS may run many tunnels. High CPU or memory on the exit server degrades all tunnel throughput. Monitor resource usage via a cron heartbeat on the exit server:

#!/bin/bash
# /usr/local/bin/inlets-exit-resources.sh
# Run on the exit server VPS
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_RESOURCES_HEARTBEAT_ID"
CPU_THRESHOLD=70   # alert at 70% CPU
MEM_THRESHOLD=80   # alert at 80% memory

# 5-second CPU average
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d. -f1)

# Memory usage percentage
MEM_TOTAL=$(grep MemTotal /proc/meminfo | awk '{print $2}')
MEM_AVAILABLE=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
MEM_USED=$(( MEM_TOTAL - MEM_AVAILABLE ))
MEM_PERCENT=$(( MEM_USED * 100 / MEM_TOTAL ))

if [ "$CPU_USAGE" -lt "$CPU_THRESHOLD" ] && [ "$MEM_PERCENT" -lt "$MEM_THRESHOLD" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

Schedule every 5 minutes on the exit server. Set the heartbeat interval to 10 minutes to allow for a single missed check:

*/5 * * * * /usr/local/bin/inlets-exit-resources.sh

Step 7: Client-Side Process Health

The Inlets client process in the private network must stay running for the tunnel to exist. Monitor it with a heartbeat that confirms the client process is alive:

#!/bin/bash
# /usr/local/bin/inlets-client-health.sh
# Run on the private network host
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CLIENT_HEARTBEAT_ID"

if pgrep -x "inlets" > /dev/null; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi
# If the inlets client process is not running, heartbeat is skipped → Vigilmon alerts

If you run Inlets as a systemd service, check the service state directly:

if systemctl is-active --quiet inlets-client; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

Schedule every 2 minutes with a 5-minute heartbeat interval in Vigilmon.

Step 8: Multi-Tunnel Health (Per-Tunnel Liveness)

If you run multiple Inlets clients (one per private service), you need per-tunnel health checks rather than a single aggregate check. For each tunnel, create a separate end-to-end HTTP monitor in Vigilmon:

| Tunnel | Exit Domain | Vigilmon Monitor | |---|---|---| | Home Assistant | homeassistant.example.com/api/ | HTTP → 200 | | Gitea | gitea.example.com/ | HTTP → 200 | | Nextcloud | nextcloud.example.com/status.php | HTTP + keyword "installed":true | | SSH bastion | ssh.example.com:22 | TCP |

For each service:

1. Click Add Monitor → HTTP / HTTPS (or TCP for non-HTTP tunnels).
2. Enter the specific tunnel domain and path.
3. Set Check interval to 1 minute.
4. Enable SSL certificate monitoring if the tunnel serves HTTPS.
5. Click Save.

Group all tunnel monitors in a Vigilmon status page so you can see the health of every tunnel at a glance.

Step 9: Throughput Alert via Log Metrics

If your exit server runs Caddy as the TLS terminator (common with Inlets Pro), parse Caddy's structured access log to detect throughput anomalies:

#!/bin/bash
# /usr/local/bin/inlets-throughput-check.sh
# Run on the exit server
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_THROUGHPUT_HEARTBEAT_ID"
MIN_REQUESTS_PER_MIN=1  # alert if no requests through the tunnel in the past minute
CADDY_LOG="/var/log/caddy/access.log"

if [ ! -f "$CADDY_LOG" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
  exit 0
fi

ONE_MIN_AGO=$(date -d "1 minute ago" "+%Y-%m-%dT%H:%M")
REQUEST_COUNT=$(awk -v since="$ONE_MIN_AGO" '$0 >= since' "$CADDY_LOG" | wc -l)

if [ "$REQUEST_COUNT" -ge "$MIN_REQUESTS_PER_MIN" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

This is most useful for tunnels serving live user traffic where zero requests for 1+ minute is anomalous. Adjust MIN_REQUESTS_PER_MIN based on your expected traffic baseline.

Step 10: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and connect your Slack, PagerDuty, or email.
2. For TCP monitors (exit server ports, public IP): 2 consecutive failures — TCP blips can be transient.
3. For the end-to-end HTTP tunnel probe: 2 consecutive failures — brief WebSocket reconnects cause a single probe failure.
4. For the SSL certificate monitor: alert at 14 days expiry.
5. For client process heartbeat: 1 missed ping — if the client is dead, the tunnel is already down.
6. For resource usage heartbeat: 2 missed pings — a single high-CPU spike that resolves itself is less urgent.
7. For a multi-tunnel setup, create a Vigilmon status page with all tunnel monitors visible to your team at a glance.

Summary

| Monitor | Type | Interval | Alert Threshold | |---|---|---|---| | Exit server TCP data port | TCP | 30s | 2 failures | | Exit server TCP control port (8090) | TCP | 30s | 2 failures | | Exit server public IP (SSH port) | TCP | 1m | 2 failures | | End-to-end tunnel HTTP probe | HTTP | 1m | 2 failures | | SSL certificate expiry | HTTP+SSL | 1m | 14 days warning | | Tunnel reconnect rate | Cron heartbeat | 15m | 1 missed ping | | Exit server CPU/memory | Cron heartbeat | 5m | 2 missed pings | | Client process liveness | Cron heartbeat | 2m | 1 missed ping | | Per-tunnel HTTP liveness (×N) | HTTP | 1m | 2 failures |

Inlets gives you full control over your tunnel infrastructure — and full responsibility for knowing when it breaks. The TCP and HTTP monitors in Vigilmon catch the obvious failures (exit server down, tunnel connection lost), while the cron heartbeat pattern handles the subtler operational issues: high reconnect rates, resource exhaustion on the exit server, and client process crashes in the private network. Together, these checks give you the confidence to rely on self-hosted tunnels for production traffic.