Monitoring Endlessh with Vigilmon

Endlessh is an SSH tarpit that wastes attackers' time by serving a never-ending SSH banner on port 22, keeping automated bots tied up for hours while your real SSH service listens on a non-standard port. When Endlessh works, brute-force bots are throttled and you get valuable threat intelligence from the connection logs. When it stops working, those same bots hit your real SSH daemon directly. Vigilmon gives you the uptime monitoring, attack rate tracking, and alert routing to keep both your tarpit and your real SSH service visible.

What You'll Set Up

• TCP port check on Endlessh's trap port (22 or 2222)
• Prometheus metrics scrape health check (port 2112 for Endlessh-go)
• Real SSH service health check on your non-standard port
• Active connection count alerting
• Log pipeline heartbeat
• Alert channels for tarpit outage notifications

Prerequisites

• Endlessh or Endlessh-go installed and running
• Real SSH service moved to a non-standard port (e.g., 2222 or 22222)
• A free Vigilmon account

Why Monitor Endlessh?

Endlessh is a passive security control — when it's working, you don't notice it. When it fails:

• Brute-force bots reach your real SSH service — the entire point of the tarpit is to keep bots occupied; downtime removes that protection entirely
• Silent failure is the norm — Endlessh doesn't send you alerts when it crashes; you only notice when SSH attack logs reappear in auth.log
• Connection exhaustion becomes a risk — without an upper bound on concurrent tarpit connections, a sustained DDoS via connection exhaustion can starve your server's available file descriptors
• Log pipeline failures hide attack intelligence — if Endlessh logs stop flowing to your dashboards, you lose visibility into attack campaigns targeting your IP

Monitoring Endlessh isn't glamorous, but a 15-minute outage during an active scanning campaign means thousands of brute-force attempts reach your SSH daemon.

Key Metrics to Monitor

| Metric | Why It Matters | |---|---| | Trap port TCP binding | Downtime means bots reach real SSH | | Real SSH port liveness | Ensures legitimate access still works | | Active trapped connections | Alert on >1000 — risk of connection exhaustion | | Connections per minute | Spike indicates active scanning campaign | | Prometheus /metrics scrape | Access to endlessh_connections_total, endlessh_client_seconds_total | | Log ingestion health | Heartbeat confirms log pipeline is flowing |

Step 1: TCP Port Check on the Trap Port

Endlessh doesn't serve HTTP — it speaks a slow SSH banner on a raw TCP port. Use a TCP port monitor in Vigilmon to check the trap port binding:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to TCP Port.
3. Enter your server IP or hostname.
4. Set Port to 22 (or 2222 if running Endlessh on a non-standard trap port).
5. Set Check interval to 1 minute.
6. Click Save.

Vigilmon connects to the TCP port and confirms it accepts connections. If Endlessh is down, the port refuses connections and the monitor goes red. This is your primary liveness check.

Step 2: Monitor the Prometheus Metrics Endpoint

Endlessh-go (the Go reimplementation) exposes Prometheus metrics on port 2112 by default, including:

• endlessh_connections_total — cumulative connections trapped
• endlessh_client_seconds_total — total attacker time wasted
• endlessh_clients_connected — currently active trapped connections

Monitor the metrics endpoint availability:

1. Click Add Monitor in Vigilmon.
2. Set Type to HTTP / HTTPS.
3. Enter: http://YOUR_SERVER_IP:2112/metrics
4. Set Expected HTTP status to 200.
5. Set Check interval to 2 minutes.
6. Click Save.

If you're running the original C implementation of Endlessh (without Prometheus support), skip this step and rely on the TCP port check plus log-based alerting.

Step 3: Monitor Your Real SSH Service

After moving legitimate SSH to a non-standard port, monitor it explicitly — if it goes down while Endlessh is still running on port 22, you lose access to your own server.

1. Click Add Monitor in Vigilmon.
2. Set Type to TCP Port.
3. Enter your server IP.
4. Set Port to your real SSH port (e.g., 2222 or 22222).
5. Set Check interval to 1 minute.
6. Click Save.

This monitor is as important as the Endlessh monitor itself — losing SSH access to your server is a more severe incident than the tarpit going down.

Step 4: Process Heartbeat via Cron

For servers where Endlessh runs as a systemd service, add a heartbeat that pings Vigilmon while the process is alive:

# Add to crontab with: crontab -e
*/5 * * * * pgrep -x endlessh > /dev/null && curl -fsS https://vigilmon.online/beat/YOUR_HEARTBEAT_ID

For Endlessh-go:

*/5 * * * * pgrep -x endlessh-go > /dev/null && curl -fsS https://vigilmon.online/beat/YOUR_HEARTBEAT_ID

Create the heartbeat in Vigilmon under Heartbeats, set the expected interval to 10 minutes, and replace YOUR_HEARTBEAT_ID above. If the heartbeat stops — even though the TCP port still responds for a brief grace period — you know the process has stopped.

Step 5: Attack Rate Alerting via Log Pipeline Heartbeat

Endlessh logs each connection event. If you're piping logs to a SIEM, Loki, or a custom dashboard, add a Vigilmon heartbeat that your log ingestion pipeline pings after successful processing:

# Example: systemd log forwarder cron
*/2 * * * * /usr/local/bin/forward-endlessh-logs.sh && curl -fsS https://vigilmon.online/beat/YOUR_LOG_PIPELINE_HEARTBEAT_ID

Create a second heartbeat in Vigilmon with a 6 minute expected interval. If logs stop flowing (Endlessh crash, log rotation misconfiguration, or disk full), the heartbeat silences and Vigilmon alerts.

Step 6: Monitoring Active Connections with Endlessh-go Metrics

If running Endlessh-go with the Prometheus endpoint, set up a scrape job in Prometheus to alert when active connections exceed your server's safe threshold:

# prometheus.yml
scrape_configs:
  - job_name: 'endlessh'
    static_configs:
      - targets: ['localhost:2112']

Add an alert rule:

# alerts.yml
groups:
  - name: endlessh
    rules:
      - alert: EndlesshConnectionExhaustion
        expr: endlessh_clients_connected > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Endlessh has >1000 active trapped connections"
          description: "{{ $value }} connections may exhaust file descriptors"

      - alert: EndlesshHighAttackRate
        expr: rate(endlessh_connections_total[5m]) * 60 > 100
        for: 2m
        labels:
          severity: info
        annotations:
          summary: "Endlessh seeing >100 new connections/minute"
          description: "Active scanning campaign detected"

Use Vigilmon for uptime and Prometheus for metric-based alerting — they complement each other.

Step 7: Configure Alert Channels

Set up appropriate alert routing for the two distinct failure scenarios:

Tarpit down (Endlessh process crashed):

• Route to a low-urgency notification channel (Slack DM, email)
• This is important but not an emergency requiring immediate wake-up

Real SSH port down:

• Route to a high-urgency on-call channel (PagerDuty, SMS)
• Losing SSH access to the server is immediately critical

In Vigilmon:

1. Go to Settings → Alert Channels.
2. Add a Slack channel for tarpit monitoring.
3. Add a PagerDuty integration for real SSH monitoring.
4. Assign monitors: tarpit TCP check → Slack, real SSH TCP check → PagerDuty.

Step 8: Verify the Setup

Test both critical monitors:

Tarpit monitor test:

1. Temporarily stop Endlessh: sudo systemctl stop endlessh
2. Verify the TCP port 22 monitor goes red in Vigilmon within 2 minutes.
3. Restart: sudo systemctl start endlessh
4. Confirm the monitor recovers.

Real SSH monitor test (careful — do this with a backup console session active):

1. Confirm you have an active SSH session before testing.
2. The real SSH monitor should be green with your SSH service running normally.
3. Do not stop real SSH during testing without an out-of-band console (IPMI/KVM) available.

Troubleshooting Common Endlessh Issues

Port 22 already in use after restart If Endlessh crashes and you restart it, the kernel may hold the port in TIME_WAIT state. Check: ss -tlnp | grep :22. If another process holds port 22, identify it with fuser 22/tcp. If real SSH is somehow running on 22 again, check /etc/ssh/sshd_config for the Port directive.

Endlessh-go metrics not appearing Confirm the metrics server is enabled in your Endlessh-go config: --prometheusEnabled flag or prometheus_enabled: true in config. Check the port binding: ss -tlnp | grep 2112. If using Docker, ensure port 2112 is exposed or mapped.

High CPU from Endlessh under heavy attack Endlessh is intentionally lightweight — high CPU usually means an abnormally high connection rate. Check endlessh_connections_total rate in Prometheus. If you're seeing thousands of connections per minute, consider adding iptables rate limiting on port 22 to cap incoming connections without disabling the tarpit.

Real SSH service unreachable after changing port Verify the firewall allows the new SSH port: ufw allow 2222/tcp (or equivalent). Check the sshd config: grep Port /etc/ssh/sshd_config. Restart sshd: sudo systemctl restart sshd. Always verify the new port works from a second terminal before closing the original session.

Conclusion

Endlessh is a set-and-forget security tool that quietly wastes attacker resources in the background — which means it can also fail quietly. A simple TCP port check on the trap port and a separate check on your real SSH port gives you the monitoring needed to know immediately when the tarpit goes offline and when legitimate access is at risk.

Start with the two TCP port monitors (trap port and real SSH port) — those are the most critical signals. Add the Endlessh-go Prometheus metrics scrape monitor if you're using the Go implementation, and a process heartbeat for belt-and-suspenders coverage.

Sign up for Vigilmon and add the TCP port check for port 22 as your first Endlessh monitor.