Monitoring ZNC IRC Bouncer with Vigilmon

ZNC is the backbone of persistent IRC infrastructure — it keeps connections open to IRC networks, buffers messages while you're offline, and lets multiple clients attach to the same session. But ZNC failures are silent: if the daemon crashes or an upstream IRC connection drops, you miss messages and your users lose access without any immediate notification. Vigilmon monitors ZNC's admin web UI, process health, and TLS certificates so you catch failures before your users report them.

What You'll Set Up

• HTTP uptime monitor for the ZNC admin web interface
• TLS certificate expiry alerts for the ZNC port
• Process heartbeat via cron to detect silent daemon crashes
• Disk space monitoring for ZNC message buffers
• Alert routing for IRC team channels

Prerequisites

• ZNC installed and running, with the admin web UI accessible over HTTPS
• A free Vigilmon account

Step 1: Monitor the ZNC Admin Web Interface

ZNC's admin web UI runs on the same port as IRC client connections (typically 6697 with TLS). The web interface lets you manage users, networks, and modules — if it's unreachable, something is seriously wrong with the ZNC process.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the admin URL: https://your-server:6697/ (or your configured port).
4. Set Expected HTTP status to 200.
5. Set Check interval to 1 minute.
6. Click Save.

If you have the ZNC webadmin module enabled, the root path returns an HTML page with a 200 status when ZNC is running. A connection refused or timeout here means the daemon is down.

Note: ZNC uses a self-signed certificate by default. In Vigilmon, you can disable certificate validation for this monitor if you haven't yet configured a proper TLS certificate, but switching to a CA-signed cert is strongly recommended.

Step 2: Add a ZNC Status Page Check

ZNC's webadmin module serves a status page at /mods/global/webadmin/ or the root page shows the login form. You can also probe the :status IRC command output by hitting the web API if you have the modpython or modperl status modules enabled.

For a simpler check, monitor the login form response:

1. Add a Vigilmon monitor with Type HTTP / HTTPS.
2. Set the URL to: https://your-server:6697/
3. Under Content match, enter ZNC (the page title contains "ZNC").
4. Set Expected HTTP status to 200.
5. Click Save.

The content match catches cases where something else (e.g., nginx) answers on that port but ZNC itself is not running.

Step 3: Add a Process Heartbeat

The ZNC daemon can crash without taking down the host — the port becomes closed, but no external alert fires unless you're watching. Use a heartbeat monitor to verify the process is alive:

1. In Vigilmon, click Add Monitor → Cron Job / Heartbeat.
2. Set Expected interval to 5 minutes.
3. Copy the heartbeat URL.

On your server, add a cron job that pings only when ZNC is actually running:

# /etc/cron.d/znc-heartbeat
*/5 * * * * znc pgrep -x znc && curl -fsS --retry 3 https://vigilmon.online/ping/your-heartbeat-id > /dev/null 2>&1

If ZNC crashes, pgrep returns non-zero, the && prevents the curl, and Vigilmon marks the heartbeat as missed.

If you run ZNC under systemd, you can add the heartbeat to a systemd timer instead:

# /etc/systemd/system/znc-heartbeat.service
[Unit]
Description=ZNC Vigilmon heartbeat
After=znc.service
Requires=znc.service

[Service]
Type=oneshot
User=znc
ExecStart=/usr/bin/curl -fsS --retry 3 https://vigilmon.online/ping/your-heartbeat-id

# /etc/systemd/system/znc-heartbeat.timer
[Unit]
Description=ZNC heartbeat every 5 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

systemctl enable --now znc-heartbeat.timer

Step 4: TLS Certificate Monitoring

ZNC serves TLS on the IRC port for both client connections and the admin web UI. An expired certificate breaks all TLS client connections immediately — IRC clients connecting on port 6697 will refuse the handshake.

1. Open the HTTP / HTTPS monitor created in Step 1.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

ZNC generates its own self-signed certificate at ~/.znc/znc.pem. To replace it with a Let's Encrypt certificate:

# Concatenate cert and key into znc.pem format
cat /etc/letsencrypt/live/your-domain/privkey.pem \
    /etc/letsencrypt/live/your-domain/fullchain.pem \
    > ~/.znc/znc.pem
chmod 600 ~/.znc/znc.pem
systemctl restart znc

Add a renewal hook to update znc.pem automatically after Certbot renews:

# /etc/letsencrypt/renewal-hooks/deploy/znc.sh
#!/bin/bash
cat /etc/letsencrypt/live/your-domain/privkey.pem \
    /etc/letsencrypt/live/your-domain/fullchain.pem \
    > /home/znc/.znc/znc.pem
chmod 600 /home/znc/.znc/znc.pem
systemctl reload znc

Step 5: Buffer Disk Space Monitoring

ZNC stores channel and query replay buffers in ~/.znc/users/*/networks/*/moddata/. On a busy IRC network with large buffer limits, this can fill a disk and cause ZNC to stop writing buffers or crash.

Add a cron heartbeat monitor that only pings when disk usage is healthy:

# /etc/cron.d/znc-disk-heartbeat
*/15 * * * * znc \
  USAGE=$(df ~/.znc --output=pcent | tail -1 | tr -d ' %') && \
  [ "$USAGE" -lt 85 ] && \
  curl -fsS --retry 3 https://vigilmon.online/ping/your-disk-heartbeat-id > /dev/null 2>&1

Set the heartbeat interval to 20 minutes — if disk usage exceeds 85%, the ping stops and Vigilmon alerts you.

Alternatively, expose a tiny status HTTP server and monitor it:

# /usr/local/bin/znc-health.sh
#!/bin/bash
USAGE=$(df ~/.znc --output=pcent | tail -1 | tr -d ' %')
ZNC_PID=$(pgrep -x znc)

if [ -n "$ZNC_PID" ] && [ "$USAGE" -lt 85 ]; then
  echo '{"status":"ok","disk_pct":'"$USAGE"'}'
  exit 0
else
  echo '{"status":"degraded","disk_pct":'"$USAGE"'}'
  exit 1
fi

Serve it over a lightweight HTTP wrapper and monitor at http://your-server:9090/health.

Step 6: Configure Alerts

1. In Vigilmon, go to Settings → Alerts.
2. Add your preferred alert channel: Slack, email, PagerDuty, or webhook.
3. Set alert thresholds to 2 consecutive failures for the admin UI monitor to avoid spurious alerts.
4. Set heartbeat missed threshold to 2 missed intervals.

Recommended alert escalation:

| Monitor | Alert trigger | Priority | |---|---|---| | Admin web UI | 1 failure | Critical | | Process heartbeat | 2 missed | Critical | | TLS certificate | < 21 days remaining | Warning | | Buffer disk space | 1 missed heartbeat | Warning |

What to Watch

| Monitor | Check | Interval | |---|---|---| | Admin web interface | HTTP 200 + content match | 1 minute | | Process heartbeat | Cron ping | 5 minutes | | TLS certificate | Expiry < 21 days | Passive | | Buffer disk space | Disk < 85% heartbeat | 15 minutes |

Conclusion

ZNC is critical infrastructure for IRC-dependent teams — and now you have end-to-end visibility across the daemon, web UI, TLS certificates, and storage. Vigilmon's free tier handles all the monitors above. The next time ZNC's process crashes or a certificate silently expires, you'll have an alert in hand before anyone notices missing message history.

Sign up for Vigilmon free →