Monitoring Atheme IRC Services with Vigilmon

Atheme IRC Services is the backbone of identity and channel management on self-hosted IRC networks — when it goes offline, there's no NickServ authentication, no ChanServ mode enforcement, and no MemoServ delivery. Vigilmon helps you monitor the Atheme daemon, its connection to the IRCd, the optional XMLRPC/JSON-RPC API, and the email relay so you catch failures before your users do.

What You'll Set Up

• Cron heartbeat for the Atheme daemon process
• TCP port check for the IRCd uplink (server-to-server link)
• HTTP monitor for the XMLRPC/JSON-RPC API (port 8080)
• SMTP connectivity monitor for outbound email
• Database file freshness monitoring via a heartbeat script
• Process heartbeat for detecting silent crashes

Prerequisites

• Atheme IRC Services installed and running, linked to an IRCd (UnrealIRCd, InspIRCd, or similar)
• A free Vigilmon account

Step 1: Monitor the Atheme Process with a Heartbeat

The most critical signal is whether the atheme daemon is alive. A process heartbeat catches crashes that don't leave any port open:

1. Log in to vigilmon.online and click Add Monitor → Cron Job / Heartbeat.
2. Set Expected interval to 2 minutes.
3. Copy the heartbeat URL (e.g., https://vigilmon.online/ping/your-heartbeat-id).

Add a cron job on the services host:

# /etc/cron.d/atheme-heartbeat
*/2 * * * * root pgrep -x atheme > /dev/null && curl -fsS --retry 3 https://vigilmon.online/ping/your-heartbeat-id

If Atheme crashes, the pgrep guard prevents the ping, and Vigilmon marks the heartbeat as missed.

To automatically restart Atheme on failure, pair this with a systemd service:

[Service]
ExecStart=/usr/local/sbin/atheme-services -n
Restart=on-failure
RestartSec=10

Step 2: Monitor the IRCd Uplink (S2S Link)

Atheme connects to your IRCd (e.g., UnrealIRCd) via a server-to-server protocol link. If this link drops, all IRC services disappear from the network — users see NickServ go offline. Monitor the link listener port that Atheme's uplink IRCd exposes:

1. Click Add Monitor → TCP Port.
2. Enter the IRCd server's IP and set Port to the link port (the port configured in Atheme's serverinfo / uplink block — commonly 7029 or 6900; check your atheme.conf).
3. Set Check interval to 1 minute.
4. Click Save.

This validates that the IRCd is accepting service-to-server connections on that port. If the IRCd restarts and the link doesn't re-establish, Vigilmon will alert you via this TCP check within a minute.

For deeper link verification, add a heartbeat script that parses Atheme's log for successful uplink messages:

#!/bin/bash
# atheme-link-health.sh
LOGFILE="/var/log/atheme/atheme.log"
# Alert if no successful link event in the last 10 minutes
RECENT=$(grep "Server \[" "$LOGFILE" | awk -v cutoff="$(date -d '10 minutes ago' '+%F %T')" '$0 > cutoff' | wc -l)
[ "$RECENT" -gt 0 ] && curl -fsS https://vigilmon.online/ping/your-link-heartbeat-id

Step 3: Monitor the XMLRPC / JSON-RPC API

If you run web panels (Anope's webpanel, custom bots, or IRC bridges) that talk to Atheme via its XMLRPC or JSON-RPC interface on port 8080, API availability is critical:

1. Click Add Monitor → HTTP / HTTPS.
2. Set the URL to http://your-server:8080/xmlrpc (or /jsonrpc depending on your config).
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

The Atheme XMLRPC endpoint returns a 200 with an XML response even for unauthenticated requests — a non-200 response means the API server has stopped.

If you restrict the API to specific source IPs via Atheme's config, the Vigilmon check IP may be blocked. In that case, run a local health-check forwarder:

# On the Atheme server — expose API health to external monitor
socat TCP-LISTEN:18080,reuseaddr,fork TCP:127.0.0.1:8080 &

Then point your Vigilmon monitor at port 18080 on the public interface.

Step 4: Monitor the Atheme Database File

Atheme stores its entire state — registered nicknames, channels, memos, group memberships — in a flat text file (atheme.db by default). If the database file hasn't been written recently, it could indicate a save failure or filesystem issue. Monitor the file's last-modified timestamp with a heartbeat script:

1. Create a Vigilmon Cron Job / Heartbeat monitor with Expected interval 10 minutes.
2. Copy the heartbeat URL.

Add a cron job that only pings if the database was written in the last 15 minutes:

# /etc/cron.d/atheme-db-heartbeat
*/10 * * * * root \
  DB="/var/lib/atheme/atheme.db"; \
  [ -f "$DB" ] && \
  [ $(( $(date +%s) - $(stat -c %Y "$DB") )) -lt 900 ] && \
  curl -fsS https://vigilmon.online/ping/your-db-heartbeat-id

Atheme writes its database on a configurable interval (the commit_interval setting, default every 5 minutes). If the file age exceeds 15 minutes, the ping is skipped and Vigilmon alerts.

Also monitor for file write permissions — a read-only filesystem will silently prevent saves:

# Check write permission before pinging
[ -w "$DB" ] || { logger "atheme.db not writable!"; exit 1; }

Step 5: Monitor Email / SMTP Connectivity

Atheme can send emails for nick registration confirmation and password resets. If the SMTP relay is unreachable, new user registrations stall and password resets silently fail. Monitor the SMTP server directly:

1. Click Add Monitor → TCP Port.
2. Enter the SMTP relay hostname and set Port to 587 (submission) or 25.
3. Set Check interval to 5 minutes.
4. Click Save.

If you use a local Postfix/Stalwart relay, monitor localhost:25. For external SMTP relays (SendGrid, Mailgun), monitor their submission endpoint directly.

For higher-confidence email health, add an HTTP check on the relay's status page if it provides one.

Step 6: Monitor ChanServ and MemoServ Presence

ChanServ mode enforcement and MemoServ delivery both require Atheme to be fully linked and functional. You can validate these services are responding to IRC commands by building a lightweight IRC bot health checker:

#!/usr/bin/env python3
# atheme-chanserv-health.py — connect to IRC and check CHANSERV is present
import socket, time, requests, sys

HOST, PORT = "irc.yourdomain.com", 6667
PING_URL   = "https://vigilmon.online/ping/your-chanserv-heartbeat-id"

try:
    s = socket.create_connection((HOST, PORT), timeout=10)
    s.send(b"NICK healthbot\r\nUSER healthbot 0 * :healthbot\r\n")
    time.sleep(2)
    s.send(b"WHOIS ChanServ\r\n")
    time.sleep(2)
    resp = s.recv(4096).decode(errors="ignore")
    s.send(b"QUIT\r\n")
    s.close()
    if "ChanServ" in resp:
        requests.get(PING_URL, timeout=5)
except Exception as e:
    print(f"Health check failed: {e}", file=sys.stderr)
    sys.exit(1)

Run this every 5 minutes via cron and point it at a Vigilmon heartbeat.

Step 7: Module Load Monitoring

Atheme loads its functionality as modules at startup. Failed module loads are logged but don't crash the daemon — they silently disable features. Monitor for module errors in the Atheme log:

1. Create a Vigilmon Cron Job / Heartbeat monitor with Expected interval 10 minutes.
2. Use a cron script that checks the log for recent module errors:

# /etc/cron.d/atheme-module-check
*/10 * * * * root \
  ERRORS=$(grep -c "failed to load module" /var/log/atheme/atheme.log 2>/dev/null); \
  [ "${ERRORS:-0}" -eq 0 ] && curl -fsS https://vigilmon.online/ping/your-module-heartbeat-id

If any module failed to load since last log rotation, the ping is skipped and Vigilmon alerts.

Step 8: Configure Alerts

1. Go to Settings → Alerts in Vigilmon.
2. Add a Slack webhook or email alert channel.
3. Set 2 consecutive failures as the threshold to avoid transient false positives.
4. Apply the alert profile to all Atheme monitors.

Recommended priority:

| Monitor | Priority | Notes | |---|---|---| | Atheme process heartbeat | Critical | Services are fully offline | | IRCd uplink TCP check | Critical | NickServ/ChanServ invisible to users | | XMLRPC/JSON-RPC API | High | Web panels and bots lose access | | Database file freshness | High | Data loss risk if saves stop | | SMTP connectivity | Medium | Registration/password flow broken | | Module load check | Low | Feature degradation, not outage |

What to Watch

| Monitor | Type | Check | Alert threshold | |---|---|---|---| | Atheme process | Heartbeat | Cron ping every 2 min | 2 missed | | IRCd uplink port | TCP | Link port open | 1 failure | | XMLRPC API | HTTP | Port 8080 responds 200 | 2 failures | | Database freshness | Heartbeat | File age < 15 min | 2 missed | | SMTP relay | TCP | Port 587 / 25 open | 2 failures | | ChanServ presence | Heartbeat | WHOIS responds | 2 missed | | Module load log | Heartbeat | No errors in log | 2 missed |

Conclusion

Atheme is the authority for every registered nickname and channel on your IRC network — when it goes silent, user trust erodes fast. Vigilmon's free tier covers all the monitors configured here, giving you full-stack visibility into the daemon, the IRCd link, the API, and the database without any extra infrastructure.

Sign up for Vigilmon free →