Monitoring Fossil SCM with Vigilmon: Web Server, Timeline, Clone Endpoint & Process Heartbeat Checks

Fossil SCM is an unusual distributed version control system: unlike Git, every Fossil repository is a single self-contained SQLite file that ships its own built-in web server, bug tracker, wiki, and forum. Teams and projects like SQLite itself use Fossil for its simplicity, auditability, and the fact that the entire project history fits in a single file. When the Fossil server process goes down, developers lose access to the repository web UI, the clone/sync endpoint, and all the integrated project management features. Vigilmon gives you external visibility into the Fossil web interface, the repository content layer, the HTTP sync endpoint used by fossil clone and fossil sync, and the health of the Fossil server process itself.

What You'll Build

• An HTTP monitor on the Fossil web server's status page
• A repository timeline check that confirms the SQLite database is readable
• An HTTP clone/sync endpoint probe for transport-layer health
• A process heartbeat to detect when the fossil server daemon has crashed
• SSL certificate expiry alerting for your Fossil domain

Prerequisites

• Fossil server running (either fossil server standalone or behind a reverse proxy on port 8080 or 443)
• At least one .fossil repository file accessible to the running process
• A free account at vigilmon.online

Step 1: Understand Fossil's Architecture

Fossil can be served in several ways:

• fossil server repo.fossil — Fossil's built-in HTTP server on port 8080 (or configured port)
• CGI mode — Fossil binary invoked as a CGI script via Apache or nginx
• fossil server --port 8080 behind a reverse proxy on port 443

The monitoring strategy is the same in all cases: probe the HTTP endpoints Fossil exposes. The key difference is whether you're hitting Fossil directly on its native port or through a proxy.

Step 2: Monitor the Fossil Web Interface

Fossil exposes a /stat endpoint that returns repository statistics without requiring authentication. This is the ideal health check — it requires Fossil to open the SQLite database and read commit counts, making it a comprehensive liveness check:

curl https://fossil.yourdomain.com/stat
# Returns the repository stat page with timeline info

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://fossil.yourdomain.com/stat
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: Repository Size (text present on the stat page).
7. Label: Fossil Web Server
8. Click Save.

A 200 on /stat confirms:

• The fossil server process is running
• The SQLite .fossil database file is readable
• The HTTP layer (or reverse proxy) is routing correctly

Step 3: Monitor the Repository Timeline

The /timeline page renders the commit log and is the most database-intensive page in Fossil's web UI. Monitoring it separately from /stat verifies that the timeline query layer is healthy:

1. Add Monitor → HTTP.
2. URL: https://fossil.yourdomain.com/timeline
3. Expected status: 200.
4. Keyword: Timeline (present in the page heading).
5. Check interval: 120 seconds.
6. Label: Fossil Timeline Health

If /stat passes but /timeline fails, the timeline query against the SQLite event table is failing — this can indicate database corruption or a very large repository overwhelming the query planner.

Step 4: Verify the HTTP Clone and Sync Endpoint

Fossil's distributed workflow relies on HTTP sync. Developers run fossil clone https://fossil.yourdomain.com/ to clone and fossil sync to exchange commits. The /xfer endpoint handles this protocol. Probe it to verify sync transport is working:

curl -I https://fossil.yourdomain.com/login
# HTTP/2 200  — confirms Fossil's HTTP protocol layer is responding

For a more targeted test, check the JSON info endpoint if your Fossil version supports it:

curl https://fossil.yourdomain.com/json/stat
# {"fossil":"2.23","procId":...,"sqlite":"3.43.0",...}

1. Add Monitor → HTTP.
2. URL: https://fossil.yourdomain.com/json/stat
3. Expected status: 200.
4. Keyword: fossil (present in the JSON response).
5. Check interval: 300 seconds.
6. Label: Fossil HTTP Sync Endpoint

If your Fossil version predates the JSON API, use /login as the probe URL with expected status 200.

Step 5: Process Heartbeat Monitor for the Fossil Server Daemon

Unlike Git-based tools with persistent web server processes managed by systemd, a fossil server process can be killed without a service manager noticing. Set up a cron heartbeat that only fires when the Fossil server process is running:

#!/bin/bash
# /etc/cron.d/fossil-heartbeat — runs every 5 minutes

if pgrep -x "fossil" > /dev/null; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-HEARTBEAT-ID
fi

Or verify the HTTP endpoint directly from within the server:

#!/bin/bash
# /etc/cron.d/fossil-heartbeat — runs every 5 minutes

if curl -fsS "http://localhost:8080/stat" > /dev/null 2>&1; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-HEARTBEAT-ID
fi

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: 5 minutes.
3. Grace period: 15 minutes.
4. Label: Fossil Server Process

Step 6: Monitor the SQLite Database File Size

Fossil stores everything — all commits, wiki pages, bug reports, and forum posts — in a single SQLite file. As this file grows, query performance degrades and backup times increase. Monitor disk usage where the .fossil file lives:

#!/bin/bash
# /etc/cron.d/fossil-disk-heartbeat — runs every 5 minutes

DISK_USAGE=$(df /home/fossil --output=pcent | tail -1 | tr -d '% ')
REPO_SIZE_MB=$(du -sm /home/fossil/repo.fossil | cut -f1)

# Alert if disk > 85% or repo file > 5 GB
if [ "$DISK_USAGE" -lt 85 ] && [ "$REPO_SIZE_MB" -lt 5120 ]; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-DISK-HEARTBEAT-ID
fi

In Vigilmon, create a separate Heartbeat monitor with a 15-minute grace period labeled Fossil Disk Health.

Step 7: Monitor Your SSL Certificate

If Fossil is served over HTTPS (either natively with --tls or via a reverse proxy), a lapsed SSL certificate blocks all web UI access and all fossil clone/fossil sync operations:

1. Add Monitor → SSL Certificate.
2. Domain: fossil.yourdomain.com
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days.

Fossil deployments are often long-lived, minimal-maintenance servers. The 30-day advance warning catches renewal failures well before developers lose sync access.

Common Fossil SCM Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | fossil server process killed or crashed | Process heartbeat stops; alert fires | | SQLite database file locked or corrupted | Web server check fails (SQLite error 500) | | Timeline query fails on large repo | Timeline monitor catches it | | HTTP sync endpoint broken after upgrade | JSON stat check fails | | Reverse proxy misconfigured | Web server check returns 502 | | Let's Encrypt certificate expired | SSL monitor alerts at 30 days | | Disk full — SQLite write failure | Disk heartbeat stops; alert fires | | Repository file moved or permissions changed | All web monitors fail simultaneously | | DNS misconfiguration | All monitors fire simultaneously |

Fossil's design philosophy is self-contained simplicity, but that simplicity extends to its lack of built-in health monitoring. A Vigilmon setup covering the web interface, timeline query health, HTTP sync transport, process liveness, SQLite disk usage, and SSL certificates gives you complete external visibility over what is otherwise a black box.

Start monitoring Fossil SCM in under 5 minutes — register free at vigilmon.online.