tutorial

Monitoring SourceHut with Vigilmon

SourceHut (sr.ht) is a federation of independent services — git.sr.ht, builds.sr.ht, todo.sr.ht, and more — each running on its own port. Here's how to monitor every sr.ht service independently with Vigilmon so a single failing component doesn't go unnoticed.

SourceHut is a software forge built as a suite of loosely-coupled services: git.sr.ht hosts repositories, builds.sr.ht runs CI pipelines, todo.sr.ht tracks issues, lists.sr.ht handles mailing lists, and several more. Each service runs as its own Python web application behind an nginx reverse proxy, which means each one can fail independently. Vigilmon lets you monitor every sr.ht service as a separate uptime check so you know exactly which component went down — not just that "SourceHut is broken."

What You'll Set Up

  • Individual HTTP uptime monitors for each sr.ht service
  • Cron heartbeat for the builds.sr.ht job queue worker
  • SSL certificate expiry alerts for each service subdomain
  • TCP port check for the nginx reverse proxy

Prerequisites

  • SourceHut installed via the official srht meta-package (or individual service packages)
  • Each service accessible on its own subdomain (e.g. git.yourdomain.com, builds.yourdomain.com)
  • A free Vigilmon account

Step 1: Monitor git.sr.ht

The Git hosting service (git.sr.ht) is typically the most critical component. It runs on an internal port (default 5001) and is proxied by nginx on port 80/443.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your git service URL: https://git.yourdomain.com.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

SourceHut services return their main page with a 200 when healthy. A 502 from nginx means the upstream Python worker (gunicorn or waitress) has crashed.

Step 2: Monitor builds.sr.ht

The build service (builds.sr.ht) runs separately from the Git service and can fail independently. It handles CI job submission and log streaming.

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: https://builds.yourdomain.com.
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.
  5. Click Save.

If you're self-hosting the full SourceHut suite, builds.sr.ht also runs background build workers (buildsrht-worker) that execute CI jobs. These are separate processes from the web frontend — Step 5 covers monitoring them.

Step 3: Monitor Additional sr.ht Services

Each SourceHut service you've enabled needs its own monitor. Add one for each subdomain you've configured:

todo.sr.ht (issue tracker):

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: https://todo.yourdomain.com.
  3. Check interval: 2 minutes. Expected status: 200. Save.

lists.sr.ht (mailing lists):

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: https://lists.yourdomain.com.
  3. Check interval: 2 minutes. Expected status: 200. Save.

meta.sr.ht (user accounts and OAuth):

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: https://meta.yourdomain.com.
  3. Check interval: 1 minute. Expected status: 200. Save.

meta.sr.ht is worth monitoring at 1-minute intervals because it handles authentication for all other services — if it goes down, users can't log in to any part of SourceHut.

paste.sr.ht and hub.sr.ht (if enabled): Add monitors with 5-minute intervals — these are lower-priority services.

Step 4: TCP Port Monitor for nginx

All sr.ht services sit behind a shared nginx reverse proxy. If nginx goes down, every service returns a connection refused error. Add a single TCP check on the nginx port:

  1. Click Add Monitor → TCP Port.
  2. Enter your server hostname.
  3. Set Port to 443 (HTTPS) or 80 (HTTP if not using TLS).
  4. Set Check interval to 1 minute.
  5. Click Save.

This is your "everything is down" signal. If this TCP check fails while individual HTTP checks also fail, the nginx layer is the culprit rather than an individual service.

Step 5: Cron Heartbeat for the builds.sr.ht Job Queue

SourceHut's build system uses a background worker daemon to pick up and execute CI jobs. The worker is separate from the builds.sr.ht web frontend — the web UI can be up while the job queue is stuck.

Add a heartbeat job to one of your SourceHut build manifests:

image: alpine/latest
packages:
  - curl
tasks:
  - heartbeat: |
      curl -fs https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID

Submit this as a scheduled build using the SourceHut API or the hut CLI tool (set it to run on a cron schedule in a repository build trigger). Then in Vigilmon:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval slightly longer than your build schedule (e.g. 65 minutes for an hourly trigger).
  3. Copy the heartbeat URL and paste it into the build manifest above.
  4. Click Save.

If the build worker is stuck processing a stalled job, or if the worker daemon has crashed, the scheduled build won't run and Vigilmon will alert.

You can also send the heartbeat directly from the build worker host using a cron job on the server:

# /etc/cron.hourly/vigilmon-builds-heartbeat
#!/bin/bash
systemctl is-active --quiet buildsrht-worker && \
  curl -fs https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID

This variant only pings Vigilmon when the worker service is actually active — if buildsrht-worker is stopped or failed, the ping doesn't happen and Vigilmon alerts.

Step 6: SSL Certificate Alerts

SourceHut services each have their own subdomain and certificate. Add SSL monitoring for each one:

For each service monitor created in Steps 1–3:

  1. Open the monitor.
  2. Enable Monitor SSL certificate.
  3. Set Alert when certificate expires in less than 21 days.
  4. Click Save.

With multiple subdomains, you may be using a wildcard certificate (*.yourdomain.com) or individual Let's Encrypt certificates per service. Either way, Vigilmon will alert if the certificate being served by nginx is within the expiry window.

To check which certificate nginx is serving for a service:

openssl s_client -connect git.yourdomain.com:443 -servername git.yourdomain.com \
  </dev/null 2>/dev/null | openssl x509 -noout -dates

Step 7: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
  2. For meta.sr.ht, set Consecutive failures before alert to 1 — authentication downtime affects all users immediately.
  3. For secondary services (paste, hub), set Consecutive failures before alert to 3 to avoid noise from brief restarts.
  4. Group your monitors in Vigilmon with a sourcehut label to get a dashboard view of all sr.ht service health at once.

Summary

| Monitor | Target | What It Catches | |---|---|---| | git.sr.ht | https://git.yourdomain.com | Git worker crash, 502 from nginx | | builds.sr.ht | https://builds.yourdomain.com | Build frontend down | | meta.sr.ht | https://meta.yourdomain.com | Auth outage affecting all services | | todo.sr.ht | https://todo.yourdomain.com | Issue tracker unavailable | | lists.sr.ht | https://lists.yourdomain.com | Mailing list service down | | nginx TCP | TCP :443 | Entire reverse proxy layer down | | Build worker heartbeat | Heartbeat URL | CI job queue stalled or worker crashed | | SSL certificates | All subdomains | Certificate expiry per service |

SourceHut's microservice architecture means granular monitoring pays off — each service can fail independently, and a single Vigilmon dashboard showing all seven services gives you instant root-cause visibility when something goes wrong.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles