Monitoring Borgmatic with Vigilmon

Borgmatic is the most popular YAML-based wrapper for BorgBackup, turning complex backup shell scripts into a single configuration file with built-in database hooks (PostgreSQL, MySQL, MongoDB, MariaDB), retention policies, and before/after backup lifecycle hooks. Operations teams and self-hosters use it to automate zero-touch BorgBackup management across multiple repositories without writing shell scripts. The problem: Borgmatic itself has no outbound alerting. If systemd fails to start the timer, if a hook crashes silently, or if the backup completes but excludes a critical directory, you find out at restore time. Vigilmon integrates directly with Borgmatic's monitoring hooks to catch every missed or failed backup.

What You'll Set Up

• Borgmatic after_backup and on_error hook integration to POST to Vigilmon heartbeat on success and failure
• Heartbeat monitor for Borgmatic's scheduled run detection (alert when backup window passes without a ping)
• TCP port monitor for the underlying BorgBackup SSH repository server
• SSL certificate alerts for HTTPS backup repository endpoints configured in borgmatic.yaml

Prerequisites

• Borgmatic 1.7+ installed (pip install borgmatic or system package)
• A working borgmatic.yaml with at least one BorgBackup repository configured
• Borgmatic running via cron or systemd timer
• A free Vigilmon account

Step 1: Configure Borgmatic Monitoring Hooks for Vigilmon

Borgmatic has a built-in monitoring hooks section that sends HTTP pings on backup lifecycle events. This is the primary Vigilmon integration — configure it once in borgmatic.yaml and every backup run is automatically reported.

First, create a Cron Heartbeat monitor in Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to Cron Heartbeat.
3. Set Expected ping interval to match your borgmatic schedule (e.g. 1440 minutes for daily).
4. Set Grace period to 90 minutes to allow for database hook duration.
5. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Now add the monitoring hooks to /etc/borgmatic/config.yaml (or ~/.config/borgmatic/config.yaml):

hooks:
  # Ping Vigilmon on successful backup completion
  after_backup:
    - curl -fsS https://vigilmon.online/heartbeat/abc123

  # Ping a separate failure heartbeat if borgmatic encounters an error
  on_error:
    - curl -fsS "https://vigilmon.online/heartbeat/abc123?status=fail&msg=borgmatic-error"

The after_backup hook fires after all repositories have been backed up and pruned successfully. If any step fails, borgmatic skips after_backup and runs on_error instead, so the success heartbeat is never sent and Vigilmon alerts after the grace period expires.

Verify your configuration parses correctly:

borgmatic config validate

Step 2: Separate Error Alerting with a Second Heartbeat

For stricter alerting, use two heartbeats: one for success detection (missed runs) and one for active failure detection. Create a second Cron Heartbeat in Vigilmon (interval: 1440 minutes, grace: 10 minutes — it should only ever receive the failure ping, so configure it to alert immediately when it receives a ?status=fail param).

Update borgmatic.yaml:

hooks:
  after_backup:
    # Success ping — Vigilmon uses absence of this ping to detect missed backups
    - curl -fsS "https://vigilmon.online/heartbeat/abc123"

  on_error:
    # Active failure ping — immediate alert regardless of schedule
    - |
      curl -fsS \
        "https://vigilmon.online/heartbeat/def456?status=fail&msg=borgmatic+backup+failed+on+$(hostname)"

The on_error hook has access to environment variables that borgmatic sets — you can include $BORGMATIC_HOOK_ERROR_CODE and $BORGMATIC_HOOK_REPOSITORY to identify which repository and error triggered the alert.

Step 3: Monitor Last-Run Timestamp via Borgmatic Logs

Borgmatic writes structured logs to syslog and optionally to a log file. Add a simple shell heartbeat that parses the last borgmatic success timestamp and pings Vigilmon only when the log shows a recent successful run:

#!/bin/bash
# Check borgmatic ran successfully within the last 25 hours
VIGILMON_URL="https://vigilmon.online/heartbeat/ghi789"
LOG_FILE="/var/log/borgmatic.log"
MAX_AGE_HOURS=25

if [ -f "$LOG_FILE" ]; then
  # Look for the last successful completion marker
  LAST_SUCCESS=$(grep -E "summary.*completed" "$LOG_FILE" | tail -1 | awk '{print $1, $2}')
  if [ -n "$LAST_SUCCESS" ]; then
    LAST_TS=$(date -d "$LAST_SUCCESS" +%s 2>/dev/null)
    NOW=$(date +%s)
    AGE_HOURS=$(( (NOW - LAST_TS) / 3600 ))
    if [ "$AGE_HOURS" -lt "$MAX_AGE_HOURS" ]; then
      curl -fsS "$VIGILMON_URL" > /dev/null
    fi
  fi
fi

Add to crontab to run every hour:

0 * * * * /usr/local/bin/check-borgmatic-log.sh

This provides a belt-and-suspenders check that runs independently of borgmatic's hooks — useful when borgmatic itself fails to start (e.g. systemd timer misconfiguration).

Step 4: TCP Port Monitor for the BorgBackup SSH Repository

Borgmatic's hooks won't catch the case where the remote SSH repository server is unreachable before borgmatic even attempts the backup. Add a TCP monitor for immediate notification when the SSH server goes down:

1. In Vigilmon, click Add Monitor.
2. Set Type to TCP Port.
3. Enter your backup server hostname from borgmatic.yaml (e.g. backup.example.com).
4. Set Port to 22 (or your custom SSH port).
5. Set Check interval to 5 minutes.
6. Click Save.

For multiple repositories in borgmatic.yaml, add one TCP monitor per unique backup server:

# borgmatic.yaml — each of these servers gets a TCP monitor in Vigilmon
repositories:
  - path: user@primary-backup.example.com:/backups/main
    label: primary
  - path: user@offsite-backup.example.com:/backups/offsite
    label: offsite

Step 5: SSL Certificate Alerts for HTTPS Repository Endpoints

Borgmatic supports HTTPS-accessible repositories via rsync.net, BorgBase, or custom HTTPS frontends. An expired certificate on these endpoints blocks borgmatic from accessing the repository.

Add an HTTP monitor for each HTTPS repository endpoint in Vigilmon:

1. In Vigilmon, click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the base URL of your HTTPS repository host (e.g. https://borgbase.com or https://rsync.net).
4. Enable Monitor SSL certificate under the SSL section.
5. Set Alert when certificate expires in less than 21 days.
6. Click Save.

For self-hosted HTTPS frontends fronting borg serve, monitor the proxy health endpoint:

# Add to your nginx proxy config
location /health {
    return 200 "ok";
    add_header Content-Type text/plain;
}

Then monitor https://borg-proxy.example.com/health in Vigilmon with SSL monitoring enabled.

Step 6: Configure Alert Channels and Thresholds

1. Go to Alert Channels in Vigilmon and add email, Slack, or PagerDuty.
2. For the borgmatic heartbeat monitors, set consecutive failures before alert to 1 — a missed backup always warrants immediate notification.
3. For TCP and HTTP monitors, set consecutive failures to 2 to tolerate brief SSH server reboots.
4. Add a Maintenance window in Vigilmon when running borgmatic --export-tar or repository compaction (borg compact) which can delay normal backup timing.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat | after_backup hook URL | Missed backup, hook crash, borgmatic scheduler failure | | Error heartbeat | on_error hook URL | Active backup failures, database hook errors | | Log-parse heartbeat | Hourly cron check | Borgmatic fails to start (systemd timer issue) | | TCP port | backup-server:22 | SSH repository unreachable before backup runs | | SSL certificate | HTTPS repository host | TLS certificate expiry on HTTPS-backed remotes |

Borgmatic's YAML-native hooks make Vigilmon integration nearly zero-configuration — a single after_backup curl line is enough to catch missed backups across all configured repositories. Combined with TCP monitoring for SSH servers and SSL alerts for HTTPS endpoints, you get complete visibility into your entire Borgmatic backup pipeline.