tutorial

Monitoring Restic with Vigilmon: REST Server Health, Backup Freshness & Storage Backend Alerts

How to monitor Restic backup infrastructure with Vigilmon — REST server health checks, S3 backend connectivity, backup job completion via HTTP hooks, and repository lock monitoring.

Restic is a fast, encrypted backup program that works with local storage, SFTP, S3-compatible object storage, Backblaze B2, and a dozen other backends. Its simplicity is its strength — a single binary, no database, content-addressed deduplication. But that simplicity means there is no built-in operations dashboard, no scheduled-job health feed, and no external visibility into whether backups are actually completing. If your nightly restic backup cron job fails silently — due to a locked repository, a full disk, an S3 outage, or a misconfigured environment variable — you will not discover the failure until you attempt a restore. Vigilmon gives you external visibility into your Restic infrastructure: the REST server health endpoint, storage backend connectivity, backup job completion via HTTP alerting hooks, and repository freshness checks.

What You'll Build

  • A monitor on the Restic REST server health endpoint to detect server failures
  • An S3 storage backend connectivity check
  • A backup freshness monitor using Restic's --after-backup-hook HTTP alerting integration
  • Repository lock monitoring to catch stuck backup jobs
  • A scheduled check that verifies backup recency against your retention window

Prerequisites

  • Restic running as a scheduled job (cron, systemd timer, Kubernetes CronJob, or CI pipeline)
  • Optionally: Restic REST server (rest-server) deployed for remote backup target access
  • An HTTP endpoint your backup jobs can POST to after completion (simple healthcheck service or Vigilmon webhook)
  • A free account at vigilmon.online

Step 1: Understand Restic's Monitoring Surface

Restic is a client-side tool — it has no persistent daemon, no management API, and no health endpoint by default. Monitoring Restic requires instrumenting the backup jobs themselves and optionally monitoring the REST server if you use one as a backup target.

The Restic REST server (rest-server) exposes a health endpoint on its configured port (default 8000):

curl http://restic-rest-server.example.com:8000/
# Returns: 200 OK with the repository list page or a 404 for empty repos

For a more explicit health check:

curl -o /dev/null -w "%{http_code}" http://restic-rest-server.example.com:8000/
# Returns: 200

If you back up directly to S3, B2, or Azure Blob Storage, the REST server is not in play — you instead monitor storage backend connectivity and backup job output directly.

Step 2: Create a Vigilmon HTTP Monitor for the Restic REST Server

If you run rest-server as a centralized backup target:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://restic-rest-server.example.com:8000/.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Label: Restic REST server.
  7. Click Save.

This monitor catches:

  • REST server process crashes or OOM kills
  • Port binding failures after configuration changes
  • TLS certificate failures if the server is configured with HTTPS
  • Host machine restarts that leave the server service offline
  • Disk full conditions on the REST server host that prevent the process from starting

Alert sensitivity: Set to trigger after 1 consecutive failure. When the REST server is down, all Restic clients targeting it will fail their backup jobs immediately, and every restic backup invocation returns a connection error.

Step 3: Monitor S3 Storage Backend Connectivity

For Restic deployments that target S3-compatible storage (AWS S3, Wasabi, MinIO, Backblaze B2 via the S3-compatible API), monitor the bucket endpoint directly:

# Check S3 endpoint connectivity (replace with your endpoint)
curl -o /dev/null -w "%{http_code}" https://s3.amazonaws.com/your-restic-bucket/
# Returns: 403 (expected — bucket exists, credentials not provided in curl)

A 403 Forbidden is the expected response when querying S3 without credentials — it proves the S3 endpoint is reachable and the bucket exists. A 404 means the bucket is gone; a connection timeout means the S3 endpoint is unreachable.

  1. Add Monitor → HTTP.
  2. URL: https://s3.amazonaws.com/your-restic-bucket/ (or your MinIO/Wasabi endpoint).
  3. Check interval: 5 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 403.
  6. Label: Restic S3 backend.
  7. Click Save.

For MinIO deployed in-cluster, use the MinIO health endpoint at /minio/health/live instead: curl https://minio.example.com/minio/health/live returns 200 OK without credentials, making it a cleaner liveness check than the bucket endpoint.

Step 4: Configure Backup Job Completion Alerting with HTTP Hooks

Restic 0.14+ supports --after-backup-hook flags that run a command after a successful backup. Use this to POST to a heartbeat URL that Vigilmon monitors for freshness. The typical pattern is to use a simple HTTP healthcheck service like healthchecks.io or a custom endpoint, but the monitoring principle is the same regardless of service:

# In your restic backup script or cron command:
restic backup /data \
  --repo s3:s3.amazonaws.com/your-restic-bucket \
  --password-file /etc/restic/password && \
curl -s -X POST https://backup-heartbeat.example.com/restic/ping

For systemd timer-based backups, add the ping to your ExecStartPost directive:

[Service]
ExecStart=/usr/local/bin/restic backup /data --repo ...
ExecStartPost=/bin/bash -c "[ $EXIT_STATUS -eq 0 ] && curl -s -X POST https://backup-heartbeat.example.com/restic/ping"

Then verify the heartbeat:

curl https://backup-heartbeat.example.com/restic/status
# Returns: {"last_ping": "2026-07-01T02:00:00Z", "status": "ok"}
  1. Add Monitor → HTTP.
  2. URL: https://backup-heartbeat.example.com/restic/status.
  3. Check interval: 6 hours (or half your backup schedule interval for early warning).
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Restic backup freshness.
  8. Click Save.

Step 5: Monitor Repository Lock Status

Restic uses file-based locks in the repository to prevent concurrent writes. A lock left behind by a crashed backup job prevents all subsequent backups from running until the lock is manually removed with restic unlock. Lock monitoring requires an external check that queries the repository lock state:

# Check if any locks exist in the repository
restic --repo s3:s3.amazonaws.com/your-restic-bucket --password-file /etc/restic/password \
  list locks
# Returns empty output when no locks; returns lock IDs when stuck

Automate this check as a cron job that writes to a health endpoint:

# Cron: run every hour, expose result as HTTP endpoint
LOCK_COUNT=$(restic list locks 2>/dev/null | wc -l)
if [ "$LOCK_COUNT" -eq 0 ]; then
  curl -X POST https://backup-heartbeat.example.com/restic/lock-status \
    -d '{"locks": 0, "status": "ok"}'
else
  curl -X POST https://backup-heartbeat.example.com/restic/lock-status \
    -d "{\"locks\": $LOCK_COUNT, \"status\": \"locked\"}"
fi
  1. Add Monitor → HTTP.
  2. URL: https://backup-heartbeat.example.com/restic/lock-status.
  3. Check interval: 1 hour.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "status": "ok".
  7. Label: Restic repository lock.
  8. Click Save.

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | REST server | Non-200 or connection error | Restart rest-server service; check disk space on backup host | | S3 backend | Connection timeout or 404 | Check S3 bucket exists; verify IAM permissions; check VPC routing | | Backup freshness | ok keyword missing or stale | Check cron/systemd timer logs; verify restic binary is executable; check storage credentials | | Repository lock | "status": "ok" missing | Run restic unlock after verifying no backup job is actively running |

Alert after: 1 consecutive failure for REST server. 2 consecutive failures for backup freshness and lock status.

Common Restic Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | REST server process crash | HTTP monitor returns connection error; alert within 2 min | | S3 bucket accidentally deleted | S3 backend returns 404; alert within 5 min | | Backup cron job silently failing | Backup freshness heartbeat goes stale; alert at 6-hour check | | Repository locked by crashed job | Lock status monitor returns locked; alert within 1 hour | | Storage disk full on backup host | REST server stops accepting writes; backup jobs error; freshness alert fires | | S3 credentials rotated without updating restic env | Backup jobs fail with auth error; freshness heartbeat stops; alert fires | | Network partition isolating backup host | REST server unreachable; S3 endpoint unreachable; both monitors fire | | restic binary updated, incompatible repository version | Backup jobs fail; freshness heartbeat stops updating | | Retention policy prune fills disk | REST server host disk full; subsequent backups fail | | Forget/prune script removes too many snapshots | Freshness monitor may stay green; requires snapshot count monitoring |

Restic's strength is its simplicity, but that same simplicity means failed backup jobs disappear into log files with no external alert. Vigilmon gives you external visibility into your Restic infrastructure — the REST server availability, storage backend connectivity, backup job completion freshness, and repository lock health — so you know before your next disaster recovery test whether your backups are actually running and your data is actually protected.

Start monitoring Restic in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles