tutorial

How to Monitor Litestream with Vigilmon

Monitor your Litestream SQLite replication process with Vigilmon — process health, replication lag, restore verification, and alerting for disaster recovery readiness.

Litestream continuously replicates your SQLite database to S3, GCS, or Azure Blob Storage by streaming WAL changes in real time. It is the backbone of disaster recovery for SQLite-backed applications — including Elixir/Phoenix apps using Ecto with the SQLite adapter — but Litestream itself runs as a sidecar process that can silently fall behind, crash, or lose its connection to object storage without producing any obvious application error. Vigilmon monitors the Litestream process and your replication health from the outside so you know before a disaster that your backup is still working.

In this guide you'll build a health endpoint that exposes Litestream replication state, set up Vigilmon HTTP and heartbeat monitors, and wire in alerting so your on-call team is notified the moment replication falls behind or the process dies.

What You'll Build

  • A /health/litestream endpoint that reports replication lag and last checkpoint time
  • A Vigilmon HTTP monitor that checks the endpoint every minute
  • A Vigilmon heartbeat for Litestream's background replication loop
  • A restore-verification cron job that proves your backup can actually be restored
  • Email and Slack alert channels

Prerequisites

  • Litestream installed (brew install benbjohnson/litestream/litestream or apt install litestream)
  • A SQLite-backed application (Elixir/Phoenix + Ecto with SQLite, or any other stack)
  • Object storage configured in litestream.yml (S3, GCS, or Azure Blob)
  • A free Vigilmon account

Step 1: Enable Litestream Metrics

Litestream exposes Prometheus metrics when you set addr in your config:

# litestream.yml
addr: "0.0.0.0:9090"   # metrics and health endpoint

dbs:
  - path: /data/app.db
    replicas:
      - type: s3
        bucket: my-app-backups
        path: app-db
        region: us-east-1
        access-key-id: $AWS_ACCESS_KEY_ID
        secret-access-key: $AWS_SECRET_ACCESS_KEY

With addr set, Litestream exposes:

  • GET /metrics — Prometheus metrics including replication lag and WAL size
  • GET / — HTML page with replica status

Verify it works:

litestream replicate -config litestream.yml &
curl -s http://localhost:9090/metrics | grep litestream

Key metrics:

| Metric | Description | |---|---| | litestream_replica_wal_index | Current WAL index being replicated | | litestream_replica_sync_count_total | Total sync operations completed | | litestream_replica_operation_bytes_total | Bytes uploaded to object storage | | litestream_db_wal_size | Current WAL file size in bytes |

Step 2: Build a Health Endpoint

Litestream's built-in / page is HTML, not machine-readable. Wrap the metrics in a purpose-built health endpoint so Vigilmon can assert on its response body.

For Elixir/Phoenix — add a controller:

# lib/my_app_web/controllers/litestream_health_controller.ex
defmodule MyAppWeb.LitestreamHealthController do
  use MyAppWeb, :controller

  @metrics_url "http://localhost:9090/metrics"
  @max_lag_seconds 300  # alert if replication is more than 5 minutes behind

  def check(conn, _params) do
    with {:ok, %{body: body, status: 200}} <- fetch_metrics(),
         {:ok, lag} <- parse_lag(body) do
      if lag <= @max_lag_seconds do
        json(conn, %{status: "ok", replication_lag_seconds: lag})
      else
        conn
        |> put_status(503)
        |> json(%{status: "degraded", replication_lag_seconds: lag,
                   message: "Replication lag exceeds threshold"})
      end
    else
      _ ->
        conn
        |> put_status(503)
        |> json(%{status: "error", message: "Cannot reach Litestream metrics endpoint"})
    end
  end

  defp fetch_metrics do
    :httpc.request(:get, {@metrics_url |> String.to_charlist(), []}, [], [])
    |> case do
      {:ok, {{_, status, _}, _headers, body}} ->
        {:ok, %{status: status, body: to_string(body)}}
      error -> error
    end
  end

  defp parse_lag(metrics_body) do
    # litestream_replica_wal_index{db=..., name=...} 42
    # Use the WAL index age as a proxy for lag — compare current time vs last sync
    # A simpler check: verify the process is running and syncing
    case Regex.scan(~r/litestream_replica_sync_count_total\{[^}]+\}\s+(\d+)/, metrics_body) do
      [[_, count] | _] -> {:ok, String.to_integer(count)}
      _ -> {:error, :no_metrics}
    end
  end
end

Add the route in router.ex:

scope "/health", MyAppWeb do
  get "/litestream", LitestreamHealthController, :check
end

For a standalone script (language-agnostic) — serve a health endpoint with Python:

#!/usr/bin/env python3
# litestream_health_server.py
import re
import time
import urllib.request
from http.server import HTTPServer, BaseHTTPRequestHandler
import json

METRICS_URL = "http://localhost:9090/metrics"
MAX_LAG_BYTES = 10 * 1024 * 1024  # 10 MB WAL backlog = degraded

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health/litestream":
            self.send_response(404)
            self.end_headers()
            return
        try:
            with urllib.request.urlopen(METRICS_URL, timeout=5) as r:
                body = r.read().decode()
            wal_match = re.search(r"litestream_db_wal_size\S*\s+([\d.]+)", body)
            wal_size = float(wal_match.group(1)) if wal_match else None
            healthy = wal_size is not None and wal_size < MAX_LAG_BYTES
            status = "ok" if healthy else "degraded"
            payload = json.dumps({"status": status, "wal_size_bytes": wal_size}).encode()
            self.send_response(200 if healthy else 503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(payload)
        except Exception as e:
            payload = json.dumps({"status": "error", "message": str(e)}).encode()
            self.send_response(503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(payload)

    def log_message(self, *args):
        pass

HTTPServer(("0.0.0.0", 9101), Handler).serve_forever()

Test it:

python3 litestream_health_server.py &
curl -s http://localhost:9101/health/litestream | jq .
# {"status": "ok", "wal_size_bytes": 4096.0}

Step 3: Set Up the Vigilmon HTTP Monitor

  1. Log in to Vigilmon and click New Monitor → HTTP.
  2. Enter http://your-server:9101/health/litestream (or https://your-domain.com/health/litestream if behind a reverse proxy).
  3. Set check interval to 60 seconds.
  4. Add assertions:
    • Status code equals 200
    • Response body contains "status":"ok"
  5. Save the monitor.

Vigilmon probes from multiple regions; two consecutive failures trigger your alert channels.

Step 4: Set Up a Heartbeat Monitor for Litestream

The HTTP health check verifies that Litestream responds, but it doesn't verify that replication is actively progressing. Set up a heartbeat that only pings Vigilmon after a confirmed replication event.

  1. In Vigilmon, go to New Monitor → Heartbeat.
  2. Set expected interval to 10 minutes.
  3. Copy the ping URL (e.g., https://vigilmon.online/ping/YOUR_HEARTBEAT_ID).

Write a script that checks replication progress and pings the heartbeat:

#!/usr/bin/env bash
# /usr/local/bin/litestream-heartbeat.sh
HEARTBEAT_URL="https://vigilmon.online/ping/YOUR_HEARTBEAT_ID"
METRICS_URL="http://localhost:9090/metrics"

# Get current WAL index
wal_index=$(curl -sf "$METRICS_URL" | \
  grep 'litestream_replica_wal_index' | \
  grep -v '^#' | \
  awk '{print $2}' | head -1)

if [[ -n "$wal_index" && "$wal_index" -gt 0 ]]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
  echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) heartbeat sent (wal_index=$wal_index)"
else
  echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) heartbeat withheld — replication not progressing"
fi

Add to cron:

chmod +x /usr/local/bin/litestream-heartbeat.sh
crontab -e
# Add:
# */10 * * * * /usr/local/bin/litestream-heartbeat.sh >> /var/log/litestream-heartbeat.log 2>&1

If Litestream crashes or WAL replication stalls, the heartbeat stops and Vigilmon alerts after the expected interval passes.

Step 5: Restore Verification Cron Job

Monitoring that Litestream is replicating is necessary but not sufficient — you also need to verify that the replicated data can actually be restored. Add a daily restore verification:

#!/usr/bin/env bash
# /usr/local/bin/litestream-verify-restore.sh
set -euo pipefail

RESTORE_PATH="/tmp/litestream-verify-$(date +%s).db"
SOURCE_DB="/data/app.db"
HEARTBEAT_URL="https://vigilmon.online/ping/YOUR_RESTORE_VERIFY_HEARTBEAT_ID"

cleanup() { rm -f "$RESTORE_PATH"; }
trap cleanup EXIT

# Attempt restore from object storage
litestream restore \
  -config /etc/litestream.yml \
  -replica s3 \
  -o "$RESTORE_PATH" \
  "$SOURCE_DB"

# Verify the restored database is readable
row_count=$(sqlite3 "$RESTORE_PATH" "SELECT COUNT(*) FROM sqlite_master;")
if [[ "$row_count" -ge 0 ]]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
  echo "$(date -u) Restore verified OK (sqlite_master rows: $row_count)"
else
  echo "$(date -u) Restore verification FAILED — database not readable"
  exit 1
fi

Create a separate Vigilmon heartbeat with a 24-hour interval for restore verification. Schedule the script as a daily cron:

chmod +x /usr/local/bin/litestream-verify-restore.sh
# Run at 03:00 UTC daily
echo "0 3 * * * root /usr/local/bin/litestream-verify-restore.sh >> /var/log/litestream-verify.log 2>&1" \
  >> /etc/cron.d/litestream-verify

Step 6: Supervise the Litestream Process

Litestream should run under a process supervisor so it restarts automatically on crash. Use systemd:

# /etc/systemd/system/litestream.service
[Unit]
Description=Litestream SQLite Replication
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=app
ExecStart=/usr/bin/litestream replicate -config /etc/litestream.yml
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal

# Environment for S3 credentials
EnvironmentFile=/etc/litestream.env

[Install]
WantedBy=multi-user.target

Enable and start:

systemctl daemon-reload
systemctl enable --now litestream
systemctl status litestream

Verify the process is running and metrics are available:

curl -s http://localhost:9090/metrics | grep -c litestream

Step 7: Configure Alert Channels

In Vigilmon under Alerts → Channels:

Email — add your on-call address. You'll receive an alert within 30 seconds of two consecutive HTTP check failures and a recovery notification when the check passes again.

Slack — click Add Channel → Slack, paste your incoming webhook URL. Route CRITICAL monitors to both email and Slack under Alerts → Routing.

Create two alert policies:

  • Litestream process health (HTTP monitor): notify immediately on first failure
  • Restore verification (heartbeat): notify if restore hasn't been verified in 25 hours

Step 8: Verify End-to-End

  1. Confirm the Vigilmon dashboard shows your HTTP monitor as UP.
  2. Stop the Litestream process: systemctl stop litestream — verify Vigilmon fires an alert.
  3. Restart: systemctl start litestream — verify you receive a recovery notification.
  4. Trigger the restore verification script manually and confirm the heartbeat is received in the Vigilmon dashboard.

Production Checklist

  • [ ] addr set in litestream.yml to expose Prometheus metrics
  • [ ] Health endpoint (/health/litestream) returning 200 with JSON body
  • [ ] Vigilmon HTTP monitor with status code + body assertions
  • [ ] Heartbeat monitor (10-minute interval) for replication progress
  • [ ] Heartbeat monitor (24-hour interval) for restore verification
  • [ ] Litestream running under systemd with Restart=on-failure
  • [ ] Daily restore verification cron job
  • [ ] Email and Slack alert channels configured

Summary

You now have Litestream monitored end-to-end with Vigilmon:

  • HTTP monitor verifying the Litestream process is alive and WAL replication is not backed up
  • Heartbeat confirming replication is actively progressing every 10 minutes
  • Daily restore verification proving your backup can actually be recovered
  • Email and Slack alerts firing within seconds of any failure

Litestream gives SQLite-backed applications S3-grade durability. Vigilmon makes sure Litestream itself never silently stops working.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles