Garage Monitoring with Vigilmon: S3 API, Admin API, Cluster Node Health & Bucket Endpoint Checks

You replaced your cloud S3 costs with a self-hosted Garage cluster across three nodes. Backups, media uploads, and application assets all flow through it. One Saturday afternoon, a node lost its network interface after a kernel update. Garage continued operating — it's designed for node failures — but it entered a degraded state where write durability dropped below your configured replication factor. No alert fired. Backups "succeeded" by writing to the healthy nodes, but the cluster was one more node failure away from data unavailability. The degraded state lasted until Monday morning when someone logged in and noticed the admin dashboard.

Garage is a self-hosted distributed S3-compatible object storage system designed for lightweight, multi-node deployments. Its design tolerates node failures, but tolerating a failure is different from silently operating in a degraded state. Vigilmon gives you the external monitoring to detect Garage health changes — from a crashed S3 API to a cluster node going offline — before they compound into data availability incidents.

This tutorial walks through monitoring Garage end-to-end with Vigilmon.

What You'll Build

• A Vigilmon HTTP monitor on the S3 API endpoint
• An admin API health check
• Cluster node health monitoring
• A bucket endpoint response check
• SSL certificate expiry alerts

Prerequisites

• A running Garage cluster (v0.9.x or later)
• A free account at vigilmon.online
• Your Garage admin token (from garage.toml under [admin])

Step 1: Monitor the S3 API Endpoint

Garage's S3 API is the primary interface for all clients — backup tools (restic, rclone), application SDKs, and CLI tools. The S3 protocol's root path returns an XML response for unsigned requests, which is enough to confirm the API process is alive.

Test it:

curl -I https://s3.yourdomain.com/

Without authentication, you'll receive either a 403 Forbidden (access denied, but S3 API is responding) or a 200 OK with an empty ListAllMyBucketsResult. Both confirm the Garage S3 API is up and handling requests. A 502 or connection timeout means the API process is unreachable.

Set up the Vigilmon monitor:

1. Log in to Vigilmon and click New Monitor → HTTP.
2. Set URL to https://s3.yourdomain.com/.
3. Set Expected status codes to 200, 403.
4. Set Check interval to 60 seconds.
5. Save the monitor.

Step 2: Monitor the Admin API

Garage exposes an admin REST API (default port 3903) for cluster management, health queries, and layout operations. The /v1/health admin endpoint reports overall cluster health, including node connectivity and layout status.

Test it:

curl -H "Authorization: Bearer YOUR_ADMIN_TOKEN" \
  https://garage-admin.yourdomain.com/v1/health

Healthy response:

{
  "status": "healthy",
  "knownNodes": 3,
  "goodNodes": 3,
  "storage": {
    "ok": true,
    "storageClass": "default"
  }
}

The status field should be "healthy". A "degraded" status means one or more nodes are unreachable or the cluster layout has not converged. A "dangerous" status indicates write durability is below the minimum replication factor.

Set up the monitor:

1. Click New Monitor → HTTP in Vigilmon.
2. Set URL to https://garage-admin.yourdomain.com/v1/health.
3. Set HTTP Headers, add: Authorization: Bearer YOUR_ADMIN_TOKEN.
4. Set Expected status code to 200.
5. Under Advanced → Keyword check, add:
   • Keyword present: "healthy"
   • Keyword absent: "dangerous"
6. Set Check interval to 60 seconds.
7. Save.

If the admin API is not exposed externally, configure this check from within your private network using Vigilmon's internal monitoring agent, or expose the admin API on a private IP with TLS.

Step 3: Cluster Node Health Monitoring

Each Garage node runs its own HTTP API process. For a three-node cluster, you can monitor each node's health individually. This gives you per-node failure visibility that the cluster-level admin health check doesn't provide in isolation — you'll know which node went offline, not just that the cluster is degraded.

For each node, check its node-local health endpoint:

curl http://node1.internal:3902/health

Garage's node-local health endpoint returns a simple response confirming the local process is alive.

Set up a Vigilmon monitor for each node:

1. Click New Monitor → HTTP in Vigilmon.
2. Set URL to the node's health endpoint (use internal hostnames if monitoring from within your network).
3. Set Expected status code to 200.
4. Set Name to something descriptive like Garage Node 1 Health.
5. Set Check interval to 60 seconds.
6. Save.

Repeat for each cluster node. When a node monitor fails while the cluster admin health shows "degraded", you've immediately identified the failing node.

Step 4: Bucket Endpoint Response Check

Beyond the S3 API being reachable, it's worth verifying that your most critical bucket is accessible end-to-end. This catches cases where the API process is running but bucket routing, metadata, or storage is broken for a specific bucket.

With a publicly readable bucket, test it directly:

curl -o /dev/null -s -w "%{http_code}" \
  "https://your-bucket.s3.yourdomain.com/"

With path-style addressing:

curl -o /dev/null -s -w "%{http_code}" \
  "https://s3.yourdomain.com/your-bucket/"

You should get 200 OK (public bucket listing) or 403 Forbidden (private bucket — but the bucket is reachable). A 404 NoSuchBucket indicates the bucket doesn't exist or metadata is corrupted. A 502 means the storage backend is unreachable for this bucket.

Set up the monitor:

1. Click New Monitor → HTTP in Vigilmon.
2. Set URL to your bucket endpoint.
3. Set Expected status codes to 200, 403.
4. Set Check interval to 5 minutes.
5. Save.

Step 5: SSL Certificate Monitoring

All S3 clients connecting over HTTPS will reject an expired certificate immediately. SDK-based clients (AWS SDK, boto3, rclone) will throw TLS errors and fail their operations. Backup jobs will start failing at the TLS handshake before they even attempt to write data.

Vigilmon monitors SSL certificate validity automatically on all HTTPS monitors.

1. Open your S3 API endpoint monitor in Vigilmon.
2. Under Advanced → SSL, ensure Alert before expiry is enabled.
3. Set the alert lead time to 14 days.
4. Save.

Enable SSL alerts on your bucket endpoint monitor as well, since wildcard certificates for S3 virtual-hosted-style addressing have their own expiry to track.

Step 6: Alert Channels

Go to Notifications → New Channel in Vigilmon and configure:

• Email — for your storage operations team
• Webhook — for your infrastructure channel in Slack or PagerDuty

A cluster degraded alert looks like:

🔴 DOWN: Garage Admin API Health (HTTP)
Keyword "dangerous" found in response body
Response: {"status":"dangerous","knownNodes":3,"goodNodes":1}
Triggered: 2026-07-03 14:22 UTC

Because Garage is often used for backups and cold storage, failures may not cause immediate user-visible impact — but they will cause silent backup failures. Configure alerts to fire immediately rather than after a grace period.

What You've Built

| Scenario | How Vigilmon catches it | |---|---| | S3 API process crash | S3 endpoint monitor fails (no 200/403) | | Cluster node offline | Per-node health monitor fails; admin shows "degraded" | | Write durability below minimum | Admin health monitor detects "dangerous" | | Bucket routing broken | Bucket endpoint monitor returns 404/502 | | SSL certificate expired | HTTPS monitors report TLS error | | Admin API unreachable | Admin API monitor fails | | Network partition between nodes | Admin health shows "degraded" or "dangerous" |

Garage's distributed design tolerates node failures — but silent tolerance is not the same as healthy operation. A degraded cluster that nobody knows about is one more hardware failure away from data unavailability. Vigilmon's external monitoring ensures that every change in Garage's health status reaches your team immediately.

Start monitoring your Garage cluster today — register free at vigilmon.online.