tutorial

Monitoring SeaweedFS with Vigilmon: Uptime, Cluster Health & Write Verification for Self-Hosted Distributed Storage

How to monitor a self-hosted SeaweedFS distributed storage cluster with Vigilmon — covering master server availability, volume server health, the cluster status API, S3 gateway checks, and write operation heartbeats.

SeaweedFS is a fast distributed object storage system designed to handle billions of small files with high throughput and low latency. Teams running SeaweedFS self-hosted get impressive storage capabilities at low cost — but a distributed system has more failure modes than a single-process app. A master server going offline, a volume server losing connectivity, or a full disk on one node can cascade into write failures that are hard to detect from the inside. Vigilmon gives you external uptime monitoring that watches your SeaweedFS cluster from outside and alerts you when any layer breaks — before your application starts returning storage errors to users.

What You'll Build

  • A Vigilmon HTTP monitor on the SeaweedFS master server availability
  • A cluster status API health check with keyword assertion
  • Volume server availability monitors
  • A Vigilmon heartbeat to confirm write operations succeed end-to-end
  • SSL certificate monitoring (if the S3 gateway is proxied with HTTPS)

Prerequisites

  • A running SeaweedFS cluster (master + at least one volume server) accessible over the network
  • A free account at vigilmon.online

Step 1: Verify the Master Server and Cluster Status API

The SeaweedFS master server runs on port :9333 by default. It exposes a /cluster/status API endpoint that reports the cluster's health — master leader election, connected volume servers, and free capacity.

curl http://your-master-host:9333/cluster/status

A healthy cluster returns JSON similar to:

{
  "IsLeader": true,
  "Leader": "master1:9333",
  "Peers": ["master2:9333"],
  "Volume": {
    "Free": 150,
    "Max": 200
  }
}

If the master process has crashed or lost quorum, this endpoint will not respond. If it responds but IsLeader is false on all masters, your cluster has lost its leader election.

Exposing the API securely: The master API is typically on an internal network. If you expose it publicly, put it behind an authenticated reverse proxy. For Vigilmon to reach it, either expose it on a public IP or use a lightweight tunnel.

Step 2: Create an HTTP Monitor for the Master Server

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://your-master-host:9333/cluster/status (or HTTPS if proxied).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Click Save.

Vigilmon will probe your SeaweedFS master server every 60 seconds from an external location. If the master process crashes or the port becomes unreachable, you'll know within 60–120 seconds.

Step 3: Add Keyword Assertions for Cluster Health

A 200 status confirms the master is responding, but not that the cluster is in a healthy state. Add keyword assertions:

Assertion 1 — Leader election healthy:

  • Keyword: IsLeader":true
  • Keyword must be present: yes

This fails the check if the master responds but has lost leader election — a sign of split-brain or quorum loss in a multi-master setup.

Assertion 2 — Volume servers connected:

  • Keyword: Volume
  • Keyword must be present: yes

This confirms the response includes volume server data, not just a partial or error response body.

Step 4: Monitor Individual Volume Servers

Volume servers run on port :8080 by default and expose a status endpoint. Each volume server should be monitored independently because a single failed volume reduces your cluster's write capacity.

For each volume server, add an HTTP monitor:

  1. Add Monitor → HTTP.
  2. URL: http://your-volume-host:8080/status.
  3. Check interval: 60 seconds.
  4. Expected status: 200.
  5. Name it descriptively (e.g., SeaweedFS Volume Server 1).
# Verify the volume status endpoint first
curl http://your-volume-host:8080/status

A healthy volume server returns JSON with disk usage and volume counts.

Step 5: Heartbeat Monitor for Write Operation Verification

The most reliable way to know your SeaweedFS cluster is working end-to-end is to perform a real write and verify it succeeds. A cluster can have all its HTTP endpoints returning 200 while being in a read-only state (e.g., all volumes full, or a permissions issue).

Create a Heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat.
  2. Name: SeaweedFS write heartbeat.
  3. Expected ping interval: 5 minutes.
  4. Grace period: 3 minutes.
  5. Copy the generated heartbeat URL.

Add a cron job that writes a test file, reads it back, deletes it, and pings the heartbeat only on success:

# /etc/cron.d/seaweedfs-write-heartbeat
*/5 * * * * root \
  ASSIGN=$(curl -sf "http://localhost:9333/dir/assign") && \
  FID=$(echo "$ASSIGN" | grep -oP '"fid":"\K[^"]+') && \
  URL=$(echo "$ASSIGN" | grep -oP '"url":"\K[^"]+') && \
  curl -sf -F "file=@/etc/hostname" "http://$URL/$FID" > /dev/null && \
  curl -sf "http://localhost:9333/dir/lookup?volumeId=${FID%%,*}" > /dev/null && \
  curl -sf -X DELETE "http://$URL/$FID" > /dev/null && \
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-HEARTBEAT-ID

This script:

  1. Asks the master to assign a file ID and volume URL
  2. Uploads /etc/hostname (a tiny, always-present file) to the assigned volume
  3. Looks up the file to confirm it's registered with the master
  4. Deletes the test file to keep the cluster clean
  5. Pings the Vigilmon heartbeat only if all steps succeed

If any step fails — master unreachable, volume full, network partition — the heartbeat stops and Vigilmon alerts you within 8 minutes.

Step 6: Monitor the S3 API Gateway (If Enabled)

SeaweedFS ships an S3-compatible API gateway that applications can use to store and retrieve objects. If your applications use the S3 gateway, monitor it separately:

  1. Add Monitor → HTTP.
  2. URL: https://your-s3-gateway-domain.com/ (or http://your-host:8333/).
  3. Expected status: 200 or 403 (S3 returns 403 on unauthenticated root requests, which still confirms the gateway is alive).
  4. Name: SeaweedFS S3 Gateway.

If you expect a 403 on the root path, set Expected status to 403 rather than 200 to avoid false alerts.

Step 7: SSL Certificate Monitoring

If the S3 gateway or master API is behind an HTTPS reverse proxy:

  1. Add Monitor → SSL Certificate.
  2. Domain: your-s3-gateway-domain.com.
  3. Alert when: certificate expires within 14 days.

An expired certificate on your S3 gateway will cause all applications using the S3 SDK to throw TLS errors — silent storage failures that are hard to trace without this monitor.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications:

| Trigger | Channel | Action | |---|---|---| | Master /cluster/status non-200 | Email + Slack | Check master: systemctl status weed-master | | IsLeader":true keyword missing | Email + Slack | Investigate quorum: check Raft logs | | Volume server /status non-200 | Email | Check volume: systemctl status weed-volume | | Write heartbeat misses | Email + Slack | Run manual assign/upload test; check disk space | | S3 gateway non-200/403 | Email | Check S3 process: systemctl status weed-s3 | | SSL expires within 14 days | Email | Renew certificate |

Alert after: 1 consecutive failure for write heartbeats and leader election. Use 2 consecutive failures for volume server checks to allow brief network hiccups.

Running SeaweedFS with systemd

# /etc/systemd/system/weed-master.service
[Unit]
Description=SeaweedFS Master
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/weed master -port=9333 -mdir=/data/seaweedfs/master
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/weed-volume.service
[Unit]
Description=SeaweedFS Volume Server
After=network.target weed-master.service

[Service]
Type=simple
ExecStart=/usr/local/bin/weed volume -port=8080 -dir=/data/seaweedfs/volumes -mserver=localhost:9333
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

What Vigilmon Catches That Cluster Logs Miss

| Scenario | Internal monitoring | Vigilmon | |---|---|---| | Master process crash | Peers may detect eventually | HTTP monitor fires within 60–120 s | | Leader election failure (split-brain) | May not self-report clearly | Keyword assertion IsLeader fails | | Volume server disk full | Error on next write | Write heartbeat stops immediately | | S3 gateway TLS certificate expired | Not logged until connection fails | SSL monitor alerts 14 days early | | Network partition isolates cluster | Unreachable from outside | External Vigilmon detects from outside |

SeaweedFS's distributed architecture is its strength for scale, but that same distribution means more components to monitor. Vigilmon gives you a single external view across master servers, volume nodes, and the S3 gateway — with write heartbeats that verify the whole pipeline, not just that processes are running.

Start monitoring your SeaweedFS cluster in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles