Firefish Monitoring with Vigilmon: Web Server, BullMQ Workers, Federation & Search Index Health

A Firefish instance operator ran a popular community for six months without a single major outage. Then one morning, users reported that search was broken and notifications had stopped delivering. The operator checked the web interface — everything looked fine. Logs revealed that the BullMQ worker processes had been stuck in a retry loop for eight hours after a Redis memory spike, and Meilisearch had quietly crashed and stayed down. The web interface never showed an error because Firefish degrades gracefully on non-critical service failures.

Firefish is an open-source ActivityPub microblogging platform forked from Calckey (itself a fork of Misskey), built with Node.js and TypeScript. It brings rich features including emoji reactions, drive-based media management, and highly customizable theming to the Fediverse. Under the hood, it relies on PostgreSQL, Redis, BullMQ job queues, an optional search backend (Meilisearch or Elasticsearch), and S3-compatible object storage. Vigilmon gives you the external monitoring to catch failures across all of these layers before your users do.

What You'll Set Up

• HTTP uptime monitor for the Firefish web interface
• API health check on the instance metadata endpoint
• PostgreSQL database connectivity check
• Redis cache health monitor
• BullMQ job queue worker heartbeat monitor
• Meilisearch or Elasticsearch search index availability check
• ActivityPub federation inbox/outbox endpoint monitors
• S3 object storage media service check
• WebSocket push event stream health monitor
• SMTP email delivery health check
• SSL certificate expiry alerts

Prerequisites

• A running Firefish instance on port 3000 or behind a reverse proxy
• Redis running on its default port (6379)
• A free Vigilmon account

Step 1: Monitor the Firefish Web Interface

The Firefish web application is a single-page app served by the Node.js backend. A healthy response confirms that the Node.js process is running, the reverse proxy is forwarding correctly, and the application is serving traffic.

Test it:

curl -I https://firefish.yourdomain.com/

You should see 200 OK with Content-Type: text/html.

Set up the Vigilmon monitor:

1. Log in to Vigilmon and click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/.
3. Set Check interval to 60 seconds.
4. Set Expected status code to 200.
5. Save.

Step 2: Monitor the Instance API Metadata Endpoint

Firefish exposes an API endpoint at /api/meta that returns instance configuration, software version, features enabled, and admin contact. This is the canonical health endpoint used by the Firefish web client itself, federation peers, and API consumers. A failure here that isn't caught by the basic web UI monitor typically indicates an application crash or database connectivity loss.

The /api/meta endpoint accepts a POST with an empty JSON body:

curl -X POST -H "Content-Type: application/json" -d '{}' https://firefish.yourdomain.com/api/meta

Healthy response includes "version", "name", and feature flags.

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/api/meta.
3. Set Method to POST.
4. Set Request body to {}.
5. Set Request header Content-Type to application/json.
6. Set Expected status code to 200.
7. Under Keyword check, add keyword present: "version".
8. Save.

Step 3: Monitor PostgreSQL Database Connectivity

PostgreSQL stores all Firefish data: users, notes, reactions, follows, ActivityPub objects, drive files, and federation records. A database outage causes an immediate and complete application failure.

Firefish has a built-in health check endpoint:

curl https://firefish.yourdomain.com/healthcheck

If available, it returns a simple health status. If not exposed, the /api/meta POST monitor (Step 2) doubles as an effective database connectivity check, since the metadata is read from PostgreSQL on every request.

For an explicit database check, add a custom endpoint if you've customized your Firefish deployment:

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/healthcheck.
3. Set Expected status code to 200.
4. Save.

If /healthcheck isn't exposed, rely on the /api/meta monitor from Step 2 as your database connectivity signal — it will fail with a 500 if PostgreSQL is unreachable.

Step 4: Monitor Redis Cache Health

Redis serves multiple roles in Firefish: session storage, rate limiting, pub/sub for real-time events, and as the backing store for BullMQ job queues. A Redis failure causes BullMQ to stop accepting and dispatching jobs immediately, breaking all async federation and notification delivery. Real-time features (live notifications, streaming timelines) also stop working.

Firefish's admin API exposes a server stats endpoint that includes Redis connection status. Check it with an admin token:

1. Log in to your Firefish instance as an admin.
2. Go to Settings → API and generate an API key.
3. Test:

curl -X POST -H "Content-Type: application/json" \
  -d '{"i": "YOUR_API_KEY"}' \
  https://firefish.yourdomain.com/api/admin/server-info

The response includes Redis version and memory usage, confirming connectivity.

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/api/admin/server-info.
3. Set Method to POST.
4. Set Request body to {"i": "YOUR_ADMIN_API_KEY"}.
5. Set Request header Content-Type to application/json.
6. Set Expected status code to 200.
7. Under Keyword check, add keyword present: "redisVersion".
8. Save.

Step 5: Monitor BullMQ Job Queue Workers

Firefish uses BullMQ (a Redis-backed job queue) for all background processing: ActivityPub federation deliveries, incoming federation processing, media transcoding, notification dispatch, chart updates, and scheduled maintenance. BullMQ workers run as part of the main Firefish Node.js process in most deployments.

Worker health can be monitored through the admin queue stats endpoint:

curl -X POST -H "Content-Type: application/json" \
  -d '{"i": "YOUR_API_KEY"}' \
  https://firefish.yourdomain.com/api/admin/queue/stats

This returns queue depths and worker counts for all BullMQ queues. A waiting count that keeps growing without a corresponding active count indicates workers have stopped.

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/api/admin/queue/stats.
3. Set Method to POST.
4. Set Request body to {"i": "YOUR_ADMIN_API_KEY"}.
5. Set Request header Content-Type to application/json.
6. Set Expected status code to 200.
7. Under Keyword check, add keyword present: "deliver" (the federation delivery queue).
8. Save.

Heartbeat for worker liveness:

For deeper monitoring, set up a Vigilmon heartbeat that's pinged by a periodic Firefish job. Create a heartbeat monitor in Vigilmon, then add a custom BullMQ job to Firefish's job scheduler that pings it every 5 minutes:

1. Click New Monitor → Heartbeat in Vigilmon.
2. Set Name to Firefish BullMQ Workers.
3. Set Expected heartbeat interval to 5 minutes.
4. Save. Copy the heartbeat URL.

Then add to Firefish's scheduler (edit packages/backend/src/queue/index.ts):

// In the job queue setup, add a recurring heartbeat job
const heartbeatWorker = new Worker('heartbeat', async () => {
  await fetch('https://vigilmon.online/heartbeat/YOUR_TOKEN');
}, { connection: redisConnection });

const heartbeatQueue = new Queue('heartbeat', { connection: redisConnection });
await heartbeatQueue.add('ping', {}, { repeat: { every: 300000 } }); // 5 min

If the BullMQ workers stop processing, this periodic job won't run and the heartbeat ping won't reach Vigilmon.

Step 6: Monitor Meilisearch or Elasticsearch Search Index

Firefish's full-text search feature uses either Meilisearch or Elasticsearch as a backend search index. When the search backend is unavailable, search returns no results — a visible but non-critical failure. However, if the index diverges significantly from the database (missed writes during an outage), search quality degrades permanently until re-indexed.

For Meilisearch:

curl https://meilisearch.yourdomain.com/health
# Returns: {"status": "available"}

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://meilisearch.yourdomain.com/health (or http://localhost:7700/health if internal-only, proxied through a secure route).
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: available.
5. Save.

For Elasticsearch:

curl http://localhost:9200/_cluster/health
# Returns: {"status": "green", ...}

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to your Elasticsearch health endpoint (proxied securely).
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: "status", keyword absent: "red".
5. Save.

Step 7: Monitor ActivityPub Federation Endpoints

Firefish federates with Mastodon, Misskey, and all other ActivityPub servers. The WebFinger endpoint is how remote servers discover your users; the inbox is how they deliver activities to you.

WebFinger check:

curl "https://firefish.yourdomain.com/.well-known/webfinger?resource=acct:admin@firefish.yourdomain.com"

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/.well-known/webfinger?resource=acct:admin@firefish.yourdomain.com.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: subject.
5. Save.

NodeInfo check (used by federation peers for capability discovery):

1. Click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/.well-known/nodeinfo.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: links.
5. Save.

Shared inbox check (should return 405 for GET):

1. Click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/inbox.
3. Set Expected status code to 405.
4. Save.

Step 8: Monitor S3 Object Storage Media Service

Firefish stores all user-uploaded media (images, videos, audio) and cached remote media in an S3-compatible object store. When S3 is unavailable, media uploads fail with errors, existing media returns 403 or 404, and remote media caching stops working.

Upload a small test file to your S3 bucket and monitor its URL:

echo "ok" | aws s3 cp - s3://your-firefish-bucket/health-check.txt --acl public-read

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://your-bucket.s3.amazonaws.com/health-check.txt (or your S3-compatible endpoint URL).
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: ok.
5. Save.

For Wasabi, Backblaze B2, or MinIO, adjust the URL to match your provider's endpoint format.

Step 9: Monitor WebSocket Real-Time Push Event Stream

Firefish's streaming API uses WebSocket connections to deliver real-time events — new notes, notifications, and timeline updates — to connected clients. The WebSocket endpoint is at /streaming.

While Vigilmon doesn't natively monitor WebSocket connections, you can verify that the WebSocket upgrade endpoint is reachable via HTTP:

curl -I https://firefish.yourdomain.com/streaming

A healthy Firefish instance returns 101 Switching Protocols for a WebSocket upgrade request, or 400 Bad Request if the request is missing WebSocket headers (both indicate the endpoint is alive).

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://firefish.yourdomain.com/streaming.
3. Set Expected status code to 400 (or 200 depending on your Firefish version — test first).
4. Save.

If the streaming endpoint returns a 502 or 504, the WebSocket proxy configuration is broken and real-time features are down for all users.

Step 10: Monitor SMTP Email Delivery

Firefish sends transactional emails for account registration, password resets, and email change confirmations. SMTP delivery failures are silent from the user perspective — the user simply never receives the email and may think the registration failed.

Use a periodic outgoing test email plus an email-to-heartbeat service to confirm delivery:

1. Set up a Vigilmon heartbeat monitor (New Monitor → Heartbeat, 24-hour interval).
2. Configure an email parsing service that triggers the heartbeat URL when it receives a test email.
3. Schedule a daily job on your Firefish server to send a test email to that parsing address:

# /etc/cron.d/firefish-email-health
0 9 * * * root node /opt/firefish/scripts/send-test-email.js

Alternatively, set up a tool like healthchecks.io-style SMTP test by monitoring your mail queue directly on the server:

# Check if mail queue is empty (no stuck messages)
*/30 * * * * root [ $(mailq | wc -l) -lt 5 ] && curl -fsS https://vigilmon.online/heartbeat/YOUR_TOKEN

Step 11: SSL Certificate Expiry Alerts

Firefish requires HTTPS for ActivityPub federation. An expired certificate causes federation to fail immediately across all connected servers.

1. Open the web interface monitor from Step 1.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Save.

Alerting Configuration

Configure notification channels in Vigilmon for immediate alerting:

1. Go to Alerts → Notification Channels.
2. Add an Email channel with your admin address.
3. Add a Slack or Discord webhook for your community.
4. Assign all Firefish monitors to your notification channels.

Recommended alert thresholds:

| Monitor | Priority | Alert after | |---|---|---| | Web interface | Critical | 2 consecutive failures | | Instance API (/api/meta) | Critical | 2 consecutive failures | | PostgreSQL / healthcheck | Critical | 1 failure (immediate) | | Redis (server-info) | Critical | 1 failure | | BullMQ worker heartbeat | Critical | 1 missed heartbeat (5 min) | | WebFinger / NodeInfo | High | 3 consecutive failures | | Shared inbox (405 check) | High | 2 consecutive failures | | Meilisearch / Elasticsearch | Medium | 5 consecutive failures | | S3 media storage | High | 3 consecutive failures | | WebSocket endpoint | Medium | 5 consecutive failures | | SSL certificate | High | 21 days before expiry |

PostgreSQL, Redis, and BullMQ workers are the highest-priority monitors. A database or Redis failure causes immediate total outage; a BullMQ worker stoppage causes silent federation failure. Search index failures are medium priority — degraded but non-critical.

Conclusion

Firefish's rich feature set — reactions, drive, custom themes, full-text search — comes with a correspondingly complex infrastructure. The web interface is designed to degrade gracefully, which means serious failures in BullMQ workers, search indexes, or media storage can go unnoticed for hours.

With Vigilmon watching your web interface, PostgreSQL, Redis, BullMQ worker heartbeat, search index, ActivityPub endpoints, S3 media backend, WebSocket stream, and SSL certificate, you have a complete picture of your Firefish instance's health. Catch failures in minutes, not hours.

Start monitoring your Firefish instance for free at vigilmon.online.