Iceshrimp Monitoring with Vigilmon: Web Server, PostgreSQL, Redis, BullMQ & Federation Health

Your Iceshrimp instance had been running smoothly for a month. Users posted, boosted, and interacted with the rest of the Fediverse without complaint. Nobody noticed when the BullMQ job workers quietly stalled after a Redis memory spike — federation deliveries to remote servers backed up silently while the web interface continued to load fine. By the time someone reported that their followers on Mastodon hadn't seen a post in 36 hours, the queue had 12,000 unprocessed jobs.

Iceshrimp is an open source fork of Misskey and Calckey, built with Node.js and TypeScript. It implements ActivityPub federation with an emphasis on simplicity and community moderation tooling. Keeping Iceshrimp healthy means keeping PostgreSQL, Redis, Meilisearch or Elasticsearch, BullMQ workers, S3-compatible media storage, WebSocket push streams, and email delivery all alive simultaneously. Vigilmon gives you external uptime monitoring and alerting to catch failures before your users do.

What You'll Set Up

• HTTP uptime monitor for the Iceshrimp web interface
• PostgreSQL database connectivity check
• Redis cache health endpoint monitor
• Meilisearch or Elasticsearch search index availability check
• Heartbeat monitor for BullMQ background job workers
• ActivityPub federation inbox and outbox endpoint checks
• S3/object storage media service availability check
• WebSocket real-time push event stream health monitor
• Email notification delivery heartbeat
• Federation relay connectivity check
• SSL certificate expiry alerts

Prerequisites

• A running Iceshrimp instance (recent release) on port 3000 behind Nginx or Caddy
• PostgreSQL, Redis, and optionally Meilisearch running and accessible
• A free Vigilmon account

Step 1: Monitor the Iceshrimp Web Interface

The Iceshrimp homepage is the primary health signal for the full stack. A 200 OK response confirms that Node.js is serving requests, the reverse proxy is routing correctly, and the database is answering basic queries for the timeline render.

Test it:

curl -I https://iceshrimp.yourdomain.com/

You should see 200 OK with Content-Type: text/html.

Set up the Vigilmon monitor:

1. Log in to Vigilmon and click New Monitor → HTTP.
2. Set URL to https://iceshrimp.yourdomain.com/.
3. Set Check interval to 60 seconds.
4. Set Expected status code to 200.
5. Under Advanced → Keyword check, add keyword present: Iceshrimp.
6. Save the monitor.

Step 2: Monitor PostgreSQL Database Connectivity

Iceshrimp stores all user accounts, notes, relationships, ActivityPub actor records, and federation data in PostgreSQL via TypeORM. A database connectivity failure causes immediate application errors on every authenticated request.

Iceshrimp exposes a built-in health check endpoint at /healthcheck that tests database connectivity as part of its startup probe logic. Test it:

curl https://iceshrimp.yourdomain.com/healthcheck

If your Iceshrimp version doesn't expose /healthcheck, add a thin proxy health route in Nginx that checks the app's /api/meta endpoint — a successful response indicates the database is serving data:

curl https://iceshrimp.yourdomain.com/api/meta | jq '.version'

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://iceshrimp.yourdomain.com/api/meta.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: version.
5. Save.

When this monitor fails while the homepage monitor is healthy, the problem is almost always database connectivity or a TypeORM pool exhaustion issue.

Step 3: Monitor Redis Cache Health

Iceshrimp uses Redis as a job queue backend (for BullMQ), session store, and caching layer. Redis downtime causes BullMQ workers to lose their queue connections, user sessions to fail, and rate limiting to stop working. Performance degrades immediately.

Add a health check endpoint to your Iceshrimp configuration or expose a thin route through Nginx. Alternatively, check the Iceshrimp API metadata endpoint — it internally depends on Redis being responsive for certain operations:

curl https://iceshrimp.yourdomain.com/api/v1/instance

For a more targeted Redis health check, create a minimal Express middleware or a sidecar health endpoint that issues a Redis PING:

// health/redis.ts — mount at /health/redis
import { createClient } from 'redis';
const client = createClient({ url: process.env.REDIS_URL });
app.get('/health/redis', async (_req, res) => {
  try {
    await client.ping();
    res.json({ status: 'ok' });
  } catch {
    res.status(503).json({ status: 'error' });
  }
});

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://iceshrimp.yourdomain.com/health/redis.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: ok.
5. Save.

Step 4: Monitor Meilisearch or Elasticsearch Search Index

Iceshrimp supports Meilisearch (default) or Elasticsearch for full-text note search. Search index downtime doesn't cause a complete outage — users can still post and federate — but the search feature fails silently, frustrating users who rely on note search.

Meilisearch exposes a health endpoint at port 7700:

curl http://localhost:7700/health
# {"status":"available"}

Expose this through a reverse proxy subpath or a sidecar health endpoint:

location /health/search {
    proxy_pass http://127.0.0.1:7700/health;
    allow 127.0.0.1;
    # allow your Vigilmon check IPs or restrict and use a token
}

For Elasticsearch:

curl http://localhost:9200/_cluster/health?pretty
# "status": "green"

In Vigilmon (for Meilisearch):

1. Click New Monitor → HTTP.
2. Set URL to https://iceshrimp.yourdomain.com/health/search.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: available.
5. Save.

Step 5: Heartbeat Monitor for BullMQ Job Workers

BullMQ workers process all of Iceshrimp's async work: outgoing ActivityPub delivery, federation inbox processing, media processing, email dispatch, and cleanup jobs. When workers stall — due to a Redis disconnect, memory limit, or unhandled exception — federation grinds to a halt. Unlike web request failures, worker crashes are completely invisible without explicit monitoring.

Step 5a: Create a Vigilmon heartbeat monitor.

1. Click New Monitor → Heartbeat in Vigilmon.
2. Set Name to Iceshrimp BullMQ Workers.
3. Set Expected heartbeat interval to 5 minutes.
4. Save. Copy the Heartbeat URL shown (e.g., https://vigilmon.online/heartbeat/abc123).

Step 5b: Configure the worker to ping Vigilmon.

Add a recurring Bull job that pings the heartbeat URL:

// workers/heartbeat.ts
import Queue from 'bull';
const heartbeatQueue = new Queue('heartbeat', process.env.REDIS_URL);

heartbeatQueue.process(async () => {
  await fetch('https://vigilmon.online/heartbeat/YOUR_TOKEN');
});

heartbeatQueue.add({}, { repeat: { cron: '*/5 * * * *' } });

Or use a simpler system cron approach:

# /etc/cron.d/iceshrimp-worker-heartbeat
*/5 * * * * deploy pgrep -f "iceshrimp.*worker" && \
  curl -fsS https://vigilmon.online/heartbeat/YOUR_TOKEN

If BullMQ workers have stopped, the pgrep check returns non-zero, the curl is skipped, and Vigilmon alerts you after the expected interval passes without a ping.

Step 6: Monitor ActivityPub Federation Endpoints

Iceshrimp's ActivityPub implementation exposes inbox and outbox endpoints for each user actor. These are the entry points for all federated traffic — remote servers delivering posts, boosts, and interactions to your instance.

Check the WebFinger discovery endpoint first — other Fediverse servers use it to look up your users:

curl "https://iceshrimp.yourdomain.com/.well-known/webfinger?resource=acct:admin@iceshrimp.yourdomain.com"

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://iceshrimp.yourdomain.com/.well-known/webfinger?resource=acct:admin@iceshrimp.yourdomain.com.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: subject.
5. Save.

Check the NodeInfo endpoint for federation capability advertisement:

1. Click New Monitor → HTTP.
2. Set URL to https://iceshrimp.yourdomain.com/.well-known/nodeinfo.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: links.
5. Save.

Step 7: Monitor S3/Object Storage Media Service

Iceshrimp stores uploaded media (images, videos, attachments) in S3-compatible object storage — either AWS S3, Wasabi, Backblaze B2, or a local MinIO instance. A storage backend failure causes media uploads to fail and existing media to return 404 errors on posts.

Check your media proxy endpoint, which serves stored media through Iceshrimp:

curl -I https://iceshrimp.yourdomain.com/files/YOUR_EXISTING_ATTACHMENT_PATH

Or check the object storage bucket directly if it's publicly accessible:

curl -I https://your-bucket.s3.yourdomain.com/

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://iceshrimp.yourdomain.com/files/ (or a known static asset URL).
3. Set Expected status code to 200 or 403 (bucket root may return 403 while being healthy).
4. Save.

If you're using a MinIO instance, monitor its health endpoint:

curl http://localhost:9000/minio/health/live
# Returns 200 OK when healthy

Step 8: Monitor WebSocket Real-Time Push Event Stream

Iceshrimp uses WebSocket connections to push real-time timeline updates, notifications, and streaming events to connected clients. When the WebSocket server fails, users see a degraded experience — the timeline doesn't update live and notifications don't appear without a page refresh.

Test WebSocket connectivity with a simple HTTP upgrade check:

curl -I -H "Upgrade: websocket" -H "Connection: Upgrade" \
  https://iceshrimp.yourdomain.com/streaming

You should see a 101 Switching Protocols response or a 400 Bad Request (which still confirms the endpoint is alive and reached the WebSocket handler).

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://iceshrimp.yourdomain.com/streaming.
3. Set Expected status code to 101 or 400.
4. Save.

Since WebSocket failures degrade user experience but don't completely break the platform, configure this monitor with a slightly higher alert threshold.

Step 9: Monitor Email Notification Delivery

Iceshrimp sends email notifications for mentions, direct messages, and account-related events. Email delivery failures are silent from the user's perspective — they simply stop receiving notifications.

The most reliable approach is a Vigilmon heartbeat monitor triggered by a scheduled test email delivery. Create a cron job that attempts to send a test email through your configured SMTP relay and pings Vigilmon only on success:

# /etc/cron.d/iceshrimp-email-heartbeat
0 */6 * * * deploy node /opt/iceshrimp/scripts/test-email.js && \
  curl -fsS https://vigilmon.online/heartbeat/YOUR_EMAIL_TOKEN

Where test-email.js attempts an SMTP connection and test delivery:

import nodemailer from 'nodemailer';
const transporter = nodemailer.createTransport({ /* your SMTP config */ });
await transporter.verify();
await fetch('https://vigilmon.online/heartbeat/YOUR_EMAIL_TOKEN');

Create the heartbeat monitor first:

1. Click New Monitor → Heartbeat in Vigilmon.
2. Set Name to Iceshrimp Email Delivery.
3. Set Expected heartbeat interval to 6 hours.
4. Save. Use the provided URL in the script above.

Step 10: Monitor Federation Relay Health

If your Iceshrimp instance subscribes to a federation relay to receive a broader stream of public posts, relay connectivity affects the quality of your public timeline. A relay outage is non-critical but worth tracking.

Check your relay's health or subscription endpoint:

curl https://your-relay.yourdomain.com/inbox
# Should return 405 Method Not Allowed — confirms the endpoint exists

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://your-relay.yourdomain.com/inbox.
3. Set Expected status code to 200, 405, or 401.
4. Save as a low-priority warning monitor.

Step 11: SSL Certificate Expiry Alerts

Iceshrimp requires HTTPS for ActivityPub federation with virtually all other servers. A lapsed certificate causes federated servers to reject deliveries from your instance.

1. Open the web UI monitor from Step 1.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Save.

Alerting Configuration

Set up notification channels so you're alerted immediately on failures:

1. In Vigilmon, go to Alerts → Notification Channels.
2. Add an Email channel with your admin address.
3. Optionally add a Slack webhook for your ops channel.
4. Assign all Iceshrimp monitors to these notification channels.

Recommended alert thresholds:

| Monitor | Alert after | |---|---| | Web UI | 2 consecutive failures (2 min) | | PostgreSQL health | 1 failure (immediate) | | Redis health | 2 consecutive failures | | Meilisearch/Elasticsearch | 3 consecutive failures | | BullMQ worker heartbeat | 1 missed heartbeat (5 min) | | WebFinger/NodeInfo | 3 consecutive failures | | S3/Object storage | 3 consecutive failures | | WebSocket streaming | 5 consecutive failures | | Email delivery heartbeat | 1 missed heartbeat (6 hr) | | Federation relay | 5 consecutive failures | | SSL certificate | 21 days before expiry |

PostgreSQL and the BullMQ worker heartbeat are highest priority — a database failure causes immediate errors and a stalled BullMQ worker stops all federation silently. WebSocket and relay monitors can have more tolerance since they affect enhanced features rather than core functionality.

Conclusion

A healthy Iceshrimp instance is more than just an accessible homepage. Federation depends on BullMQ workers staying alive and Redis staying responsive. Note search depends on Meilisearch or Elasticsearch remaining indexed. Media posts depend on S3 object storage staying available. Without external monitoring, any of these can fail silently for hours.

With Vigilmon watching your Iceshrimp web interface, PostgreSQL connectivity, Redis cache, search index, BullMQ workers, ActivityPub federation endpoints, S3 media storage, WebSocket streaming, email delivery, and federation relay health, you'll know about failures within minutes rather than learning from your users.

Start monitoring your Iceshrimp instance for free at vigilmon.online.