PieFed Monitoring with Vigilmon: Web Server, Celery, Redis, PostgreSQL & Federation Health

Your PieFed instance had been running without visible issues for weeks. Posts appeared, communities grew, and users browsed content without complaints. What nobody noticed was that the Celery worker had silently exited three days ago. Incoming ActivityPub deliveries from federated Lemmy and Mastodon instances were queuing in Redis but never being processed. Federation sync had stopped cold — your communities had three days of posts from remote instances that would never arrive.

PieFed is an open source federated link aggregator and forum platform built with Python and Flask, compatible with ActivityPub and interoperable with Lemmy, Mastodon, and the broader Fediverse. Keeping PieFed healthy means keeping Flask, Celery workers, Redis, PostgreSQL, and search all running simultaneously. Vigilmon gives you external uptime monitoring and alerting to catch failures before your users notice.

What You'll Set Up

• HTTP uptime monitor for the PieFed web interface
• Heartbeat monitor for Celery async task workers
• Redis queue broker connectivity check
• PostgreSQL database connectivity check
• ActivityPub federation inbox and outbox endpoint checks
• Search index service health monitor
• Email notification delivery heartbeat
• Scheduled federation sync and cleanup job heartbeat
• Media storage backend availability check
• WebFinger discovery endpoint monitor
• SSL certificate expiry alerts

Prerequisites

• A running PieFed instance behind Nginx or Caddy on port 5000
• Celery worker and Redis running and accessible
• A free Vigilmon account

Step 1: Monitor the PieFed Web Interface

The PieFed homepage confirms that Flask is serving requests, Nginx is routing correctly, and PostgreSQL is returning community and post data.

Test it:

curl -I https://piefed.yourdomain.com/

You should see 200 OK with Content-Type: text/html.

Set up the Vigilmon monitor:

1. Log in to Vigilmon and click New Monitor → HTTP.
2. Set URL to https://piefed.yourdomain.com/.
3. Set Check interval to 60 seconds.
4. Set Expected status code to 200.
5. Under Advanced → Keyword check, add keyword present: PieFed.
6. Save the monitor.

Step 2: Heartbeat Monitor for Celery Async Task Workers

Celery workers are the engine behind PieFed's federation. All incoming ActivityPub delivery handling, outgoing post delivery to federated servers, user notification dispatch, and scheduled cleanup tasks run as Celery jobs. When Celery workers stop — due to an unhandled exception, memory limit, or Redis disconnect — federation stalls completely and the web interface continues to work normally, hiding the failure.

Step 2a: Create a Vigilmon heartbeat monitor.

1. Click New Monitor → Heartbeat in Vigilmon.
2. Set Name to PieFed Celery Workers.
3. Set Expected heartbeat interval to 5 minutes.
4. Save. Copy the Heartbeat URL shown (e.g., https://vigilmon.online/heartbeat/abc123).

Step 2b: Configure Celery to ping Vigilmon.

Add a periodic Celery beat task that pings the heartbeat URL:

# tasks/heartbeat.py
from celery import shared_task
import requests

@shared_task
def ping_vigilmon_heartbeat():
    requests.get('https://vigilmon.online/heartbeat/YOUR_TOKEN', timeout=10)

Register the periodic task in your Celery beat schedule:

# celery_config.py
beat_schedule = {
    'vigilmon-heartbeat': {
        'task': 'tasks.heartbeat.ping_vigilmon_heartbeat',
        'schedule': 300.0,  # every 5 minutes
    },
}

Or use a simpler system cron check:

# /etc/cron.d/piefed-celery-heartbeat
*/5 * * * * deploy pgrep -f "celery.*worker" && \
  curl -fsS https://vigilmon.online/heartbeat/YOUR_TOKEN

If the Celery worker has stopped, the pgrep check fails, the curl is skipped, and Vigilmon alerts you after the expected interval passes without a ping.

Step 3: Monitor Redis Queue Broker Connectivity

Redis serves as the Celery message broker and result backend for PieFed. If Redis becomes unavailable, Celery workers disconnect and can no longer pick up tasks — even if the workers themselves are still running. The web interface continues to serve cached content while federation silently stops.

Add a Redis health check endpoint to your PieFed Flask application:

# routes/health.py
from flask import jsonify
import redis

r = redis.from_url(current_app.config['REDIS_URL'])

@health_bp.route('/health/redis')
def redis_health():
    try:
        r.ping()
        return jsonify({'status': 'ok'})
    except Exception:
        return jsonify({'status': 'error'}), 503

Register this blueprint and ensure it's accessible (ideally restricted to known monitoring IPs in your firewall or Nginx config):

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://piefed.yourdomain.com/health/redis.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: ok.
5. Save.

Step 4: Monitor PostgreSQL Database Connectivity

PieFed stores all communities, posts, comments, user accounts, votes, federation records, and OAuth data in PostgreSQL via SQLAlchemy. A database connectivity failure causes application errors on virtually every page request.

Add a database health check endpoint:

# routes/health.py
from sqlalchemy import text

@health_bp.route('/health/db')
def db_health():
    try:
        db.session.execute(text('SELECT 1'))
        return jsonify({'status': 'ok'})
    except Exception:
        return jsonify({'status': 'error'}), 503

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://piefed.yourdomain.com/health/db.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: ok.
5. Save.

When this monitor fails while the homepage monitor passes, the issue is PostgreSQL connectivity or SQLAlchemy connection pool exhaustion.

Step 5: Monitor ActivityPub Federation Endpoints

PieFed's ActivityPub implementation exposes inbox and outbox endpoints for communities and users. Remote Lemmy instances, Mastodon servers, and other ActivityPub-compatible platforms deliver posts through these endpoints.

Check the WebFinger discovery endpoint — federated servers use this to look up your communities and users:

curl "https://piefed.yourdomain.com/.well-known/webfinger?resource=acct:yourcommunity@piefed.yourdomain.com"

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://piefed.yourdomain.com/.well-known/webfinger?resource=acct:admin@piefed.yourdomain.com.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: subject.
5. Save.

Check the NodeInfo endpoint for federation capability advertisement:

1. Click New Monitor → HTTP.
2. Set URL to https://piefed.yourdomain.com/.well-known/nodeinfo.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: links.
5. Save.

Step 6: Monitor Search Index Service Health

PieFed supports full-text search for posts and communities. Search index downtime doesn't break the platform but degrades discoverability — users searching for content get errors or empty results.

If PieFed uses PostgreSQL full-text search, this is already covered by Step 4. If you're running a separate search backend (Elasticsearch or Meilisearch), add a health check:

# For Meilisearch
@health_bp.route('/health/search')
def search_health():
    try:
        resp = requests.get('http://localhost:7700/health', timeout=5)
        if resp.json().get('status') == 'available':
            return jsonify({'status': 'ok'})
        return jsonify({'status': 'error'}), 503
    except Exception:
        return jsonify({'status': 'error'}), 503

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://piefed.yourdomain.com/health/search.
3. Set Expected status code to 200.
4. Under Keyword check, add keyword present: ok.
5. Save.

Step 7: Monitor Email Notification Delivery

PieFed sends email notifications for replies, mentions, moderation events, and account actions. SMTP failures are silent — users simply stop receiving emails.

Create a periodic Celery task that verifies SMTP connectivity and pings a Vigilmon heartbeat:

@shared_task
def check_email_delivery():
    import smtplib
    try:
        with smtplib.SMTP(current_app.config['MAIL_SERVER'],
                          current_app.config['MAIL_PORT']) as s:
            s.noop()
        requests.get('https://vigilmon.online/heartbeat/YOUR_EMAIL_TOKEN', timeout=10)
    except Exception:
        pass  # Vigilmon will alert on missing heartbeat

Schedule it in Celery beat every 6 hours:

beat_schedule = {
    'email-health-check': {
        'task': 'tasks.heartbeat.check_email_delivery',
        'schedule': 21600.0,  # 6 hours
    },
}

Create the heartbeat monitor first:

1. Click New Monitor → Heartbeat in Vigilmon.
2. Set Name to PieFed Email Delivery.
3. Set Expected heartbeat interval to 6 hours.
4. Save. Use the provided URL in the task above.

Step 8: Monitor Scheduled Federation Sync and Cleanup Jobs

PieFed runs periodic Celery beat jobs to synchronize remote community data, expire old federation tokens, clean up stale content, and maintain database health. If Celery beat stops — separate from the worker processes — these scheduled tasks stop running while on-demand tasks continue.

Create a dedicated heartbeat monitor for Celery beat:

1. Click New Monitor → Heartbeat in Vigilmon.
2. Set Name to PieFed Celery Beat Scheduler.
3. Set Expected heartbeat interval to 15 minutes.
4. Save. Copy the heartbeat URL.

Add a Celery beat task that pings this separate heartbeat URL:

beat_schedule = {
    'vigilmon-beat-heartbeat': {
        'task': 'tasks.heartbeat.ping_beat_vigilmon',
        'schedule': 600.0,  # every 10 minutes
    },
}

@shared_task
def ping_beat_vigilmon():
    requests.get('https://vigilmon.online/heartbeat/YOUR_BEAT_TOKEN', timeout=10)

A missed heartbeat here specifically tells you that Celery beat has stopped scheduling, even if the workers themselves are running.

Step 9: Monitor Media Storage Backend Availability

PieFed stores uploaded images and media attachments either locally or in S3-compatible object storage. Storage backend failures cause media uploads to fail and attached images in posts to return errors.

Check a known static asset or the media proxy endpoint:

curl -I https://piefed.yourdomain.com/static/favicon.ico

For S3-backed storage, check the bucket endpoint:

curl -I https://your-bucket.s3.yourdomain.com/

In Vigilmon:

1. Click New Monitor → HTTP.
2. Set URL to https://piefed.yourdomain.com/static/favicon.ico.
3. Set Expected status code to 200.
4. Save.

Step 10: SSL Certificate Expiry Alerts

PieFed must run over HTTPS for ActivityPub federation. A lapsed certificate causes remote servers to reject deliveries to your instance.

1. Open the web UI monitor from Step 1.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Save.

Alerting Configuration

Set up notification channels so you're alerted immediately on failures:

1. In Vigilmon, go to Alerts → Notification Channels.
2. Add an Email channel with your admin address.
3. Optionally add a Slack webhook for your ops channel.
4. Assign all PieFed monitors to these notification channels.

Recommended alert thresholds:

| Monitor | Alert after | |---|---| | Web UI | 2 consecutive failures (2 min) | | Celery worker heartbeat | 1 missed heartbeat (5 min) | | Redis health | 2 consecutive failures | | PostgreSQL health | 1 failure (immediate) | | WebFinger/NodeInfo | 3 consecutive failures | | Search index | 5 consecutive failures | | Email delivery heartbeat | 1 missed heartbeat (6 hr) | | Celery beat heartbeat | 1 missed heartbeat (15 min) | | Media storage | 3 consecutive failures | | SSL certificate | 21 days before expiry |

PostgreSQL and Celery worker are highest priority — a database failure causes immediate user-visible errors and a stopped Celery worker halts all federation silently. Search and media storage have higher tolerance as they don't affect core community functionality.

Conclusion

A healthy PieFed instance is more than just an accessible homepage. Federation depends on Celery workers staying alive and Redis staying responsive. Moderation depends on scheduled jobs running. User communication depends on email delivery working. Without external monitoring, any of these can fail silently for hours while the web interface continues to look fine.

With Vigilmon watching your PieFed web interface, Celery workers, Redis broker, PostgreSQL connectivity, ActivityPub federation endpoints, search index, email delivery, scheduled jobs, and media storage, you'll know about failures within minutes rather than learning from your users.

Start monitoring your PieFed instance for free at vigilmon.online.