Self-Hosted SuperAGI Monitoring — Backend, Celery Workers, and Agent Health (Free)

SuperAGI lets you build, manage, and run autonomous AI agents that can browse the web, search, write files, and execute code across multi-step task plans. Under the hood it's a FastAPI backend (port 8001), a React frontend (port 3000), PostgreSQL, Redis, Celery workers, and optionally Qdrant for vector memory. Each component is a failure point — and a downed Celery worker silently stops all running agents without any error visible in the UI.

This guide sets up HTTP health checks, Celery worker heartbeats, database connectivity monitoring, and agent run status alerts — free on Vigilmon.

The failure modes SuperAGI users miss

Celery worker crash — Agents run as Celery tasks. When the Celery worker process dies, all in-flight agent executions stop mid-task. No new agents can start. The FastAPI backend returns 200s, and the frontend looks healthy, but agents are permanently paused.

Redis connectivity loss — Celery uses Redis as its message broker. If Redis becomes unavailable, Celery workers can't receive new tasks. Active agents may continue briefly before timing out, but no new work is dispatched.

PostgreSQL connectivity loss — Agent definitions, run histories, tool configurations, and API keys are stored in PostgreSQL. A database outage means agents can't load their configurations or save results.

Qdrant vector store downtime — Agents with long-term memory depend on Qdrant. If Qdrant becomes unreachable, agents that rely on memory retrieval fail mid-execution without a clear error.

Stuck agent runs — An agent can get stuck in a loop, waiting for an LLM response that never arrives, or retrying a failed tool call indefinitely. Without monitoring, stuck agents hold Celery worker slots and block other work.

Step 1: Add a comprehensive health endpoint to SuperAGI

SuperAGI's FastAPI backend runs on port 8001. Add a health router that checks every dependency:

# superagi/health.py
from fastapi import APIRouter
from fastapi.responses import JSONResponse
import asyncpg
import aioredis
import httpx
import os
import time

router = APIRouter()

_start_time = time.time()


async def check_postgres() -> dict:
    db_url = os.environ.get("DB_URL") or os.environ.get("DATABASE_URL")
    if not db_url:
        return {"status": "not_configured"}
    try:
        conn = await asyncpg.connect(db_url, timeout=3)
        await conn.execute("SELECT 1")
        await conn.close()
        return {"status": "ok"}
    except Exception as e:
        return {"status": "error", "detail": str(e)}


async def check_redis() -> dict:
    redis_url = os.environ.get("REDIS_URL", "redis://localhost:6379")
    try:
        r = await aioredis.from_url(redis_url, socket_timeout=3)
        await r.ping()
        await r.close()
        return {"status": "ok"}
    except Exception as e:
        return {"status": "error", "detail": str(e)}


async def check_qdrant() -> dict:
    qdrant_url = os.environ.get("QDRANT_HOST_NAME", "http://localhost:6333")
    if not qdrant_url:
        return {"status": "not_configured"}
    try:
        async with httpx.AsyncClient(timeout=3) as client:
            res = await client.get(f"{qdrant_url}/healthz")
            return {"status": "ok" if res.status_code == 200 else "error", "code": res.status_code}
    except Exception as e:
        return {"status": "error", "detail": str(e)}


async def check_celery_workers() -> dict:
    """Check Celery worker availability via Redis queue depth."""
    redis_url = os.environ.get("REDIS_URL", "redis://localhost:6379")
    try:
        r = await aioredis.from_url(redis_url, socket_timeout=3)
        # Celery default queue; adjust if you use named queues
        queue_length = await r.llen("celery")
        await r.close()
        return {
            "status": "ok",
            "queue_depth": queue_length,
            # Large queue depth with no workers draining it is a warning sign
            "backlog_warning": queue_length > 100,
        }
    except Exception as e:
        return {"status": "error", "detail": str(e)}


async def check_llm_provider() -> dict:
    openai_key = os.environ.get("OPENAI_API_KEY")
    if not openai_key:
        return {"status": "not_configured"}
    try:
        async with httpx.AsyncClient(timeout=5) as client:
            res = await client.get(
                "https://api.openai.com/v1/models",
                headers={"Authorization": f"Bearer {openai_key}"},
            )
            return {"status": "ok" if res.status_code == 200 else "error"}
    except Exception as e:
        return {"status": "error", "detail": str(e)}


@router.get("/health")
async def health_check():
    postgres, redis, qdrant, celery, llm = await asyncio.gather(
        check_postgres(),
        check_redis(),
        check_qdrant(),
        check_celery_workers(),
        check_llm_provider(),
    )

    # Critical: postgres and redis must be ok
    critical_ok = (
        postgres["status"] == "ok"
        and redis["status"] == "ok"
    )
    # Soft: qdrant and llm degradation doesn't block all operations
    all_ok = critical_ok and qdrant.get("status") in ("ok", "not_configured")

    return JSONResponse(
        status_code=200 if all_ok else 503,
        content={
            "status": "ok" if all_ok else "degraded",
            "uptime_seconds": int(time.time() - _start_time),
            "checks": {
                "postgres": postgres,
                "redis": redis,
                "qdrant": qdrant,
                "celery_queue": celery,
                "llm_provider": llm,
            },
        },
    )

Register the router in main.py:

from superagi.health import router as health_router
app.include_router(health_router)

Test:

curl http://localhost:8001/health
# {"status":"ok","uptime_seconds":3721,"checks":{"postgres":{"status":"ok"},"redis":{"status":"ok"},...}}

Step 2: HTTP monitoring with Vigilmon

With both the FastAPI backend and React frontend running, set up external monitoring via Vigilmon:

1. Sign up at vigilmon.online — free, no credit card
2. Click New Monitor → HTTP
3. Enter http://your-server:8001/health
4. Set interval to 1 minute (agent workloads need fast detection)
5. Save

Add monitors for all surfaces:

| Monitor | URL/Port | What it catches | |---|---|---| | FastAPI backend | http://server:8001/health | DB/Redis/Qdrant down, any critical failure | | React frontend | http://server:3000/ | Frontend crash, nginx failure | | Redis | TCP port 6379 | Redis process down | | Qdrant | http://server:6333/healthz | Vector store unreachable |

For the React frontend on port 3000, a root HTTP check returning 200 is sufficient — you just need to confirm nginx or the dev server is serving.

Step 3: Celery worker heartbeat monitoring

This is the most important monitor for SuperAGI. Celery workers run agents — if they crash, agents stop silently.

Add a heartbeat task to Celery that pings Vigilmon periodically:

# superagi/tasks/health_tasks.py
import os
import requests
from celery import shared_task
import logging

logger = logging.getLogger(__name__)

WORKER_HEARTBEAT_URL = os.environ.get("WORKER_HEARTBEAT_URL")


@shared_task(name="health.worker_heartbeat", bind=True, max_retries=0)
def worker_heartbeat(self):
    """Ping Vigilmon heartbeat to confirm this worker is alive."""
    if not WORKER_HEARTBEAT_URL:
        logger.warning("WORKER_HEARTBEAT_URL not set, skipping heartbeat")
        return

    try:
        requests.get(WORKER_HEARTBEAT_URL, timeout=5)
        logger.debug("Worker heartbeat sent")
    except Exception as e:
        logger.warning("Failed to send worker heartbeat: %s", e)

Schedule it to run every 5 minutes in your Celery beat configuration:

# celeryconfig.py
from celery.schedules import crontab

beat_schedule = {
    "worker-heartbeat": {
        "task": "health.worker_heartbeat",
        "schedule": 300,  # every 5 minutes
    },
    # ... your other scheduled tasks
}

If you're using Django Celery Beat or a celery.conf.beat_schedule dict, add the entry there.

In Vigilmon:

1. New Monitor → Heartbeat
2. Name: superagi-celery-worker
3. Expected interval: 10 minutes (5-min schedule with 100% buffer)
4. Copy the ping URL:

# .env
WORKER_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/your-worker-token

If the worker dies, Vigilmon stops receiving pings and alerts you within 10 minutes — before you'd notice agents are stuck.

Step 4: Agent run health monitoring

Add a heartbeat for active agent runs. This covers the case where the worker is alive but a specific agent is stuck in an infinite loop or waiting on a hung LLM call.

# superagi/tasks/agent_tasks.py
import os
import time
import requests
from celery import shared_task

AGENT_RUN_HEARTBEAT_URL = os.environ.get("AGENT_RUN_HEARTBEAT_URL")


@shared_task(name="agents.run_agent", bind=True)
def run_agent(self, agent_id: int, run_id: int):
    """Run an agent, pinging heartbeat each completed iteration."""
    max_iterations = 100
    timeout_per_iteration = 300  # 5 minutes per step

    for iteration in range(max_iterations):
        step_start = time.time()

        # Execute one agent step (your actual logic here)
        result = execute_agent_step(agent_id, run_id, iteration)

        step_duration = time.time() - step_start
        if step_duration > timeout_per_iteration:
            raise TimeoutError(
                f"Agent step {iteration} exceeded {timeout_per_iteration}s timeout"
            )

        # Ping heartbeat after each successful step
        if AGENT_RUN_HEARTBEAT_URL:
            try:
                requests.get(AGENT_RUN_HEARTBEAT_URL, timeout=5)
            except Exception:
                pass

        if result.get("done"):
            break

In Vigilmon, create a Heartbeat monitor with a 15-minute expected interval. If your agent runs regularly and each step pings, a missed ping tells you that either the worker is stuck on a single step for too long or the worker process is dead.

Step 5: Database-level agent run monitoring

For production deployments, query PostgreSQL directly to detect stuck runs. Add this to your health endpoint:

async def check_stuck_agents(db_url: str, stuck_threshold_minutes: int = 30) -> dict:
    """Detect agent runs that have been active too long without completing."""
    try:
        conn = await asyncpg.connect(db_url, timeout=3)
        stuck_runs = await conn.fetch(
            """
            SELECT id, agent_id, created_at
            FROM agent_execution
            WHERE status = 'RUNNING'
              AND created_at < NOW() - INTERVAL '$1 minutes'
            """,
            stuck_threshold_minutes,
        )
        await conn.close()
        return {
            "status": "ok",
            "stuck_run_count": len(stuck_runs),
            "stuck_run_ids": [r["id"] for r in stuck_runs],
            "warning": len(stuck_runs) > 0,
        }
    except Exception as e:
        return {"status": "error", "detail": str(e)}

Include this in the /health response. If stuck_run_count > 0 for more than 30 minutes, that's a signal worth paging on.

Step 6: TLS certificate monitoring

If SuperAGI is exposed publicly, Vigilmon automatically monitors TLS expiry on every HTTPS probe. To add explicit expiry alerting:

1. In Vigilmon, open your HTTPS backend or frontend monitor
2. Enable SSL expiry alert with a 30-day threshold

You'll receive a warning email before the certificate expires — preventing the self-signed cert browser warning that users inevitably screenshot and send to your support channel.

Step 7: Slack and email alerts

Configure alert delivery:

For Slack:

1. Create an incoming webhook
2. In Vigilmon: Notifications → New Channel → Slack → paste URL
3. Enable on all SuperAGI monitors

For email: Notifications → New Channel → Email

Alerts you'll receive:

🔴 DOWN: superagi-server:8001/health
Status: 503 degraded
Checks: redis=error (Connection refused), postgres=ok

🔴 MISSED: superagi-celery-worker heartbeat
Expected every: 10 minutes
Last ping: 23 minutes ago
→ All agent executions have stopped

🔴 DOWN: superagi-server:6333/healthz
Status: connection refused
→ Agent vector memory unavailable

Step 8: Public status page

If SuperAGI is running for a team, give them visibility without SSH access:

1. Status Pages → New Status Page in Vigilmon
2. Name: SuperAGI Platform
3. Add: FastAPI backend, React frontend, Celery worker heartbeat, agent run heartbeat
4. Share the URL with your team

When you're diagnosing an agent outage, stakeholders can check the status page and see exactly which component failed and when.

Docker Compose deployment tip

If you're running SuperAGI via Docker Compose, add a health check to your Compose services so Docker also restarts unhealthy containers:

services:
  backend:
    image: superagi/superagi:latest
    ports:
      - "8001:8001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8001/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

  celery_worker:
    image: superagi/superagi:latest
    command: celery -A superagi.worker worker -l info
    healthcheck:
      test: ["CMD", "celery", "-A", "superagi.worker", "inspect", "ping"]
      interval: 60s
      timeout: 15s
      retries: 2

Docker health checks restart crashed containers, but they don't alert you. Vigilmon's external monitoring is complementary — it tells you when a restart happened, how long the outage lasted, and whether the restart resolved it.

Summary

| Monitor | What it catches | |---|---| | HTTP backend:8001/health | DB, Redis, Qdrant down; critical dependency failures | | HTTP frontend:3000/ | React/nginx frontend crash | | Celery worker heartbeat (10 min) | Worker process crash — all agents stop | | Agent run heartbeat (15 min) | Individual agent stuck mid-execution | | TLS cert check | Certificate expiry before users notice | | Slack/email alerts | Immediate notification without dashboard watching |

SuperAGI brings serious automation power to self-hosted teams. External monitoring ensures you find out about failures within minutes — not when a user files a bug report.

Get started free at vigilmon.online →