Monitoring Apache Superset with Vigilmon

Apache Superset is a modern, open source data exploration and business intelligence platform used by teams ranging from startups to Fortune 500 companies. It handles everything from interactive SQL queries to complex dashboards with dozens of charts — but when a Celery worker dies, Redis goes stale, or a database connection pool exhausts, users see blank charts and silent failures. Vigilmon gives you the uptime monitoring layer Superset lacks natively: probing every service that the web UI depends on and alerting before a silent failure turns into a user complaint.

What You'll Set Up

• HTTP availability monitor for the Superset web UI (port 8088)
• Celery worker health check via the Flower monitoring interface
• Redis cache connectivity probe
• Database connection pool health endpoint
• Chart rendering service monitor
• Cron heartbeat for the thumbnail generation worker
• Dashboard load time tracking

Prerequisites

• Apache Superset running (Docker Compose, Kubernetes, or bare-metal)
• Celery workers configured for async queries
• Redis running as the Celery broker and result backend
• A free Vigilmon account

Why Monitoring Superset Is Different

Superset is a distributed system masquerading as a web app. The UI at port 8088 is just one piece:

• Celery workers execute long-running SQL queries and cache results asynchronously
• Redis brokers task messages and stores query results
• SQLAlchemy maintains connection pools to each connected database
• Thumbnail workers pre-render chart images for email reports and embedded dashboards

A healthy-looking web UI can coexist with broken async queries if Celery workers have crashed — users only notice when they submit a query and wait forever. Monitoring each layer independently is the only reliable approach.

Step 1: Monitor the Superset Web UI

The first and most basic check: is Superset itself up?

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Superset URL: http://your-superset-host:8088/health (or https:// if behind a proxy).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Superset exposes a /health endpoint that returns "OK" when the Flask application is running. This is better than checking the root URL, which may redirect through authentication and return a 302.

GET /health HTTP/1.1
Host: your-superset-host:8088

HTTP/1.1 200 OK
OK

If your Superset is behind a reverse proxy (nginx, Traefik, Caddy), add the SSL monitor here as well:

1. Open the monitor you just created.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.

Step 2: Monitor Celery Workers via Flower

Flower is the web-based monitoring tool for Celery. When deployed alongside Superset, it exposes a REST API you can probe to verify that workers are alive.

If Flower isn't running yet, add it to your Superset deployment:

# Docker Compose: add to docker-compose.yml
flower:
  image: mher/flower
  command: celery --broker=redis://redis:6379/0 flower --port=5555
  ports:
    - "5555:5555"
  depends_on:
    - redis

Then add a Vigilmon monitor for Flower's worker API:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-superset-host:5555/api/workers.
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.
5. Under Response body, add a keyword check: the response should contain "celery@" (the worker name prefix).
6. Click Save.

The /api/workers endpoint returns a JSON object where keys are worker names. If the list is empty or the request fails, your async queries are queuing up with no one to process them.

Step 3: Monitor Redis Cache Connectivity

Superset uses Redis as the Celery broker, result backend, and optionally as a cache backend. A dead Redis means no async queries complete and no cached results are served.

Add a TCP monitor for Redis:

1. Click Add Monitor → TCP Port.
2. Host: your-redis-host.
3. Port: 6379.
4. Set Check interval to 1 minute.
5. Click Save.

For a richer check that validates Redis is responding to commands (not just accepting connections), expose a health endpoint in your Superset application:

# superset_config.py — add a custom health check blueprint
from flask import Blueprint, jsonify
import redis

health_bp = Blueprint('health_extended', __name__)

@health_bp.route('/health/redis')
def redis_health():
    r = redis.Redis(host='redis', port=6379)
    r.ping()
    return jsonify({'redis': 'ok'})

BLUEPRINTS = [health_bp]

Then monitor http://your-superset-host:8088/health/redis as an HTTP monitor expecting status 200.

Step 4: Monitor Database Connection Pool Health

Superset connects to your analytics databases (PostgreSQL, BigQuery, Snowflake, etc.) via SQLAlchemy connection pools. If the pool exhausts — typically from long-running queries or leaked connections — new queries hang indefinitely.

Add a health endpoint that exercises the connection pool:

# In your custom health blueprint
from superset import db

@health_bp.route('/health/db')
def db_health():
    db.session.execute('SELECT 1')
    return jsonify({'database': 'ok'})

Monitor http://your-superset-host:8088/health/db with:

1. Type: HTTP / HTTPS
2. Expected status: 200
3. Check interval: 2 minutes
4. Keyword check: "ok"

This catches connection pool exhaustion and database host failures before they affect dashboards.

Step 5: Monitor SQL Lab Query Execution

SQL Lab is Superset's interactive query editor. It depends on Celery workers for async execution. You can detect worker failures by checking the Celery task queue length via Flower:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-superset-host:5555/api/queues/info.
3. Set Check interval to 5 minutes.
4. Leave the keyword check empty — you're checking that the endpoint responds.
5. Click Save.

For deeper alerting, set up a Vigilmon webhook that triggers a custom script:

#!/bin/bash
# check-celery-queue.sh — alert if queue depth > 50
QUEUED=$(curl -s http://localhost:5555/api/queues/info | jq '.celery.messages')
if [ "$QUEUED" -gt 50 ]; then
  echo "Celery queue depth is $QUEUED — workers may be stuck"
  exit 1
fi

Step 6: Heartbeat for Thumbnail Generation Worker

Superset's thumbnail worker pre-renders chart and dashboard images for email digests and embedded use cases. It runs on a schedule and typically has no HTTP endpoint to probe.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to match your thumbnail schedule (e.g., 60 minutes).
3. Copy the heartbeat URL.
4. Add the ping to your thumbnail worker script:

import requests
from superset.tasks.thumbnails import cache_dashboard_thumbnail

def run_thumbnails():
    for dashboard_id in get_active_dashboard_ids():
        cache_dashboard_thumbnail.delay(dashboard_id)
    # Signal success to Vigilmon
    requests.get('https://vigilmon.online/heartbeat/your-id', timeout=5)

If the thumbnail worker crashes mid-run and never pings, Vigilmon alerts after the expected interval.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. Set Consecutive failures before alert to 2 on HTTP monitors — transient network hiccups shouldn't wake the on-call team.
3. For the Celery worker monitor, set it to 1 — a dead worker means queries are silently failing right now.

Recommended alert routing for Superset:

| Condition | Alert channel | Urgency | |---|---|---| | Web UI /health down | Slack #incidents + email | High | | Celery workers gone | Slack #incidents | Critical | | Redis TCP down | Slack #incidents | Critical | | DB connection pool failure | Slack #oncall | High | | Thumbnail heartbeat missed | Slack #data-team | Medium |

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web UI health | :8088/health | Flask app crash, proxy failure | | Celery workers | :5555/api/workers | Worker crash, task queue stuck | | Redis TCP | :6379 | Cache/broker down | | DB connection pool | :8088/health/db | Pool exhaustion, DB unreachable | | SQL Lab queue | :5555/api/queues/info | Celery backlog | | Thumbnail heartbeat | Heartbeat URL | Worker crash, missed schedule |

Superset's power comes from orchestrating many services together — and that's exactly what makes silent failures so dangerous. With Vigilmon watching every layer from the web UI down to the Celery workers, your data team gets alerted before users notice that their Monday morning dashboard is showing stale data or empty charts.