Monitoring Apache Superset in Production

Apache Superset is a popular open-source BI platform, but running it reliably in production means more than just keeping the web server alive. You need visibility into SQL Lab query latency, Celery worker availability, cache layer health, and the underlying metadata database — all at once. This guide walks through the full monitoring stack for Superset, culminating in external uptime checks via Vigilmon.

What Can Go Wrong in Superset

Before wiring up monitors, it helps to know where Superset tends to fail:

• The web server goes down — the most obvious failure; dashboards are unreachable
• Celery workers die — async queries, email alerts, and report exports silently stop working
• Cache layer degrades — Redis becomes unavailable; every dashboard query hits the database directly, causing latency spikes
• SQL Lab becomes unresponsive — long-running queries pile up; the async queue backs up; users see "pending" indefinitely
• Metadata database is slow — chart config loads slowly; Superset feels sluggish even when the BI queries themselves are fine

Each failure mode has a different detection strategy.

1. Health Check Endpoints

Superset exposes a /health endpoint out of the box:

curl -s http://localhost:8088/health
# {"status":"ok"}

This confirms the Gunicorn/uWSGI process is alive and can reach the metadata database. Wire it into your load balancer's health check so traffic stops routing to a bad instance immediately.

For a deeper check, hit the readiness endpoint:

curl -s http://localhost:8088/api/v1/me/roles/
# Returns 401 if the app is up but you're unauthenticated — which is fine for an uptime check

A 401 means the app server is running. A 502 or connection refused means something is broken.

2. Celery Worker Monitoring

Superset relies on Celery for async query execution and scheduled reports. Worker failures are silent by default.

Check worker heartbeats

# Inspect active workers
celery -A superset.tasks.celery_app inspect active

# Check registered tasks
celery -A superset.tasks.celery_app inspect registered

If the command hangs or returns Error: No nodes replied, workers are down.

Export a worker health metric

Add a lightweight health endpoint to your Superset config:

# superset_config.py

CELERY_CONFIG = {
    "broker_url": "redis://localhost:6379/0",
    "result_backend": "redis://localhost:6379/0",
    "worker_heartbeat": 10,  # seconds
}

Then wrap the Celery ping in a small Flask route you can expose at /celery-health:

from celery.app.control import Control

def celery_healthy():
    control = Control(celery_app)
    result = control.ping(timeout=3)
    return len(result) > 0

Return HTTP 200 when celery_healthy() is True, 503 otherwise. This gives you a scrapable endpoint for your uptime monitor.

3. Cache Layer (Redis) Uptime

Superset's caching is configured via CACHE_CONFIG in superset_config.py. If Redis goes down, Superset falls back to no-cache mode — queries become expensive.

Check Redis directly:

redis-cli -h localhost -p 6379 ping
# PONG

For a scripted check you can plug into a monitor:

#!/bin/bash
RESULT=$(redis-cli -h localhost -p 6379 ping 2>&1)
if [ "$RESULT" = "PONG" ]; then
  echo "ok"
  exit 0
else
  echo "redis down: $RESULT"
  exit 1
fi

You can serve this script's exit code behind a tiny HTTP endpoint using a tool like prometheus-node-exporter or a custom Flask/FastAPI route.

4. SQL Lab Query Latency

SQL Lab is where latency issues compound. Use Superset's built-in query logging to track this.

Enable query logging in superset_config.py:

QUERY_LOGGER = True

Then tail the query log and look for slow queries:

# In your logs, look for lines with query execution time
grep "Query execution time" /var/log/superset/superset.log | awk '{print $NF}' | sort -n | tail -20

For a Prometheus-compatible setup, add statsd_host to your Celery config and use the django-statsd or celery-statsd exporter to ship query duration metrics to Grafana.

5. Metadata Database Health

Superset stores chart configs, user info, and dashboard layouts in a relational database (Postgres or MySQL). Query the metadata database directly to check its health:

-- Run this as a health probe
SELECT 1;

A more meaningful check verifies Superset's own tables are intact:

SELECT COUNT(*) FROM slices;  -- charts table
SELECT COUNT(*) FROM dashboards;

If these queries take more than a second, your metadata store is under load.

6. Integrating Vigilmon for External Uptime Checks

Internal checks are great for alerting when you're already in the problem. External checks catch outages from the user's perspective — when your load balancer, DNS, or network has a problem.

Step 1: Add a Superset monitor in Vigilmon

In your Vigilmon dashboard, create an HTTP monitor pointing to your public Superset URL:

• URL: https://superset.yourdomain.com/health
• Method: GET
• Expected status: 200
• Check interval: 60 seconds
• Alert threshold: 2 consecutive failures

Step 2: Add a Celery health check monitor

If you exposed the /celery-health endpoint described above:

• URL: https://superset.yourdomain.com/celery-health
• Method: GET
• Expected body contains: "ok"
• Check interval: 2 minutes

Step 3: Set up alert channels

In Vigilmon, configure alert destinations for your Superset monitors:

# vigilmon alert config (example)
monitors:
  - name: "Superset Web"
    url: https://superset.yourdomain.com/health
    interval: 60
    alerts:
      - type: slack
        webhook: https://hooks.slack.com/services/YOUR/WEBHOOK
      - type: email
        to: oncall@yourteam.com

  - name: "Superset Celery"
    url: https://superset.yourdomain.com/celery-health
    interval: 120
    alerts:
      - type: pagerduty
        integration_key: YOUR_PD_KEY

Step 4: Create a status page

Vigilmon lets you create a public status page that aggregates all your Superset component monitors. Share it with stakeholders so they can check service health without pinging on-call.

Alerting Strategy

Not every Superset issue needs a 3am page. Here's a tiered approach:

| Component | Check interval | Alert channel | Severity | |---|---|---|---| | Web server /health | 60s | PagerDuty | Critical | | Celery workers | 2m | Slack + email | High | | Redis cache | 2m | Slack | Medium | | SQL Lab latency p95 | 5m | Email | Low | | Metadata DB queries | 5m | Slack | Medium |

Putting It Together

A fully monitored Superset stack has:

1. Load balancer health checks hitting /health to keep traffic away from bad instances
2. Internal Celery ping every 30 seconds with alerting if workers go quiet
3. Redis PING check with a /cache-health HTTP endpoint for external monitoring
4. SQL Lab query duration tracked via statsd and graphed in Grafana
5. External Vigilmon uptime checks on the public URL with 60-second intervals
6. A public status page for stakeholders and a PagerDuty integration for the on-call team

With this stack in place you'll know about Superset degradation before your users do — and have enough signal to know exactly which layer to fix.