tutorial

JupyterHub Monitoring: Keep Your Data Science Platform Running With External Health Checks

How to monitor JupyterHub on Kubernetes — proxy availability, hub health, user server spawning, and cull-job completion — using Vigilmon HTTP and heartbeat monitoring.

JupyterHub is the shared Jupyter Notebook platform for teams and classrooms — a single URL that spawns per-user Jupyter servers on demand. Deployed on Kubernetes with Zero to JupyterHub (z2jh), it's widely used in data science teams, ML research groups, and universities. When JupyterHub goes down, every data scientist in your organization loses access to their notebooks and running kernels simultaneously.

Vigilmon gives JupyterHub the external monitoring layer it needs: continuous HTTP checks that detect proxy failures and hub authentication errors before users flood Slack with complaints, and heartbeat monitors that catch failed cull and maintenance jobs before they consume all your cluster's memory.

Why JupyterHub Needs External Monitoring

JupyterHub on Kubernetes consists of three main components: a proxy (configurable-http-proxy), a hub (the JupyterHub application), and per-user single-user servers (notebook pods). Each layer can fail independently:

Proxy failure silently kills all user connections. The configurable-http-proxy pod handles all incoming traffic. If it crashes, every active user session is interrupted — but Kubernetes may take 30–60 seconds to restart the pod, during which all traffic returns 502. Your internal probes report "pod restarting" while users are locked out.

Hub database corruption or migration failure. The hub stores user sessions and server state in a SQLite or PostgreSQL database. A failed database migration during a hub upgrade leaves the hub starting but returning 500s on login.

User server spawn failures. A user clicks "Start My Server" and waits indefinitely when the cluster has insufficient resources to schedule their pod. The hub is healthy; only spawning is broken — and this doesn't surface until a user reports it.

Cull jobs stop running. JupyterHub's idle culler shuts down inactive user servers to reclaim cluster resources. If the cull CronJob fails silently, idle servers accumulate over days until the cluster runs out of memory and begins evicting active workloads.

Key Endpoints and Signals to Monitor

| Signal | Endpoint | Failure Indicator | |--------|----------|-------------------| | Proxy availability | https://jupyterhub.yourdomain.com/ | Non-200 or connection error | | Hub health API | https://jupyterhub.yourdomain.com/hub/api/ | Non-200 or {"error": ...} | | Hub login page | https://jupyterhub.yourdomain.com/hub/login | Non-200 means auth is broken | | Cull job heartbeat | Heartbeat ping from culler | Missing ping means culler stopped |

What You'll Set Up

  • HTTP monitor for the JupyterHub proxy
  • HTTP monitor for the hub health API
  • Heartbeat monitor for the idle culler CronJob
  • Alert channel to Slack

You'll need a free Vigilmon account — no credit card required.

Step 1: Confirm Your JupyterHub Is Externally Accessible

Before setting up monitoring, verify the endpoints Vigilmon will probe:

# Test proxy availability
curl -sI https://jupyterhub.yourdomain.com/ | head -5

# Test hub API health
curl -s https://jupyterhub.yourdomain.com/hub/api/ | python3 -m json.tool

# Should return something like:
# {"version": "4.0.2"}

If the hub API returns a version string without authentication, monitoring is straightforward. If your hub is behind OAuth or other authentication, use the /hub/api/ endpoint which returns version info without credentials.

Step 2: Monitor the JupyterHub Proxy

The proxy is the first point of failure for all users. Monitor it directly:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://jupyterhub.yourdomain.com/
  4. Check interval: 1 minute
  5. Expected status code: 200 or 302 (JupyterHub typically redirects / to /hub/login)
  6. Response time threshold: 5000ms
  7. Save the monitor

This catches proxy pod crashes, Ingress controller failures, and TLS certificate expiry.

Step 3: Monitor the Hub Health API

The hub API endpoint is lightweight and doesn't require authentication — it's safe to probe from Vigilmon:

  1. Add a second monitor with URL: https://jupyterhub.yourdomain.com/hub/api/
  2. Expected status code: 200
  3. Response body contains: "version" (confirms the hub application is responding)
  4. Check interval: 1 minute
  5. Response time threshold: 3000ms

This catches hub application crashes, database connectivity failures, and misconfigured hub configuration that leaves the proxy running but the hub broken.

Step 4: Heartbeat Monitoring for the Idle Culler

The JupyterHub idle culler typically runs as a service or a CronJob. If it stops running, idle servers pile up and eventually starve the cluster. Set up a heartbeat to catch this silently.

Create the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name it: jupyterhub-idle-culler
  3. Expected interval: 1 hour (if your culler runs hourly)
  4. Grace period: 30 minutes
  5. Save — copy the heartbeat URL

Add a Heartbeat to Your Culler CronJob

If you run the culler as a Kubernetes CronJob (common in self-managed deployments):

# culler-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: jupyterhub-culler
  namespace: jhub
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: culler
              image: jupyterhub/jupyterhub:latest
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  python -m jupyterhub_idle_culler \
                    --url=http://hub:8081/hub/api \
                    --timeout=3600 \
                    --concurrency=5
                  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: culler-heartbeat-url
                - name: JUPYTERHUB_API_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: hub-secret
                      key: api-token

Store the secret:

kubectl create secret generic vigilmon-secrets \
  --namespace jhub \
  --from-literal=culler-heartbeat-url='https://vigilmon.online/heartbeat/your-unique-token'

If You Use the jupyterhub-idle-culler Service (z2jh)

For Zero to JupyterHub deployments that run the culler as a long-running service, add a periodic health ping to a wrapper script or use a sidecar:

# culler_wrapper.py — wrap the standard culler with a heartbeat
import os
import time
import urllib.request
import subprocess

HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]
CULL_INTERVAL = 3600  # seconds

while True:
    result = subprocess.run(
        ["python", "-m", "jupyterhub_idle_culler", "--once",
         "--url", "http://hub:8081/hub/api", "--timeout", "3600"],
        capture_output=True
    )
    if result.returncode == 0:
        urllib.request.urlopen(HEARTBEAT_URL, timeout=10)
    time.sleep(CULL_INTERVAL)

Step 5: Status Page for Your Data Science Team

Create a Vigilmon status page so your data scientists can self-serve during incidents:

  1. Go to Status Pages → New Status Page
  2. Name it: "Data Science Platform"
  3. Add monitors:
    • JupyterHub Proxy
    • JupyterHub Hub API
    • Idle Culler (optional — your team may not care about this one)
  4. Share the URL in your team's Slack channel pinned messages

When users report issues, they can check the status page themselves before filing a ticket — saving your platform team triage time.

Step 6: Alert Channels

  1. Create a Slack Incoming Webhook for #platform-alerts
  2. In Vigilmon, go to Alert Channels → New Channel → Webhook
  3. Paste the URL
  4. Assign to the Proxy and Hub API monitors

For the culler heartbeat, consider routing to a lower-urgency channel like #platform-ops — a missed cull won't cause immediate user impact, but needs attention within a few hours.

Recommended Thresholds

| Monitor | Check Interval | Alert After | Notes | |---------|---------------|-------------|-------| | Proxy | 1 min | 2 failures | Two consecutive failures = real outage | | Hub API | 1 min | 2 failures | Allow for pod restarts | | Idle Culler heartbeat | 1h schedule | Grace: 30min | Daily idle pods accumulate quickly |

Summary

JupyterHub's internal Kubernetes health checks keep pods running but cannot tell you whether users can actually authenticate, spawn servers, or whether idle servers are consuming your cluster's memory. Vigilmon provides the external perspective your platform needs.

| Failure Mode | Kubernetes | Vigilmon | |---|---|---| | Proxy pod crash | Restarts pod | Alerts during the 30–60s restart window | | Hub database error | ✗ | ✓ via /hub/api/ monitor | | Ingress routing broken | ✗ | ✓ | | TLS certificate expired | ✗ | ✓ | | Idle culler stopped | ✗ | ✓ via heartbeat | | Hub returns 500 on login | ✗ | ✓ via login page check |

Get your JupyterHub monitored free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles