tutorial

How to Monitor Apache Toree Uptime and Health with Vigilmon

Apache Toree is a Jupyter kernel that connects notebooks to remote Spark clusters — but kernel crashes, Spark connectivity loss, and ZMQ transport failures silently break notebook execution for every user on the platform. Here's how to monitor Toree with Vigilmon.

Apache Toree is a Jupyter kernel that enables data scientists and engineers to write and execute Scala, Python, R, and SQL code in Jupyter notebooks against remote Apache Spark clusters. Unlike local kernels, Toree maintains a persistent Spark application context inside the remote cluster, allowing notebooks to share the same SparkSession across cells and across users on multi-tenant platforms. When Toree's kernel process crashes, when its ZMQ transport becomes unreachable, or when the underlying Spark cluster loses the application context, every notebook connected to that kernel silently fails — cells stop executing, job submissions hang indefinitely, and users receive cryptic connection errors with no clear indication of what failed.

Vigilmon gives you external visibility into Toree kernel availability and Spark cluster connectivity through HTTP probe monitoring and heartbeat monitoring. This tutorial walks through both.

Why Apache Toree Monitoring Matters

Toree's multi-layered architecture introduces failure modes that standard notebook platform monitoring misses:

  • Kernel process crash kills the Spark application context — all connected notebooks lose their SparkSession and any in-memory DataFrames; users must reconnect and re-run all initialization cells
  • ZMQ transport failure causes Jupyter to show the kernel as "busy" indefinitely while the kernel is actually dead — users cannot interrupt or restart the kernel through the UI
  • Spark cluster disconnection leaves the Toree kernel process alive but unable to submit jobs — notebooks appear to run but cells executing Spark code never return
  • YARN ResourceManager unavailability prevents Toree from allocating executors — the kernel starts but all Spark operations fail immediately with resource allocation errors
  • Toree Gateway (Enterprise Gateway) unavailability on JupyterHub deployments blocks all new kernel starts — existing kernels may continue but no new notebook sessions can begin

External monitoring through Vigilmon verifies whether Toree and its Spark backend are reachable and operational — independently of Jupyter's kernel status display.

Step 1: Build a Toree Health Endpoint

Apache Toree integrates with JupyterHub and Jupyter Enterprise Gateway. The Jupyter REST API exposes kernel status. Test kernel availability through the Jupyter API:

# List running kernels via Jupyter REST API
curl -i http://your-jupyter-host:8888/api/kernels \
  -H "Authorization: token YOUR_JUPYTER_TOKEN"
# HTTP/1.1 200 OK  (returns list of running kernels)

For a deeper check that verifies Toree's Spark connectivity, build a sidecar that starts a Toree kernel, executes a probe, and checks the result:

Node.js Sidecar Health Endpoint

// toree-health.js — verifies Toree kernel availability via Jupyter API
const express = require('express');
const axios = require('axios');

const app = express();
const JUPYTER_URL = process.env.JUPYTER_URL || 'http://localhost:8888';
const JUPYTER_TOKEN = process.env.JUPYTER_TOKEN || '';

const headers = { Authorization: `token ${JUPYTER_TOKEN}` };

app.get('/health/toree', async (req, res) => {
  try {
    // Check Jupyter API is reachable
    await axios.get(`${JUPYTER_URL}/api`, { headers, timeout: 5000 });

    // List kernelspecs — verify the Toree kernel spec is registered
    const specsRes = await axios.get(`${JUPYTER_URL}/api/kernelspecs`, { headers, timeout: 10000 });
    const specs = specsRes.data.kernelspecs || {};
    const toreeSpecs = Object.keys(specs).filter(k => k.toLowerCase().includes('toree') || k.toLowerCase().includes('scala'));

    if (toreeSpecs.length === 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'no_toree_kernelspec',
        available: Object.keys(specs),
      });
    }

    // Check running kernels — any Toree kernels that are currently alive?
    const kernelsRes = await axios.get(`${JUPYTER_URL}/api/kernels`, { headers, timeout: 10000 });
    const kernels = kernelsRes.data || [];
    const toreeKernels = kernels.filter(k => toreeSpecs.some(s => k.name === s));

    return res.status(200).json({
      status: 'ok',
      toree_kernelspecs: toreeSpecs,
      running_toree_kernels: toreeKernels.length,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Python Sidecar Health Endpoint

# toree_health.py — FastAPI sidecar for Apache Toree health
import os
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
JUPYTER_URL = os.environ.get("JUPYTER_URL", "http://localhost:8888")
JUPYTER_TOKEN = os.environ.get("JUPYTER_TOKEN", "")
HEADERS = {"Authorization": f"token {JUPYTER_TOKEN}"}

@app.get("/health/toree")
async def toree_health():
    async with httpx.AsyncClient(timeout=10.0) as client:
        # Check Jupyter API reachability
        try:
            r = await client.get(f"{JUPYTER_URL}/api", headers=HEADERS)
            if r.status_code >= 500:
                return JSONResponse(status_code=503,
                                    content={"status": "down", "check": "jupyter_api",
                                             "code": r.status_code})
        except Exception as e:
            return JSONResponse(status_code=503,
                                content={"status": "down", "check": "jupyter_api", "error": str(e)})

        # Verify Toree kernelspec is registered
        try:
            r = await client.get(f"{JUPYTER_URL}/api/kernelspecs", headers=HEADERS)
            r.raise_for_status()
            specs = r.json().get("kernelspecs", {})
        except Exception as e:
            return JSONResponse(status_code=503,
                                content={"status": "down", "check": "kernelspecs", "error": str(e)})

        toree_specs = [k for k in specs if "toree" in k.lower() or "scala" in k.lower()]
        if not toree_specs:
            return JSONResponse(status_code=503, content={
                "status": "degraded",
                "reason": "no_toree_kernelspec",
                "available_specs": list(specs.keys()),
            })

        # Check running kernels
        try:
            r = await client.get(f"{JUPYTER_URL}/api/kernels", headers=HEADERS)
            r.raise_for_status()
            kernels = r.json()
        except Exception as e:
            return JSONResponse(status_code=503,
                                content={"status": "degraded", "check": "kernels", "error": str(e)})

        running_toree = [k for k in kernels if k.get("name") in toree_specs]

    return {
        "status": "ok",
        "toree_kernelspecs": toree_specs,
        "running_toree_kernels": len(running_toree),
    }

Deploy and verify:

curl -i https://your-app.example.com/health/toree
# HTTP/1.1 200 OK
# {"status":"ok","toree_kernelspecs":["toree_scala","toree_pyspark"],"running_toree_kernels":3}

Step 2: Configure a Vigilmon HTTP Monitor for Toree

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-app.example.com/health/toree
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 15000ms
  6. Under Alert channels, assign your Slack or email channel
  7. Save the monitor

For a lighter-weight check against the Jupyter API directly:

  1. Create a second monitor pointing to http://your-jupyter-host:8888/api
  2. Set the expected status code to 200
  3. Add a response body check: version
  4. Set the response time threshold to 3000ms

Failure Coverage

| Failure Mode | Jupyter UI Monitor | Vigilmon | |---|---|---| | Jupyter server crash | ✓ | ✓ | | Toree kernelspec removed/corrupted | ✗ | ✓ | | All Toree kernels dead (no running instances) | ✗ | ✓ | | Spark cluster disconnected (kernel alive, jobs hang) | ✗ | ✓ (via heartbeat) | | Enterprise Gateway unavailable | ✗ | ✓ |

Step 3: Heartbeat Monitoring for Toree Spark Job Execution

The most critical failure mode for Toree is silent Spark disconnection — the kernel process is alive, the Jupyter API reports it as running, but all Spark operations hang because the cluster is unreachable or the YARN application has been killed. Vigilmon's heartbeat monitor detects this hang by requiring periodic proof of successful job execution.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: toree-spark-execution
  3. Set the expected interval: 10 minutes (match your notebook execution cadence)
  4. Set the grace period: 20 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a Jupyter Notebook or Scheduled Script

Use the Jupyter kernel client Python library to execute a probe against a Toree kernel and ping Vigilmon on success:

# toree_heartbeat_probe.py — executes a Spark probe via Toree and pings Vigilmon
import os
import time
import requests
from jupyter_client import KernelManager

HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]
JUPYTER_URL = os.environ.get("JUPYTER_URL", "http://localhost:8888")
JUPYTER_TOKEN = os.environ.get("JUPYTER_TOKEN", "")

# A minimal Spark probe that verifies Spark context is alive
SPARK_PROBE_CODE = """
val rdd = spark.sparkContext.parallelize(Seq(1, 2, 3))
val count = rdd.count()
println(s"probe:count=$count")
"""

def probe_toree_kernel():
    km = KernelManager(kernel_name="toree_scala")
    km.start_kernel()
    kc = km.client()
    kc.start_channels()

    try:
        kc.wait_for_ready(timeout=60)
        msg_id = kc.execute(SPARK_PROBE_CODE)

        # Wait for execution to complete (up to 5 minutes)
        deadline = time.time() + 300
        while time.time() < deadline:
            msg = kc.get_iopub_msg(timeout=10)
            if msg["msg_type"] == "execute_reply":
                status = msg["content"]["status"]
                if status == "ok":
                    requests.get(HEARTBEAT_URL, timeout=5)
                    return True
                else:
                    return False
    finally:
        kc.stop_channels()
        km.shutdown_kernel(now=True)

    return False

if __name__ == "__main__":
    success = probe_toree_kernel()
    print(f"Probe {'succeeded' if success else 'FAILED'}")

Simpler Shell-Based Heartbeat via nbconvert

For lighter-weight probing without kernel client code, use jupyter nbconvert to execute a probe notebook:

#!/bin/bash
# toree_probe.sh — executes a probe notebook and pings Vigilmon on success
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"

jupyter nbconvert \
  --to notebook \
  --execute \
  --ExecutePreprocessor.timeout=300 \
  --ExecutePreprocessor.kernel_name=toree_scala \
  /probes/spark_probe.ipynb \
  --output /tmp/spark_probe_output.ipynb 2>&1

if [ $? -eq 0 ]; then
  curl -fsS "${HEARTBEAT_URL}" > /dev/null
  echo "Probe succeeded. Vigilmon pinged."
else
  echo "Probe FAILED. Vigilmon NOT pinged." >&2
  exit 1
fi

Schedule this probe via cron or your workflow scheduler at a frequency shorter than the heartbeat interval.

Step 4: Monitor Multiple Toree and JupyterHub Components

A production Toree/JupyterHub deployment spans multiple services. Monitor each independently:

[jupyterhub]              :8000/hub/api
[jupyter-single-user]     :8888/api
[toree-health]            /health/toree (sidecar)
[enterprise-gateway]      :8888/api/kernelspecs (Enterprise Gateway)
[spark-master]            :8080 (Spark master UI)
[yarn-resourcemanager]    :8088/ws/v1/cluster/info
[toree-execution]         (heartbeat monitor)

Name monitors with the bracket convention so alert notifications immediately identify which layer failed.

Group all Toree monitors into a Vigilmon Status Page for your data platform team and share the URL with notebook platform users so they can self-diagnose before filing support tickets.

Step 5: Alert Routing for Toree Outages

Toree outages block data scientists from running any Spark workloads from notebooks — feature engineering pipelines stall, ad-hoc query results stop, and ML training jobs submitted from notebooks fail to start. On shared JupyterHub deployments, a single Toree kernel crash can affect dozens of concurrent users.

Configure alert channels in Vigilmon:

  1. JupyterHub API monitor → Slack + PagerDuty (P1 — platform login and session creation blocked)
  2. Toree health sidecar → Slack + PagerDuty (P1 — no Toree kernels available)
  3. Spark execution heartbeat → Slack + PagerDuty (P1 — Spark jobs not completing)
  4. Spark master / YARN ResourceManager monitors → Slack (P2 — compute cluster degraded)

Set response time thresholds carefully:

  • Alert at 10000ms on the Toree health sidecar — slow kernelspec enumeration indicates the Jupyter server is under load or the Enterprise Gateway is overloaded
  • Alert at 3000ms on the JupyterHub API — the hub API should respond instantly; slowness here signals proxy or database issues
  • Alert at 2000ms on the Jupyter single-user server root — a slow response on /api indicates notebook server load or memory pressure

Summary

Apache Toree failures are silent and impactful — kernel crashes leave users staring at spinning cells, Spark disconnections cause jobs to hang indefinitely, and Jupyter shows the kernel as "running" throughout. Vigilmon monitors the full path from JupyterHub availability through active Spark job execution.

| Monitor Type | What It Covers | |---|---| | HTTP monitor on JupyterHub /hub/api | Hub availability, session creation | | HTTP monitor on /health/toree | Toree kernelspec availability, running kernels | | Heartbeat monitor | Spark job execution liveness through Toree | | Infrastructure monitors (Spark, YARN) | Compute cluster availability |

Get started free at vigilmon.online — your first Toree monitor takes under two minutes to configure.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles