Monitoring Databricks with Vigilmon: Cluster Health Checks, Job Heartbeats & Lakehouse Uptime

Databricks clusters start, run jobs, and terminate — but if a job driver crashes at 3 AM or a cluster fails to start, your data team might not find out until a business analyst notices stale dashboards at 9 AM. Databricks has built-in job alerts, but they're scoped to your account. Vigilmon gives you an external, independent monitor that fires alerts even when the Databricks control plane itself has a problem.

This tutorial shows you how to wire Databricks into Vigilmon for cluster availability monitoring, job heartbeats, and smart alert thresholds.

What You'll Build

• A health endpoint that checks Databricks cluster state via the REST API
• A Vigilmon HTTP monitor for cluster availability
• A heartbeat monitor for scheduled Databricks jobs
• A multi-workspace monitoring strategy

Prerequisites

• A Databricks workspace (AWS, Azure, or GCP)
• A Databricks personal access token or service principal token
• Python 3.9+ for the health proxy
• A free account at vigilmon.online

Step 1: Build a Databricks Health Endpoint

Databricks exposes a REST API for cluster state. Run a small proxy service that Vigilmon can reach from the outside.

# health_server.py
import os
import requests
from flask import Flask, jsonify

app = Flask(__name__)

DATABRICKS_HOST = os.environ["DATABRICKS_HOST"]  # e.g. https://adb-xxxx.azuredatabricks.net
DATABRICKS_TOKEN = os.environ["DATABRICKS_TOKEN"]
CLUSTER_ID = os.environ["DATABRICKS_CLUSTER_ID"]  # the cluster to watch

HEADERS = {
    "Authorization": f"Bearer {DATABRICKS_TOKEN}",
    "Content-Type": "application/json",
}

@app.route("/health")
def health():
    checks = {}
    ok = True

    # Check cluster state
    try:
        resp = requests.get(
            f"{DATABRICKS_HOST}/api/2.0/clusters/get",
            headers=HEADERS,
            params={"cluster_id": CLUSTER_ID},
            timeout=10,
        )
        resp.raise_for_status()
        cluster = resp.json()
        state = cluster.get("state", "UNKNOWN")
        checks["cluster_id"] = CLUSTER_ID
        checks["cluster_state"] = state
        checks["cluster_name"] = cluster.get("cluster_name", "")

        # RUNNING or TERMINATED are expected; ERROR/UNKNOWN are failures
        if state not in ("RUNNING", "TERMINATED", "TERMINATING", "PENDING", "RESTARTING"):
            ok = False
            checks["cluster_error"] = f"unexpected state: {state}"
    except Exception as e:
        checks["cluster_error"] = str(e)
        ok = False

    # Check Databricks workspace reachability
    try:
        workspace_resp = requests.get(
            f"{DATABRICKS_HOST}/api/2.0/workspace/list",
            headers=HEADERS,
            params={"path": "/"},
            timeout=5,
        )
        checks["workspace_reachable"] = workspace_resp.status_code == 200
        if workspace_resp.status_code != 200:
            ok = False
    except Exception as e:
        checks["workspace_error"] = str(e)
        ok = False

    status_code = 200 if ok else 503
    return jsonify({"status": "ok" if ok else "degraded", "checks": checks}), status_code

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Deploy this service on a VM or container that has network access to the Databricks REST API, then expose it publicly via a reverse proxy:

pip install flask requests
python health_server.py

Step 2: Configure the Vigilmon HTTP Monitor

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://your-server.example.com/health
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds — the Databricks REST API can be slow under load.
5. Expected status: 200.
6. JSON assertion: path status, expected value ok.

Tip: If your cluster runs interactively and is expected to be TERMINATED outside business hours, configure your health endpoint to return 200 for that state and only alert on ERROR or workspace unreachability.

Step 3: Heartbeat Monitor for Databricks Jobs

Databricks jobs run on a schedule, but if the job runner itself fails to trigger, you won't know. Add a heartbeat ping to your job notebooks or Python tasks.

Notebook (Python cell at the end of your job notebook)

import os
import urllib.request

heartbeat_url = dbutils.secrets.get(scope="vigilmon", key="heartbeat-url")

try:
    urllib.request.urlopen(
        urllib.request.Request(heartbeat_url, method="POST"),
        timeout=5,
    )
    print("Job complete — Vigilmon heartbeat sent")
except Exception as e:
    print(f"Warning: heartbeat failed: {e}")
    # Don't fail the job over a heartbeat failure

Store the heartbeat URL in Databricks Secrets:

databricks secrets create-scope vigilmon
databricks secrets put --scope vigilmon --key heartbeat-url --string-value "https://vigilmon.online/api/heartbeat/<your-id>"

Python wheel task

# tasks/etl_job.py
import os
import requests

def main():
    # ... your ETL logic ...
    run_pipeline()

    # Ping Vigilmon on success
    heartbeat_url = os.environ["VIGILMON_HEARTBEAT_URL"]
    try:
        requests.post(heartbeat_url, timeout=5)
        print("Heartbeat sent")
    except Exception as e:
        print(f"Heartbeat failed (non-fatal): {e}")

if __name__ == "__main__":
    main()

In Vigilmon, create a Heartbeat monitor with a grace period that matches your job schedule plus a buffer:

| Job schedule | Recommended grace period | |---|---| | Every 30 minutes | 35 minutes | | Hourly | 75 minutes | | Nightly (1 AM) | 26 hours |

Step 4: Multi-Workspace Monitoring Strategy

Enterprise Databricks setups often have separate workspaces for dev, staging, and production. Monitor the production workspace most aggressively.

| Workspace | Monitor name | Alert channel | |---|---|---| | Production workspace | Databricks Prod | On-call PagerDuty | | Nightly ETL job | Databricks ETL Heartbeat | Data Engineering Slack | | ML training cluster | Databricks ML | ML Platform Slack |

Create one Vigilmon monitor per workspace or job, and group production monitors under a Status Page that stakeholders can bookmark.

Step 5: Alerting Configuration

| Alert type | Recommended channel | |---|---| | Production cluster down | PagerDuty (immediate) | | ETL job missed | Slack data-engineering channel | | Workspace unreachable | Email + Slack |

Set alert escalation: notify immediately on first failure, re-notify after 5 minutes if still down. Databricks cluster startup typically takes 2–5 minutes, so a 3-minute confirmation window before alerting prevents false positives from normal cluster initialization.

What Vigilmon Catches That Databricks Job Alerts Miss

| Scenario | Databricks Job Alerts | Vigilmon | |---|---|---| | Cluster fails to start | Alert fires for job failure | HTTP monitor catches cluster state | | Scheduled job never triggers | No alert | Heartbeat grace period fires | | Databricks control plane outage | Alerts may not fire | External check — unaffected | | Workspace unreachable from app | No visibility | HTTP monitor catches immediately | | Job succeeds but writes no data | No alert | Custom check logic in health endpoint |

Databricks pipelines are the beating heart of your data platform. When they fail silently, downstream dashboards, ML models, and business decisions break too. External monitoring from Vigilmon gives your team the independent signal it needs.

Start monitoring your Databricks workspace in under 5 minutes — register free at vigilmon.online.