tutorial

Monitoring Delta Live Tables with Vigilmon: Pipeline Health, Event Log API & Databricks Workspace Availability

How to monitor Databricks Delta Live Tables declarative ETL pipelines with Vigilmon — workspace availability, pipeline event log API, cluster health endpoints, and SSL certificate validity.

Delta Live Tables is Databricks' declarative ETL framework for building reliable, maintainable data pipelines that automatically manage dependencies, handle retries, and enforce data quality expectations across streaming and batch workloads. DLT pipelines are defined as a graph of Python or SQL transformations, and the Databricks runtime orchestrates their execution — scheduling updates, managing the underlying compute clusters, tracking lineage, and persisting quality metrics to an event log that records every pipeline run result. When the Databricks workspace becomes unavailable, all DLT pipeline updates halt and no new data flows through any layer of your medallion architecture. When the REST API that exposes pipeline status is degraded, your monitoring and orchestration tools lose visibility into which pipelines are running, which have failed, and which are waiting for upstream dependencies. When the event log backing DLT's quality tracking becomes inaccessible, pipeline operators cannot audit why a quality constraint caused a row to be quarantined to the expectations table. Vigilmon gives you external HTTP visibility into the Databricks surfaces that DLT depends on — the workspace API, the pipeline REST endpoint, and the cluster management layer — so you catch infrastructure failures before they silently stall your data lakehouse.

What You'll Build

  • An HTTP monitor on the Databricks workspace REST API to detect platform outages before they halt DLT pipeline updates
  • A monitor on the DLT Pipelines API endpoint to catch degradation in pipeline status reporting and event log access
  • A monitor on the Databricks cluster policy endpoint to detect compute provisioning failures that prevent DLT clusters from starting
  • A monitor on your pipeline failure webhook receiver to verify DLT quality expectation failure notifications are being delivered
  • An SSL certificate monitor for your Databricks workspace domain
  • Alerting that distinguishes workspace API outages from pipeline compute failures and notification delivery breakdowns

Prerequisites

  • A Databricks workspace with one or more Delta Live Tables pipelines configured
  • A Databricks personal access token or service principal token for API access
  • An external health endpoint or webhook receiver that captures DLT pipeline failure events
  • A free account at vigilmon.online

Step 1: Understand Delta Live Tables' Infrastructure Architecture

DLT pipelines run on Databricks-managed infrastructure, but several surfaces are externally monitorable via the Databricks REST API:

| Component | Surface | Role | |---|---|---| | Databricks workspace API | HTTPS REST | Central control plane; manages pipelines, clusters, jobs | | DLT Pipelines API | /api/2.0/pipelines | Reports pipeline state, update history, event log entries | | Cluster provisioning | /api/2.0/clusters | Manages compute clusters that DLT updates run on | | Pipeline event webhook | Webhook / PagerDuty | Delivers pipeline failure and quality expectation violation alerts | | Databricks workspace UI | HTTPS | Engineer access during incidents; cluster logs, pipeline graphs |

External monitoring focuses on the workspace API availability, pipeline status endpoint, and cluster provisioning surface — these are where DLT failures become externally observable before they propagate into data freshness SLAs. A workspace API outage halts all pipeline updates simultaneously; a cluster provisioning failure silently fails individual pipeline updates; a degraded pipelines API makes it impossible to determine which pipelines are in a failed state from any external orchestration tool.

Step 2: Monitor the Databricks Workspace API

The Databricks workspace REST API is the control plane for all DLT operations. Every pipeline trigger, status query, cluster allocation request, and event log read flows through this API. When it becomes unavailable or returns errors, all DLT pipelines that rely on API-triggered updates stop receiving new data:

# Test workspace API availability
curl -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  https://your-workspace.azuredatabricks.net/api/2.0/clusters/list
# Returns: {"clusters": [...]} when the workspace API is healthy

# Workspace health surface (available on some deployment types)
curl https://your-workspace.azuredatabricks.net/api/2.0/workspace/list?path=/
# Returns directory listing; a 200 confirms the workspace API is responding
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-workspace.azuredatabricks.net/api/2.0/clusters/list.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: cluster_id (present in every non-empty cluster list response).
  7. Label: Databricks workspace API.
  8. Click Save.

This monitor catches:

  • Full Databricks workspace outages that halt all DLT pipeline updates
  • Regional Azure, AWS, or GCP incidents affecting the Databricks control plane
  • Authentication service failures that cause all API requests to return 401
  • Network path changes that have silently cut off your Databricks workspace from your monitoring infrastructure

Alert sensitivity: Set to trigger after 1 consecutive failure. A workspace API outage means no DLT pipeline can receive a triggered update or report its current status to any external system.

Step 3: Monitor the DLT Pipelines API Endpoint

The Pipelines API is the specific surface that exposes DLT pipeline state — running, failed, idle, or waiting — and provides access to the event log that records every pipeline update outcome, quality expectation result, and row count statistic. External orchestration tools, data observability platforms, and incident response runbooks all depend on this endpoint to understand which pipelines need attention:

# List all pipelines and their current state
curl -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  https://your-workspace.azuredatabricks.net/api/2.0/pipelines
# Returns: {"statuses": [{"pipeline_id": "...", "state": "RUNNING", ...}]}

# Check a specific pipeline
curl -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  https://your-workspace.azuredatabricks.net/api/2.0/pipelines/YOUR_PIPELINE_ID
# Returns pipeline detail including current state and last update result
  1. Add Monitor → HTTP.
  2. URL: https://your-workspace.azuredatabricks.net/api/2.0/pipelines.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: statuses (present in every successful pipeline list response).
  7. Label: DLT Pipelines API.
  8. Click Save.

Why monitor the Pipelines API separately from the workspace API? Databricks occasionally deploys partial degradation where the workspace UI and some API endpoints remain available while specific API resources return errors. The pipelines endpoint can be degraded while the clusters endpoint is healthy, leaving pipeline status invisible to external tools even though the workspace appears healthy.

Step 4: Monitor the Cluster Provisioning API

Every DLT pipeline update requires Databricks to provision or reuse a compute cluster for the pipeline's execution. When the cluster provisioning API is degraded, DLT pipeline updates that need to spin up new clusters fail with compute allocation errors — often silently, from the perspective of the pipeline's consumers who expected fresh data:

# Check cluster provisioning health by listing clusters
curl -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  https://your-workspace.azuredatabricks.net/api/2.0/clusters/list
# Returns cluster list; if provisioning API is healthy, the list endpoint responds

# For specific cluster policy verification (enforced on DLT compute)
curl -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  https://your-workspace.azuredatabricks.net/api/2.0/policies/clusters/list
# Returns cluster policies; confirms the policy API layer is healthy
  1. Add Monitor → HTTP.
  2. URL: https://your-workspace.azuredatabricks.net/api/2.0/policies/clusters/list.
  3. Check interval: 3 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: policies.
  7. Label: Databricks cluster provisioning API.
  8. Click Save.

This monitor detects provisioning-layer failures that would prevent DLT clusters from starting, without requiring you to wait for a pipeline update to fail and propagate an error through your alerting chain.

Step 5: Monitor Your Pipeline Failure Webhook Receiver

Delta Live Tables can emit events — pipeline update completions, quality expectation failures, and row quarantine events — to external webhooks that feed your incident management, data observability, or notification systems. If the webhook receiver is down when a DLT quality expectation drops rows into the expectations quarantine table, your team receives no signal and the data quality regression propagates silently downstream:

curl https://your-dlt-webhook.example.com/health
# Returns: {"status": "ok"} when the webhook receiver is operational
  1. Add Monitor → HTTP.
  2. URL: https://your-dlt-webhook.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: DLT pipeline failure webhook.
  8. Click Save.

For Slack-based DLT failure notifications, verify https://slack.com/api/api.test to confirm Slack API reachability from your notification infrastructure. For PagerDuty-routed DLT incidents, monitor https://api.pagerduty.com/ability to verify the Events API is accessible.

Step 6: Monitor SSL Certificates

Databricks workspace URLs are accessed by data engineers during incident response (pipeline graphs, event logs, cluster logs) and by automated tools that connect to the Databricks API over TLS. An expired certificate blocks both human access and automated integrations at the worst possible time — when a pipeline has failed and the team needs to inspect the event log:

openssl s_client -connect your-workspace.azuredatabricks.net:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: your-workspace.azuredatabricks.net.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for any custom domains that front your Databricks workspace or internal tooling that integrates with the DLT event log API. Databricks-managed certificates on *.azuredatabricks.net are renewed by Databricks, but custom domains and internal proxies require your team's certificate management.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Databricks workspace API | Non-200 or cluster_id missing | All DLT pipelines halted; escalate to platform team | | DLT Pipelines API | Non-200 or statuses missing | Pipeline status invisible; orchestration tools blind | | Cluster provisioning API | Non-200 or policies missing | DLT compute allocation failing; new pipeline updates will fail | | Pipeline failure webhook | Non-200 or ok missing | Quality expectation failures not being delivered | | SSL certificate | < 30 days to expiry | Renew certificate; engineers lose workspace access on expiry |

Alert after: 1 consecutive failure for the workspace API and pipelines API. 2 consecutive failures for the cluster provisioning API and webhook receiver to reduce noise from brief transient timeouts during Databricks platform maintenance windows.

Common Delta Live Tables Infrastructure Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Databricks regional outage | Workspace API monitor fires; all pipeline updates halt | | Pipelines API partial degradation | Pipelines API monitor fires; external tools lose status visibility | | Cluster quota exhaustion | Provisioning API may respond but cluster allocation requests fail | | Webhook receiver crash | Webhook monitor fires; quality expectation failures not delivered | | TLS certificate expired on workspace | SSL monitor alerts at 30-day threshold | | Control plane maintenance window | Workspace API monitor fires during window | | Authentication service degradation | Workspace API returns 401; monitor fires | | Network path to Databricks API blocked | All API monitors fire with connection timeout | | DLT streaming pipeline cluster recycled | New cluster allocation fails if provisioning API degraded | | Event log storage backend degraded | Pipeline API may return 500; pipelines API monitor fires |

Monitoring Pipeline Data Quality (Beyond Infrastructure)

Vigilmon monitors the infrastructure that DLT depends on — the Databricks workspace API, the pipeline status surface, and the cluster provisioning layer. The actual quality expectations, row counts, and data freshness that DLT manages internally require additional observability tooling:

  • DLT expectations monitoring: Use the DLT event log (system.lakeflow.pipeline_events) to track quality expectation pass/fail rates, rows dropped to the quarantine table, and expectation violation trends over time. Infrastructure health does not tell you that an expectation threshold is being violated by upstream data drift.
  • Pipeline update latency tracking: Monitor the time between scheduled pipeline triggers and the completion of the RUNNING → COMPLETED transition using the Pipelines API update history. A healthy workspace API does not guarantee that individual pipelines are completing within their SLA windows.
  • Data freshness SLA monitoring: Track the timestamp of the most recently processed record in your Gold layer tables. Infrastructure monitors cannot detect a DLT pipeline that is running but processing data that is hours old due to upstream volume changes or source connector latency.

Vigilmon catches infrastructure failures — when the Databricks workspace is unavailable, the pipelines API is degraded, or the cluster provisioning layer is rejecting allocation requests. The quality of the transformations, the coverage of expectations across your DLT pipeline graph, and the data freshness of your medallion architecture's output tables require domain-specific data observability practices layered on top of infrastructure monitoring.

Delta Live Tables brings declarative, self-managing ETL to the Databricks lakehouse, but the Databricks infrastructure it runs on — the workspace API, the pipelines endpoint, and the cluster provisioning layer — can fail in ways that silently halt data flow without surfacing visible pipeline errors to consumers. Vigilmon gives your data engineering team external visibility into that infrastructure: workspace API availability, pipeline status reachability, cluster provisioning health, and SSL certificate validity, so you know the moment the DLT platform needs attention and can restore pipeline updates before data freshness SLAs are breached.

Start monitoring your Delta Live Tables infrastructure in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles