tutorial

Monitoring Unity Catalog with Vigilmon: Databricks Open Data Governance Uptime, Metastore API Health & Catalog Service Availability

How to monitor Unity Catalog Databricks open data governance and catalog solution with Vigilmon — catalog API availability, metastore health, REST API uptime, and SSL certificate validity.

Unity Catalog is Databricks' open data governance and catalog solution — providing fine-grained access control, data lineage, audit logging, and unified metadata management across all Databricks workspaces, Delta Lake tables, ML models, and external data sources so organizations can govern data assets once and enforce those policies consistently across every workspace and cloud. When the Unity Catalog metastore becomes unavailable, Databricks notebooks and jobs cannot resolve table references or validate permissions, SQL warehouse queries against governed tables fail with catalog resolution errors, and the access control policies that restrict sensitive data from unauthorized workloads stop being enforced. When the Unity Catalog REST API degrades, Terraform providers cannot sync governance configurations, dbt models that reference Unity Catalog schemas fail at compile time, and custom data mesh tooling that queries lineage and permissions programmatically loses access to catalog metadata. Vigilmon gives you external visibility into Unity Catalog's critical infrastructure: the catalog REST API, metastore health endpoint, and any webhook delivery receivers, so you know the moment the data governance layer needs attention.

What You'll Build

  • An HTTP monitor on the Unity Catalog REST API to detect availability failures
  • A health-check monitor on the Unity Catalog metastore service to catch backend degradation
  • A monitor on the catalog data lineage service to detect lineage tracking failures
  • A monitor on your webhook receiver to verify Unity Catalog audit event delivery is flowing
  • An SSL certificate monitor for your Unity Catalog deployment
  • Alerting that distinguishes catalog API outages from metastore failures and lineage service issues

Prerequisites

  • A Unity Catalog deployment (Databricks-managed or open-source self-hosted Unity Catalog server)
  • The Unity Catalog REST API accessible via HTTPS
  • A webhook receiver endpoint configured to accept Unity Catalog audit log or event notifications
  • A free account at vigilmon.online

Step 1: Understand Unity Catalog's Service Architecture

Unity Catalog has two primary deployment modes — Databricks-managed (embedded in the Databricks platform) and open-source self-hosted. Both expose similar API surfaces:

| Component | Default port | Role | |---|---|---| | Catalog REST API | 443 (HTTPS) | Serves table metadata, schema definitions, access policies, and lineage queries | | Metastore service | 8080 or 443 (HTTPS) | Central metadata store managing catalog, schema, and table hierarchies | | Permission enforcement layer | Internal | Validates access control policies on every table and schema access request | | Lineage service | Internal | Records and queries data lineage relationships between tables, columns, and notebooks | | Audit log dispatcher | Outbound | Delivers access audit events to external SIEM, data lake, or monitoring systems |

External monitoring focuses on the catalog REST API, metastore service, and lineage API — these are the surfaces that Databricks workspaces, Terraform governance automation, dbt projects, and data mesh tooling interact with continuously. Metastore service failures are particularly impactful because they block all table resolution across every workspace connected to the affected metastore.

Step 2: Monitor the Unity Catalog REST API

The Unity Catalog REST API is the primary programmatic interface for managing and querying the data catalog — creating and updating schemas, granting and revoking table permissions, querying data lineage graphs, listing catalog assets, and validating access policies before data is used in downstream systems:

curl -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  https://your-workspace.azuredatabricks.net/api/2.1/unity-catalog/metastores
# Returns metastore list when the catalog API is healthy
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-workspace.azuredatabricks.net/api/2.1/unity-catalog/metastores (your Databricks workspace API URL).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: metastores (present in healthy catalog API responses).
  7. Click Save.

This monitor catches:

  • Unity Catalog API server failures or Databricks control plane outages
  • API rate limiting or authentication failures that block programmatic catalog access
  • DNS resolution failures for the Databricks workspace hostname
  • Databricks platform upgrades that temporarily degrade catalog API availability

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Unity Catalog API is unavailable, Terraform governance automation cannot apply access control changes, dbt projects cannot compile against catalog schemas, and data mesh tooling that enforces data product contracts through catalog permissions cannot validate access.

Step 3: Monitor the Unity Catalog Metastore Health

For open-source self-hosted Unity Catalog deployments, the metastore server exposes a dedicated health endpoint that reports whether the catalog service is ready to serve metadata requests. When the metastore is degraded, all table resolution across connected Databricks workspaces fails — notebooks cannot SELECT from Delta tables, SQL warehouses cannot execute queries against governed schemas, and any pipeline that references catalog paths with three-part names (catalog.schema.table) returns resolution errors:

curl http://your-unity-catalog-server:8080/api/2.1/unity-catalog/health
# Returns: {"status": "alive"} when the metastore is healthy
  1. Add Monitor → HTTP.
  2. URL: https://your-unity-catalog-server/api/2.1/unity-catalog/health (your Unity Catalog server URL).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: alive (present in healthy metastore responses).
  7. Label: Unity Catalog metastore health.
  8. Click Save.

Why monitor the metastore separately from the catalog API? The Databricks workspace UI and SQL warehouse endpoints may continue responding even when the Unity Catalog metastore is unavailable — workspaces may serve cached schema information for a brief period. A failing metastore means new table permission grants are not enforced, lineage is not recorded for running jobs, and any workspace operation that requires schema resolution against the current catalog state fails with confusing error messages.

Step 4: Monitor the Data Lineage Service

Unity Catalog's lineage service records the upstream and downstream relationships between tables, columns, notebooks, and SQL queries — providing the audit trail that data governance teams depend on to track data flows, assess the blast radius of schema changes, and generate the lineage evidence required for regulatory compliance. When lineage recording fails, the audit trail has gaps that may only be discovered during a compliance review or incident investigation:

curl -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  "https://your-workspace.azuredatabricks.net/api/2.0/lineage-tracking/table-lineage?table_name=catalog.schema.table"
# Returns lineage graph when the lineage service is active
  1. Add Monitor → HTTP.
  2. URL: https://your-workspace.azuredatabricks.net/api/2.0/lineage-tracking/table-lineage?table_name=main.default.health_check.
  3. Check interval: 5 minutes.
  4. Response timeout: 20 seconds.
  5. Expected status: 200.
  6. Keyword: upstream_tables (present in lineage API responses).
  7. Label: Unity Catalog lineage service.
  8. Click Save.

Why lineage service monitoring matters: Lineage gaps are often discovered at the worst possible time — during a compliance audit that requires a complete data flow trace from source to report, or during an incident investigation where the team needs to know which downstream tables were affected by a schema change. Proactive monitoring of the lineage service catches recording failures immediately, before audit gaps accumulate.

Step 5: Monitor Your Audit Log Webhook Receiver

Unity Catalog delivers audit log events — table access events, permission changes, schema modifications, and governance decisions — to external systems for SIEM integration, compliance archiving, and security monitoring. If your audit log receiver becomes unavailable, governance events are not recorded in your external compliance store: permission change events are lost, access audit trails have gaps, and the security monitoring pipeline that detects unauthorized data access loses its event feed:

curl https://your-audit-receiver.example.com/health
# Returns: {"status": "ok"} when the receiver is up
  1. Add Monitor → HTTP.
  2. URL: https://your-audit-receiver.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Unity Catalog audit log receiver.
  8. Click Save.

For Splunk or Elasticsearch-based SIEM integrations receiving Unity Catalog audit events, monitor the ingest endpoint health for those systems as well. For Databricks deployments integrated with Azure Monitor, AWS CloudTrail, or Google Cloud Audit Logs, verify the relevant logging service health endpoints to ensure the complete audit pipeline remains intact.

Step 6: Monitor SSL Certificates

Unity Catalog's REST API and metastore service are accessed over HTTPS by Databricks workspaces, Terraform providers, dbt projects, and compliance tooling. An expired certificate blocks all API clients from connecting, causes Terraform governance automation to fail with TLS errors, and prevents dbt from compiling projects against catalog schemas — at exactly the moment when the data platform team may need to make emergency governance changes:

openssl s_client -connect your-workspace.azuredatabricks.net:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: your-workspace.azuredatabricks.net (or your self-hosted Unity Catalog server domain).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Self-hosted Unity Catalog consideration: For open-source self-hosted Unity Catalog deployments, TLS certificates are the customer's responsibility. Databricks-managed Unity Catalog certificates are renewed by Databricks, but the Databricks workspace URL used to access the catalog API may still have a separately managed certificate if served through a custom domain or reverse proxy. Vigilmon's 30-day pre-expiry alert provides the lead time needed to renew certificates before they block workspace operations.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Catalog REST API | Non-200 or metastores missing | Catalog API down; Terraform, dbt, and programmatic access failing | | Metastore health | Non-200 or alive missing | Table resolution failing; all workspace queries against governed tables blocked | | Lineage service | Non-200 or upstream_tables missing | Lineage recording stopped; audit trail gaps accumulating | | Audit log receiver | Non-200 or ok missing | Compliance event delivery broken; SIEM and audit archive missing events | | SSL certificate | < 30 days to expiry | Renew certificate; coordinate with infra or Databricks team |

Alert after: 1 consecutive failure for the catalog API and metastore health monitors. 2 consecutive failures for the lineage service and audit log receiver monitors to reduce noise from brief transient timeouts.

Common Unity Catalog Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Catalog API unavailability | API monitor fires; Terraform, dbt, and programmatic catalog access fail | | Metastore service failure | Metastore monitor fires; table resolution fails across all connected workspaces | | Lineage service failure | Lineage monitor fires; audit trail gaps accumulate silently | | Audit log receiver crash | Webhook monitor fires; compliance events stop reaching SIEM | | TLS certificate expired | SSL monitor alerts at 30-day threshold; all API clients fail | | DNS misconfiguration | All monitors fire simultaneously | | Databricks control plane outage | Catalog API and metastore monitors fire | | Permission enforcement failure | Metastore monitor may fire; unauthorized access risk during degradation | | Schema sync failure | Lineage and API monitors may fire; schema inconsistencies accumulate | | Failed Unity Catalog upgrade | API and metastore monitors fire; table access may be blocked during rollback |

Monitoring Data Governance Quality Itself (Beyond Infrastructure)

Vigilmon monitors Unity Catalog's infrastructure endpoints — the catalog REST API, metastore health, lineage service, and your audit log receiver. The quality of governance coverage — which tables have access controls applied, whether column-level security policies are configured correctly, and whether lineage is being captured for all pipelines — is a separate concern managed within Unity Catalog itself:

  • Access control audits: Regularly review Unity Catalog's privilege grants to ensure that sensitive schemas and tables have appropriate access controls and that no over-permissioned roles exist. Infrastructure availability monitoring cannot detect misconfigured governance.
  • Lineage coverage reviews: Audit which tables and pipelines have active lineage recording enabled. Pipelines that bypass Unity Catalog (using external locations directly) will not have lineage captured even when the lineage service is healthy.
  • Schema evolution governance: Establish a review process for schema changes to governed tables. Unity Catalog can enforce column masking and row filters, but the policies themselves must be maintained as schemas evolve.

Vigilmon catches infrastructure-level failures — when Unity Catalog's APIs are down or the metastore has stopped serving requests. Data governance quality, access control correctness, and lineage coverage within a healthy Unity Catalog deployment require ongoing governance configuration and review.

Unity Catalog is the governance layer that enforces who can see what data across every Databricks workspace — when the metastore fails, table resolution breaks across all workspaces, access control enforcement stops, and the lineage audit trail that regulatory compliance depends on has gaps. Vigilmon gives you external visibility into Unity Catalog's availability that doesn't depend on Databricks self-reporting: catalog API uptime, metastore health, lineage service status, and SSL certificate validity, so you know the moment the data governance infrastructure needs attention and can restore catalog operations before workspace disruption or compliance exposure.

Start monitoring Unity Catalog in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles