tutorial

Monitoring AWS Deequ with Vigilmon: Data Quality Job Health, Spark Cluster Availability & Metric Store Uptime

How to monitor AWS Deequ open-source data quality checks for Apache Spark with Vigilmon — Spark cluster availability, data quality job health endpoints, metric store API, and SSL certificate validity.

AWS Deequ is the open-source data quality library for Apache Spark, originally developed at Amazon and open-sourced to allow data engineering teams to define, compute, and store data quality constraints at massive scale across petabyte-sized datasets. Deequ runs as Spark jobs that execute constraint verification suites — checking column completeness, uniqueness, distribution ranges, referential integrity, and custom business rules — and stores computed quality metrics in a configurable metrics repository so quality trends can be tracked over time. When the Spark cluster that executes Deequ jobs becomes unavailable, data quality checks stop running entirely and data pipelines promoting unchecked data downstream continue without quality gating. When the metrics repository backing your Deequ constraint store becomes inaccessible, historical quality baselines are lost and anomaly detection against rolling quality windows fails silently. When the job orchestration layer that schedules Deequ verification suites encounters failures, data consumers receive no signal that quality checks have been skipped. Vigilmon gives you external visibility into the infrastructure that Deequ depends on: the Spark cluster management API, the metrics repository endpoint, and the orchestration health surface, so your data quality gating stays operational even when the underlying platform has problems.

What You'll Build

  • An HTTP monitor on the Spark cluster manager to detect job submission failures before Deequ jobs are scheduled
  • A health-check monitor on the Deequ metrics repository (S3, JDBC, or a custom API) to catch metric storage degradation
  • A monitor on the job orchestration service that schedules Deequ verification suites
  • A monitor on your quality alert webhook receiver to verify data quality failure notifications are being delivered
  • An SSL certificate monitor for your Spark cluster management UI
  • Alerting that distinguishes cluster unavailability from metrics repository failures and orchestration breakdowns

Prerequisites

  • A Spark cluster running on EMR, Databricks, GKE, or self-managed infrastructure with an accessible management API or UI
  • Deequ integrated into your Spark pipelines with a configured metrics repository (S3 + Parquet, JDBC, or a custom store)
  • A job orchestration service scheduling Deequ checks (Airflow, Step Functions, Prefect, or equivalent)
  • A free account at vigilmon.online

Step 1: Understand Deequ's Infrastructure Architecture

Deequ is a library rather than a standalone service, which means its availability depends on the health of several underlying infrastructure components:

| Component | Surface | Role | |---|---|---| | Spark cluster manager | REST API / HTTPS UI | Accepts job submissions, manages executors, reports cluster health | | Metrics repository | S3 / JDBC endpoint / custom API | Stores and retrieves computed data quality metrics for trend analysis | | Job orchestration | Airflow / Step Functions / Prefect API | Schedules and triggers Deequ verification suite jobs | | Quality alert receiver | Webhook / SNS / Slack | Receives notifications when constraint verification suites fail | | Spark History Server | HTTPS | Provides job log access for debugging failed quality checks |

External monitoring focuses on the cluster manager API, metrics repository, and orchestration layer — these are the surfaces where Deequ failures first become observable from outside the Spark execution context. Cluster failures prevent job submission entirely; metrics repository failures allow jobs to run but silently discard quality history; orchestration failures mean checks are never scheduled at all.

Step 2: Monitor the Spark Cluster Manager

The Spark cluster manager is the first dependency for every Deequ quality check — without a healthy cluster accepting job submissions, no constraint verification suite can execute regardless of how well Deequ is integrated into your pipeline code:

# EMR cluster master node
curl http://your-emr-master:8088/ws/v1/cluster/info
# Returns JSON with cluster state: STARTED, STOPPED, etc.

# Databricks workspace
curl -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  https://your-workspace.azuredatabricks.net/api/2.0/clusters/list
# Returns cluster list with state fields
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-spark-manager.example.com/health (or the cluster manager REST endpoint for your platform).
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: STARTED (or the healthy state token for your cluster manager response).
  7. Label: Spark cluster manager.
  8. Click Save.

This monitor catches:

  • Cluster manager failures that prevent any Deequ job from being submitted
  • Auto-scaling events that have left the cluster in a degraded state
  • Networking or VPC changes that have cut off the management API
  • EMR cluster terminations that have not been caught by your orchestration layer

Alert sensitivity: Set to trigger after 1 consecutive failure. A cluster that is not accepting jobs means every Deequ verification suite scheduled during the outage window will either fail at submission time or silently not run, depending on how your orchestration handles submission errors.

Step 3: Monitor the Deequ Metrics Repository

The Deequ metrics repository is where computed quality metrics are persisted so constraint checks can be compared against historical baselines and quality trends can be visualized over time. When the repository is unavailable, Deequ jobs that write metric results fail to persist their outputs, and the anomaly detection and trend analysis that depend on historical metrics lose access to their input data:

# If your metrics repository exposes a health API (e.g. a custom service wrapping S3 or JDBC)
curl https://your-metrics-api.example.com/health
# Returns: {"status": "ok", "repository": "reachable"} when the store is accessible

# For JDBC-backed repositories, check the database health endpoint
curl https://your-db-host.example.com/health
# Returns database availability status
  1. Add Monitor → HTTP.
  2. URL: https://your-metrics-repository.example.com/health (your metrics store health endpoint or database status API).
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok (or reachable, depending on your health response format).
  7. Label: Deequ metrics repository.
  8. Click Save.

Why monitor the metrics repository separately from the cluster? A healthy Spark cluster can successfully execute Deequ constraint checks and then fail to write metric outputs to a degraded repository — the Spark job exits without error in some configurations, making the metric loss invisible. Without an independent monitor on the repository, your quality tracking silently gaps out while cluster health dashboards show green.

Step 4: Monitor the Job Orchestration Layer

The orchestration service — whether Airflow, AWS Step Functions, Prefect, or another scheduler — is what actually triggers Deequ verification suites on a schedule. If the orchestration layer fails, no Deequ jobs run regardless of cluster availability, and the absence of quality checks is often invisible to data consumers who assume checks are running:

# Apache Airflow health endpoint
curl https://your-airflow.example.com/health
# Returns: {"metadatabase": {"status": "healthy"}, "scheduler": {"status": "healthy"}}

# Prefect server health
curl https://your-prefect.example.com/api/health
# Returns: {"status": "ok"}
  1. Add Monitor → HTTP.
  2. URL: https://your-orchestration-service.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: healthy (or ok, matching your orchestration platform's health response).
  7. Label: Deequ orchestration (Airflow/Prefect).
  8. Click Save.

Silent failure mode: Orchestration services can enter degraded states where the UI is accessible and the web server responds to health checks, but the scheduler process has died and no DAG runs are being triggered. If your orchestration platform exposes a separate scheduler heartbeat endpoint (Airflow exposes this under /health), monitor the scheduler status field explicitly rather than just the HTTP status code.

Step 5: Monitor Your Quality Alert Webhook Receiver

Deequ verification failures — when constraint checks determine that a column's null rate exceeds the defined threshold, or that uniqueness has dropped below acceptable levels — generate notifications that flow through alert routing to your data quality dashboards, incident channels, or pipeline gating systems. If the webhook receiver that accepts these notifications is down, your team gets no signal when data quality degrades:

curl https://your-quality-alerts.example.com/health
# Returns: {"status": "ok"} when the alert receiver is operational
  1. Add Monitor → HTTP.
  2. URL: https://your-quality-alerts.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Deequ quality alert receiver.
  8. Click Save.

For Slack channels that receive Deequ quality failure notifications, verify https://slack.com/api/api.test to confirm the Slack API is reachable from your alerting infrastructure. For PagerDuty-routed quality escalations, monitor https://api.pagerduty.com/ability to verify the PagerDuty Events API is accessible.

Step 6: Monitor SSL Certificates

Spark cluster management UIs, Airflow dashboards, Prefect servers, and custom metrics API endpoints accessed by data engineers and automated jobs all use HTTPS. An expired certificate locks out engineers from the Spark UI during an active incident (exactly when they need job logs most) and breaks automated metric push scripts that connect to the repository API over TLS:

openssl s_client -connect your-spark-ui.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: your-spark-ui.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for your Airflow or Prefect domain and your metrics repository API domain if it exposes its own TLS endpoint. Self-managed Spark and Airflow deployments often use internal certificate authorities whose renewal schedules are not tracked in commercial certificate monitoring services — Vigilmon's pre-expiry alerts close that gap.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Spark cluster manager | Non-200 or healthy state keyword missing | Cluster unavailable; all Deequ job submissions will fail | | Metrics repository | Non-200 or ok missing | Metric writes failing; quality history gapping silently | | Orchestration service | Non-200 or healthy missing | No Deequ jobs will be scheduled; check scheduler process | | Quality alert receiver | Non-200 or ok missing | Quality failure notifications not being delivered | | SSL certificate | < 30 days to expiry | Renew certificate; engineers will lose Spark UI access on expiry |

Alert after: 1 consecutive failure for cluster manager and orchestration monitors. 2 consecutive failures for the metrics repository and webhook receiver to reduce noise from brief transient timeouts during repository scaling events.

Common Deequ Infrastructure Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Spark cluster auto-terminated or scaled to zero | Cluster monitor fires; all Deequ job submissions fail | | Metrics repository S3 outage | Repository monitor fires; quality metric history gaps silently | | Airflow scheduler process death | Orchestration monitor fires; no Deequ suites scheduled | | Webhook receiver crash | Alert receiver monitor fires; quality failure notifications lost | | TLS certificate expired on Spark UI | SSL monitor alerts at 30-day threshold; engineers lose log access | | VPC/networking change cuts off cluster API | Cluster monitor fires with timeout | | JDBC database for metrics store unreachable | Repository monitor fires; metric persistence fails | | Prefect server out of memory | Orchestration monitor fires with 5xx or timeout | | Deequ version upgrade breaks job submission format | Cluster monitor may fire if job submission endpoint changes | | EMR cluster node loss leaves cluster degraded | Cluster monitor fires with degraded state token |

Monitoring Data Quality Itself (Beyond Infrastructure)

Vigilmon monitors the infrastructure that Deequ depends on — the Spark cluster, metrics repository, and orchestration layer. The actual data quality constraint results and metric trends that Deequ produces require domain-specific tooling:

  • Deequ's built-in anomaly detection: Configure AnomalyDetectionStrategy in your Deequ verification suites to compare computed metrics against historical baselines stored in the metrics repository. This detects data quality regressions that are invisible to infrastructure monitors.
  • Metrics repository trend analysis: Build dashboards over the Deequ metrics repository to visualize quality metric trends — completeness rates, uniqueness ratios, distribution statistics — across pipeline runs over time. Infrastructure health does not tell you that null rates are rising.
  • Constraint suite coverage audits: Regularly review which tables and columns are covered by Deequ constraint suites and which are not. Infrastructure monitoring cannot tell you that a new table was added to a pipeline but has no corresponding quality checks defined.

Vigilmon catches infrastructure failures — when the Spark cluster is down, the metrics store is unreachable, or the orchestration layer has stopped scheduling jobs. The quality of the constraint suites themselves, the coverage of Deequ checks across your data assets, and the interpretation of metric trends within a healthy Deequ deployment require domain-specific data quality governance processes.

AWS Deequ brings data quality constraint verification to Spark-scale datasets, but the infrastructure it runs on — clusters, metrics stores, and orchestrators — can fail in ways that silently disable quality gating without surfacing visible errors. Vigilmon gives your data engineering team external visibility into that infrastructure: cluster availability, metrics repository health, orchestration status, and SSL certificate validity, so you know the moment the quality checking layer needs attention and can restore constraint verification before unchecked data propagates downstream.

Start monitoring your Deequ infrastructure in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles