re_data is the open-source data reliability framework that extends dbt to automatically profile your data warehouse tables — tracking row counts, null rates, schema changes, and distribution shifts across every dbt model in your project, then surfacing anomalies and quality regressions in a dedicated web dashboard. When the re_data dashboard becomes unavailable, data engineers lose visibility into the reliability scores and anomaly history that drive data quality decisions. When the re_data notification system fails, Slack alerts for quality regressions stop flowing and incidents go unnoticed until downstream consumers report bad numbers. When the underlying dbt job scheduler breaks, re_data stops producing fresh metrics and the quality feed goes stale. Vigilmon gives you external visibility into the re_data infrastructure: the web UI, the notification webhook receivers, and the job health endpoints, so you know the moment the data reliability layer itself needs attention.
What You'll Build
- An HTTP monitor on the re_data web UI to detect dashboard availability failures
- A monitor on the re_data API endpoint to catch backend processing failures
- A monitor on your webhook receiver to verify re_data alert delivery is flowing
- A monitor on your dbt job runner endpoint to catch scheduled profiling failures
- An SSL certificate monitor for your re_data domain
- Alerting that distinguishes UI outages from API and alert delivery failures
Prerequisites
- A self-hosted re_data deployment with the web UI exposed via HTTPS
- A webhook receiver endpoint configured to accept re_data anomaly notifications
- Access to your dbt job runner (Airflow, dbt Cloud, or a custom scheduler)
- A free account at vigilmon.online
Step 1: Understand re_data's Service Architecture
re_data operates as a multi-layer data reliability system with several monitorable components:
| Component | Default port | Role |
|---|---|---|
| re_data web UI | 443 (HTTPS) | Dashboard for reviewing reliability scores, anomaly history, and table-level quality metrics |
| re_data API server | 8080 (HTTP/HTTPS) | Serves computed metrics, accepts webhook configuration, integrates with notification channels |
| dbt job runner | Varies | Executes re_data dbt models on schedule to refresh quality metrics |
| Notification dispatcher | Outbound | Delivers anomaly alerts to Slack, email, and custom webhook integrations |
External monitoring focuses on the web UI and API server — these surface the availability of the entire re_data deployment and are the interfaces your data team uses to review quality regressions and investigate reliability incidents.
Step 2: Monitor the re_data Web UI
The re_data web dashboard is the primary interface for reviewing table reliability scores, exploring anomaly timelines, comparing metric runs, and identifying which dbt models have quality regressions that require investigation:
curl -I https://your-redata.example.com
# Returns HTTP 200 with HTML when the dashboard is healthy
- Log in to Vigilmon → Add Monitor → HTTP.
- URL:
https://your-redata.example.com(your re_data web UI hostname). - Check interval: 60 seconds.
- Response timeout: 15 seconds.
- Expected status:
200. - Keyword:
re_data(appears in the page title and HTML content). - Click Save.
This monitor catches:
- re_data UI container crashes or failed Docker restarts
- Web server misconfigurations that return 5xx errors
- Failed upgrades that break the frontend before the API is restored
- DNS resolution failures for your re_data domain
Alert sensitivity: Set to trigger after 1 consecutive failure. When the re_data dashboard is unavailable, data engineers cannot review reliability timelines, compare metric runs, or investigate active anomalies — leaving the team without the quality signal they depend on to make release decisions.
Step 3: Monitor the re_data API Server
The re_data API server stores and serves computed reliability metrics — row counts, null rates, schema drift signals, and anomaly flags for every profiled dbt model. When the API degrades, the dashboard cannot load metric history, anomaly comparisons fail, and any CI/CD integrations that query re_data for a quality gate cannot get a decision:
curl https://your-redata.example.com/api/v1/health
# Returns: {"status": "ok"} when the API backend is operational
- Add Monitor → HTTP.
- URL:
https://your-redata.example.com/api/v1/health. - Check interval: 60 seconds.
- Response timeout: 10 seconds.
- Expected status:
200. - Keyword:
ok(present in healthy API responses). - Label:
re_data API health. - Click Save.
Why monitor the API separately from the UI? The re_data web UI may continue to render cached metric snapshots even when the API is returning errors for new requests. A failing API means the dashboard shows stale reliability scores, anomaly alerts cannot be dispatched for new regressions detected in the latest dbt run, and CI quality gates cannot query re_data for a pass/fail verdict on the current pipeline.
Step 4: Monitor Your Webhook Receiver
re_data delivers anomaly notifications and quality regression alerts to your on-call tools (Slack, custom webhooks) via outbound HTTP POST requests. If your webhook receiver becomes unavailable or returns errors, notification delivery fails silently — data engineers stop receiving regression alerts and quality incidents go unacknowledged until stakeholders report bad data in dashboards:
curl https://your-webhook-receiver.example.com/health
# Returns: {"status": "ok"} when the receiver is up
- Add Monitor → HTTP.
- URL:
https://your-webhook-receiver.example.com/health. - Check interval: 2 minutes.
- Response timeout: 10 seconds.
- Expected status:
200. - Keyword:
ok. - Label:
re_data webhook receiver. - Click Save.
For Slack incoming webhooks, monitor https://slack.com/api/api.test to confirm the Slack API is reachable for re_data anomaly alert delivery. For teams routing alerts through a custom notification service, ensure the service health endpoint is also monitored so alert delivery gaps are caught before the next dbt profiling run completes.
Step 5: Monitor the dbt Job Runner Endpoint
re_data's quality metrics are only as fresh as the last successful dbt run. When the job scheduler (Airflow, dbt Cloud, or a custom runner) fails to execute the re_data dbt models on schedule, the metric feed goes stale — reliability scores stop updating, new schema changes go undetected, and anomaly detection effectively stops without any visible error in the re_data UI:
curl https://your-airflow.example.com/api/v1/health
# Returns Airflow scheduler health when using Apache Airflow
- Add Monitor → HTTP.
- URL:
https://your-airflow.example.com/api/v1/health(or your job runner's health endpoint). - Check interval: 5 minutes.
- Response timeout: 15 seconds.
- Expected status:
200. - Keyword:
healthy(present in Airflow's health response). - Label:
re_data job runner. - Click Save.
Stale metrics are invisible failures: Unlike an API returning 500 errors, a job runner that silently stops executing re_data profiles produces no visible errors — the dashboard continues to show the last successfully computed metrics as if they were current. External monitoring of the scheduler health is the only way to detect this failure mode before the staleness is noticed by data consumers.
Step 6: Monitor SSL Certificates
The re_data web UI and API are accessed over HTTPS by both data engineers using the dashboard and automated quality gates that call the API. An expired certificate breaks browser access to the reliability dashboard and causes API calls from CI/CD pipelines to fail with TLS errors:
openssl s_client -connect your-redata.example.com:443 2>/dev/null | openssl x509 -noout -dates
- Add Monitor → SSL Certificate.
- Domain:
your-redata.example.com(your re_data deployment hostname). - Alert when expiry is within: 30 days.
- Alert again: 14 days, 7 days, 3 days, 1 day.
- Click Save.
Pipeline quality gate impact: CI/CD pipelines that block on a re_data quality gate call the API on automated schedules without interactive input. When the TLS certificate expires, these calls fail immediately with SSL verification errors. If the pipeline treats the re_data quality gate as non-blocking, jobs continue and data may be promoted to production without a reliability check — undermining the entire purpose of the data quality integration.
Step 7: Configure Alerting
In Vigilmon under Settings → Notifications, configure your alert channels:
| Monitor | Trigger | Action |
|---|---|---|
| Web UI | Non-200 or re_data missing | Check re_data container; review Docker or Kubernetes logs |
| API health | Non-200 or ok missing | API down; dashboard shows stale data; check service logs |
| Webhook receiver | Non-200 or ok missing | Alerts not delivering; verify receiver deployment and logs |
| Job runner | Non-200 or healthy missing | dbt profiles not running; metrics going stale; check scheduler logs |
| SSL certificate | < 30 days to expiry | Renew certificate; verify API clients reconnect after renewal |
Alert after: 1 consecutive failure for the UI and API health monitors. 2 consecutive failures for webhook receiver and job runner monitors to reduce noise from transient restarts during dbt model execution windows.
Common re_data Failure Modes and What Vigilmon Catches
| Scenario | Vigilmon monitor | |---|---| | re_data UI container crash | Dashboard monitor fires; engineers cannot review reliability scores | | API backend failure | API monitor fires; dashboard shows stale metrics; CI gates fail | | Webhook receiver crash | Webhook monitor fires; Slack alerts for regressions stop posting | | dbt job scheduler failure | Job runner monitor fires; metrics go stale without visible error | | TLS certificate expired | SSL monitor alerts at 30-day threshold; browser and API clients fail | | DNS misconfiguration | All monitors fire simultaneously | | Database connection failure | API monitor fires with 5xx errors; no new metrics can be stored | | dbt model compilation error | Job runner may appear healthy while re_data models fail to execute |
Monitoring Data Reliability Itself (Beyond Infrastructure)
Vigilmon monitors re_data's infrastructure endpoints — the web UI, API server, webhook receiver, and job runner. The health of individual dbt model profiles and detected anomalies is a separate concern managed within re_data itself:
- re_data's built-in alerting: Configure notification channels within re_data (Slack, custom webhooks) to route anomaly alerts to on-call teams — this is distinct from monitoring re_data's own availability.
- dbt test coverage: Audit which dbt models have re_data profiling enabled to ensure new models added to your project are automatically included in reliability tracking.
- Run cadence: Establish a schedule that balances freshness against warehouse compute costs — re_data profiles should run frequently enough that anomalies are detected before downstream consumers use the affected data.
Vigilmon catches infrastructure-level failures — when re_data itself is down or not running. Individual anomaly detection coverage and data quality regressions within a healthy re_data deployment require re_data's own dbt model configuration and notification rules.
re_data is the layer that makes your dbt warehouse's reliability observable — when it goes down or stops running, anomaly detection halts, regression alerts stop flowing, and data engineers lose the quality signal they rely on to make safe release decisions. Vigilmon gives you external visibility into re_data's availability that doesn't depend on re_data self-reporting: dashboard uptime, API health, job runner status, and SSL certificate validity, so you know the moment the data reliability layer itself needs attention and can restore quality monitoring before the next pipeline run ships unvalidated data downstream.
Start monitoring re_data in under 5 minutes — register free at vigilmon.online.