tutorial

Monitoring Sifflet with Vigilmon: End-to-End Data Observability Platform Health, API Availability & Alert Delivery

How to monitor Sifflet data observability platform with Vigilmon — application uptime, API availability, alert webhook delivery, lineage service health, and SSL certificate validity.

Sifflet is the end-to-end data observability platform that combines data quality monitoring, automated lineage tracking, and incident management in a single platform — connecting to your data warehouse, dbt project, Airflow DAGs, and BI tools to detect anomalies, map column-level lineage, and route quality incidents to the right owners automatically. When the Sifflet platform becomes unavailable, data teams lose real-time visibility into pipeline health, quality monitors stop updating, and the incident routing system that connects detected anomalies to on-call engineers stops functioning. When the Sifflet API is degraded, integrations with dbt Cloud, Snowflake, and Airflow cannot sync metadata, and programmatic access to quality results and lineage data fails. Vigilmon gives you external visibility into the Sifflet infrastructure: the platform dashboard, the API endpoints, and your alert webhook receivers, so you know the moment the data observability platform itself needs attention.

What You'll Build

  • An HTTP monitor on the Sifflet platform to detect application availability failures
  • A health-check monitor on the Sifflet API to catch backend degradation
  • A monitor on your webhook receiver to verify Sifflet alert delivery is flowing
  • A monitor on the Sifflet lineage service to catch dependency graph failures
  • An SSL certificate monitor for your Sifflet domain
  • Alerting that distinguishes platform outages from API and alert delivery failures

Prerequisites

  • A Sifflet account (cloud-hosted SaaS or self-managed) with the application accessible via HTTPS
  • API credentials for Sifflet if using REST API integrations
  • A webhook receiver endpoint configured to accept Sifflet anomaly and incident alerts
  • A free account at vigilmon.online

Step 1: Understand Sifflet's Service Architecture

Sifflet operates as a multi-layer data observability platform with several monitorable components:

| Component | Default port | Role | |---|---|---| | Sifflet platform dashboard | 443 (HTTPS) | Web UI for configuring monitors, reviewing detected anomalies, managing incidents | | Sifflet REST API | 443 (HTTPS) | Programmatic access to quality results, lineage data, monitor configuration | | Lineage service | Internal | Builds and serves column-level lineage graphs from connected data sources | | Alert dispatcher | Outbound | Delivers anomaly and incident alerts to Slack, PagerDuty, email, and custom webhooks | | Connector sync engine | Outbound | Crawls Snowflake, BigQuery, dbt, Airflow, and BI tools to refresh metadata |

External monitoring focuses on the platform dashboard and REST API — these surface the availability of the entire Sifflet deployment and are the interfaces your data engineers and pipeline integrations interact with directly.

Step 2: Monitor the Sifflet Platform Dashboard

The Sifflet platform dashboard is the primary interface for reviewing detected anomalies, configuring quality monitors, investigating data incidents through the lineage graph, and managing the escalation of quality issues to data owners:

curl -I https://app.siffletdata.com
# Returns HTTP 200 with HTML when the Sifflet application is healthy
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://app.siffletdata.com (or your self-managed Sifflet hostname).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Sifflet (appears in the page title and application HTML).
  7. Click Save.

This monitor catches:

  • Sifflet platform deployment failures or infrastructure outages
  • Application load balancer misconfigurations returning 5xx errors
  • DNS resolution failures for your Sifflet workspace domain
  • Failed Sifflet upgrades that break the dashboard before the API is restored

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Sifflet dashboard is unavailable, data engineers cannot investigate active anomalies, data owners cannot review and acknowledge quality incidents, and the incident management workflows that route quality issues to resolution teams stop functioning entirely.

Step 3: Monitor the Sifflet REST API

The Sifflet REST API is the backbone of all pipeline integrations — it delivers quality check results to dbt Cloud, exposes the lineage graph for downstream tools, serves monitor configuration for automated management, and powers the incident management workflows that route quality alerts to the right owners. When the API degrades, automated quality gates embedded in pipelines cannot query Sifflet for anomaly status, lineage queries return incomplete graphs, and programmatic monitor management fails:

curl https://app.siffletdata.com/api/v1/health \
  -H "Authorization: Bearer YOUR_API_TOKEN"
# Returns: {"status": "healthy"} when the API backend is operational
  1. Add Monitor → HTTP.
  2. URL: https://app.siffletdata.com/api/v1/health.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: healthy (present in healthy API responses).
  7. Label: Sifflet API health.
  8. Click Save.

Why monitor the API separately from the dashboard? The Sifflet dashboard may continue to render previously loaded anomaly data even when the API cannot process new requests. A failing API means dbt Cloud cannot fetch quality check results for the current pipeline run, Airflow DAGs that check Sifflet for anomaly status cannot get a verdict, and engineers investigating an incident cannot pull fresh lineage or quality data from the API.

Step 4: Monitor Your Webhook Receiver

Sifflet delivers anomaly detection alerts, data quality incident notifications, and escalation triggers to your on-call tools (Slack, PagerDuty, custom webhooks) via outbound HTTP POST requests. If your webhook receiver becomes unavailable or returns errors, notification delivery fails silently — on-call engineers stop receiving quality alerts and data incidents go unacknowledged until analysts report bad data in downstream dashboards:

curl https://your-webhook-receiver.example.com/health
# Returns: {"status": "ok"} when the receiver is up
  1. Add Monitor → HTTP.
  2. URL: https://your-webhook-receiver.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Sifflet alert webhook receiver.
  8. Click Save.

For Slack incoming webhooks where Sifflet posts anomaly alerts, verify https://slack.com/api/api.test to confirm the Slack API is reachable. For PagerDuty integrations that create incidents from Sifflet quality failures, monitor https://status.pagerduty.com/api/v2/status.json to catch PagerDuty degradation that would block incident creation from Sifflet anomaly detections.

Step 5: Monitor the Sifflet Lineage Service

Sifflet's column-level lineage graph is one of its core value propositions — when an anomaly is detected, the lineage graph shows exactly which upstream sources contributed to the bad data and which downstream dashboards and reports are affected. When the lineage service degrades, Sifflet may detect anomalies correctly but cannot serve the impact analysis that data engineers need to triage and resolve quality incidents efficiently:

curl https://app.siffletdata.com/api/v1/lineage/health \
  -H "Authorization: Bearer YOUR_API_TOKEN"
# Returns lineage service status when healthy
  1. Add Monitor → HTTP.
  2. URL: https://app.siffletdata.com/api/v1/lineage/health.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: status (present in the response JSON).
  7. Label: Sifflet lineage service.
  8. Click Save.

Lineage failure during incidents is the worst time: When a data quality incident occurs, teams immediately need the lineage graph to understand blast radius — which consumers are affected and which upstream sources are the root cause. If the lineage service is degraded precisely when an anomaly fires, engineers must investigate manually without the dependency map, significantly extending mean time to resolution.

Step 6: Monitor SSL Certificates

The Sifflet platform dashboard and REST API are accessed over HTTPS by both data team members using the browser and automated integrations (dbt Cloud, Airflow operators, CI/CD pipelines) that call the API on scheduled runs. An expired certificate breaks browser access to the observability dashboard and causes all API calls from automated systems to fail with TLS errors:

openssl s_client -connect app.siffletdata.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: app.siffletdata.com (or your self-managed Sifflet hostname).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Pipeline integration impact: Sifflet API clients embedded in Airflow DAGs or dbt Cloud jobs run on automated schedules without interactive input. When the TLS certificate expires, these clients fail immediately with SSL verification errors. If the pipeline treats the Sifflet quality gate as non-blocking, jobs continue and potentially bad data is promoted to production without a quality check — undermining the entire purpose of the observability integration.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Platform dashboard | Non-200 or Sifflet missing | Check Sifflet status page; verify platform infrastructure | | API health | Non-200 or healthy missing | API down; pipeline quality gates failing; check Sifflet status | | Alert webhook receiver | Non-200 or ok missing | Anomaly alerts not delivering; verify receiver deployment and logs | | Lineage service | Non-200 or status missing | Lineage graph unavailable; incident triage impacted | | SSL certificate | < 30 days to expiry | Renew certificate; verify all API integrations reconnect after renewal |

Alert after: 1 consecutive failure for the platform dashboard, API health, and lineage service monitors. 2 consecutive failures for the alert webhook monitor to reduce noise from transient timeouts during high-volume anomaly detection windows.

Common Sifflet Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Sifflet platform outage | Dashboard monitor fires; engineers cannot review anomalies or incidents | | REST API degradation | API monitor fires; pipeline quality gates and dbt integrations fail | | Alert webhook receiver crash | Webhook monitor fires; Slack and PagerDuty alerts stop posting | | Lineage service failure | Lineage monitor fires; incident triage loses dependency graph | | Connector sync failure | Catalog grows stale; new assets and schema changes go undetected | | TLS certificate expired | SSL monitor alerts; browser and API client access fails | | DNS misconfiguration | All monitors fire simultaneously | | Anomaly detection engine overload | API may return timeouts; dashboard shows stale quality scores |

Monitoring Data Quality Itself (Beyond Infrastructure)

Vigilmon monitors Sifflet's infrastructure endpoints — the platform dashboard, REST API, alert webhook receiver, and lineage service. The health of individual data quality monitors and the accuracy of detected anomalies is a separate concern managed within Sifflet itself:

  • Monitor coverage: Audit which tables, columns, and dbt models have Sifflet monitors configured to ensure new data assets are automatically included in quality tracking when added to the warehouse.
  • Anomaly threshold tuning: Review Sifflet's anomaly detection thresholds regularly to reduce false positive alerts that cause on-call fatigue, particularly for metrics with high natural variance during business cycle peaks.
  • Incident routing: Verify that Sifflet's incident routing rules correctly assign quality incidents to the right data owner for each asset, so escalations don't sit unacknowledged in the wrong team's queue.
  • Connector sync health: Review connector sync logs within Sifflet to catch partial failures where a BigQuery or Snowflake connector completes but crawls fewer datasets than expected.

Vigilmon catches infrastructure-level failures — when Sifflet itself is down or its API is unreachable. Individual anomaly detection coverage, threshold tuning, and data quality regressions within a healthy Sifflet deployment require Sifflet's own monitor configuration and incident management workflows.

Sifflet is the end-to-end observability layer that connects data quality detection, lineage context, and incident management in a single platform — when it goes down, anomaly detection halts, quality alerts stop flowing, lineage impact analysis becomes unavailable, and data incidents take significantly longer to resolve without the platform that ties detection to response. Vigilmon gives you external visibility into Sifflet's availability that doesn't depend on Sifflet self-reporting: platform uptime, API health, lineage service status, alert webhook delivery, and SSL certificate validity, so you know the moment the data observability platform itself needs attention and can restore end-to-end quality monitoring before the next pipeline run ships unvalidated data to your downstream consumers.

Start monitoring Sifflet in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles