tutorial

Monitoring Etleap with Vigilmon: Cloud ETL Pipeline Health, API Availability & Webhook Delivery

How to monitor Etleap cloud ETL pipelines with Vigilmon — API availability, pipeline execution endpoint health, webhook receiver reachability, and SSL certificate validity.

Etleap is a cloud ETL platform designed for data engineering teams who need reliable, scalable data pipelines from operational databases, SaaS applications, and event streams into cloud data warehouses like Snowflake, Amazon Redshift, and Google BigQuery. Etleap handles schema evolution, handles late-arriving data, and maintains continuous incremental loading so your warehouse always reflects the freshest state of your source systems without the operational burden of managing custom pipeline code. When the Etleap API becomes unavailable, you cannot query pipeline status, inspect load history, trigger manual backfills, or validate that your pipelines are actively delivering data to the warehouse. When an Etleap pipeline silently stalls — due to a source credential expiry, a schema change that breaks the destination mapping, or a transient source database outage — your warehouse tables stop receiving updates and downstream analytics operate on increasingly stale data without any external alert. When the webhook receiver that captures Etleap pipeline events is down, your orchestration tools, data quality monitors, and incident management systems receive no signal about whether the upstream ETL layer completed successfully. Vigilmon gives your data engineering team external HTTP visibility into the Etleap surfaces that keep your ETL pipelines healthy — the management API, the pipeline status endpoint, the alert webhook receiver, and your SSL certificates — so you catch failures before they cascade into stale warehouse tables and broken downstream analytics.

What You'll Build

  • An HTTP monitor on the Etleap API to detect platform outages before they block pipeline management and status queries
  • A monitor on the Etleap pipeline list endpoint to catch degradation in pipeline execution reporting
  • A monitor on the Etleap pipeline alerts endpoint to verify that error notifications are reachable
  • A monitor on your Etleap webhook receiver to confirm pipeline completion events are being delivered
  • An SSL certificate monitor for your Etleap workspace domain
  • Alerting that distinguishes API outages from pipeline execution failures and notification delivery breakdowns

Prerequisites

  • An Etleap account with one or more ETL pipelines loading data into a cloud data warehouse
  • An Etleap API key for programmatic access to pipeline status and management endpoints
  • An external health endpoint or webhook receiver that captures Etleap pipeline events
  • A free account at vigilmon.online

Step 1: Understand Etleap's Infrastructure Architecture

Etleap pipelines run on managed cloud infrastructure, but several surfaces are externally monitorable via the Etleap REST API and your integration layer:

| Component | Surface | Role | |---|---|---| | Etleap API | HTTPS REST (api.etleap.com) | Central control plane; manages pipelines, connections, runs | | Pipeline list endpoint | /v2/pipelines | Lists configured pipelines and their current status | | Pipeline run endpoint | /v2/pipeline-runs | Tracks individual run history with row counts and errors | | Connection health endpoint | /v2/connections | Reports source and destination credential validity | | Alert notifications | Webhook / Email / Slack | Delivers pipeline failure and schema change alerts | | Etleap web UI | HTTPS | Engineer and admin access during ETL incidents |

External monitoring focuses on the Etleap API availability, the pipeline status endpoint, and your webhook receivers — these are where ETL failures become externally observable before they propagate into stale warehouse tables and broken downstream reports. An API outage halts all pipeline management; a credential expiry on a source database connection silently fails incremental loads; a downed webhook receiver leaves your orchestration tools and freshness monitors operating without confirmation that the ETL layer completed.

Step 2: Monitor the Etleap API

The Etleap REST API is the control plane for all ETL pipeline operations. Every pipeline status query, manual backfill trigger, connection validation, and run history inspection flows through api.etleap.com. When it becomes unavailable or returns errors, pipeline status cannot be read by external data observability tools, backfills cannot be triggered, and your team cannot validate whether source connectors are actively delivering data to the warehouse:

# Test Etleap API availability
curl -H "Authorization: Bearer $ETLEAP_API_KEY" \
  https://api.etleap.com/v2/pipelines
# Returns: {"data": [...]} when the API is healthy

# Verify API connectivity with the connections endpoint
curl -H "Authorization: Bearer $ETLEAP_API_KEY" \
  https://api.etleap.com/v2/connections
# Returns: {"data": [...]} listing all source and destination connections
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://api.etleap.com/v2/pipelines.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: data (present in every successful Etleap API response).
  7. Label: Etleap API.
  8. Click Save.

This monitor catches:

  • Full Etleap platform outages that halt all pipeline status queries and management operations
  • Infrastructure incidents affecting the Etleap API control plane
  • Authentication service failures that cause all API requests to return 401
  • Network path changes that have cut off your monitoring infrastructure from the Etleap API

Alert sensitivity: Set to trigger after 1 consecutive failure. An Etleap API outage means no pipeline can be queried, no backfill can be triggered, and no connection can be validated from any external tool.

Step 3: Monitor the Pipeline Status Endpoint

The pipelines endpoint is the specific surface that tracks which ETL pipelines are active, which have errored on the most recent run, and which are scheduled for the next incremental load cycle. Data observability platforms, orchestration tools, and incident runbooks depend on this endpoint to understand the health of your ETL layer and confirm that warehouse tables are receiving incremental updates:

# List all pipeline configurations and their current status
curl -H "Authorization: Bearer $ETLEAP_API_KEY" \
  https://api.etleap.com/v2/pipelines
# Returns: {"data": [{"id": "...", "name": "...", "status": "...", "latestRun": {...}}]}

# Get details for a specific pipeline including its last run outcome
curl -H "Authorization: Bearer $ETLEAP_API_KEY" \
  https://api.etleap.com/v2/pipelines/YOUR_PIPELINE_ID
# Returns full pipeline definition with connection IDs, schedule, and latest run status
  1. Add Monitor → HTTP.
  2. URL: https://api.etleap.com/v2/pipelines.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: data (present in every successful pipeline list response).
  7. Label: Etleap pipeline status.
  8. Click Save.

Why monitor the pipeline endpoint separately from the connections API? Etleap may experience partial degradation where the connections endpoint returns healthy responses while the pipeline management surface encounters errors. The pipelines endpoint can be degraded while general API responses appear healthy, leaving pipeline execution status invisible to external tools even though the API appears reachable.

Step 4: Monitor the Pipeline Runs Endpoint

The pipeline runs endpoint exposes the execution history of individual ETL run cycles — row counts extracted and loaded, error messages from source query failures, late-data recovery records, and schema change events. External data freshness monitors, completeness checks, and incident response playbooks query this endpoint to distinguish between a healthy incremental load and a silent zero-row run that indicates the source connector has broken:

# List recent pipeline runs across all pipelines
curl -H "Authorization: Bearer $ETLEAP_API_KEY" \
  https://api.etleap.com/v2/pipeline-runs
# Returns: {"data": [{"id": "...", "pipelineId": "...", "status": "...", "rowsLoaded": ...}]}

# Get the run history for a specific pipeline
curl -H "Authorization: Bearer $ETLEAP_API_KEY" \
  "https://api.etleap.com/v2/pipeline-runs?pipelineId=YOUR_PIPELINE_ID"
# Returns runs filtered to that pipeline with counts and error details
  1. Add Monitor → HTTP.
  2. URL: https://api.etleap.com/v2/pipeline-runs.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: data.
  7. Label: Etleap run history.
  8. Click Save.

This monitor detects availability failures on the run history surface, which external data quality and freshness tools rely on to confirm that incremental ETL loads completed before triggering downstream dbt transformations and BI dashboard refreshes.

Step 5: Monitor Your Pipeline Event Webhook Receiver

Etleap emits events — run completions, run failures, schema changes, and connection errors — to external webhooks that feed your incident management, orchestration, or notification systems. If the webhook receiver is down when an Etleap pipeline fails, your downstream automation — dbt Cloud job triggers based on load completion, Slack alerts for ETL failures, or data quality checks that run after each load cycle — receives no signal and operates on stale or incomplete warehouse data:

curl https://your-etleap-webhook.example.com/health
# Returns: {"status": "ok"} when the webhook receiver is operational
  1. Add Monitor → HTTP.
  2. URL: https://your-etleap-webhook.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Etleap pipeline webhook.
  8. Click Save.

For Slack-based Etleap failure notifications, verify https://slack.com/api/api.test to confirm Slack API reachability from your notification infrastructure. For PagerDuty-routed ETL failures, monitor https://api.pagerduty.com/ability to verify the Events API is accessible.

Step 6: Monitor SSL Certificates

The Etleap API is accessed by data engineers during incident response and by automated orchestration tools over TLS. An expired certificate on api.etleap.com or any custom domain fronting your webhook receiver blocks both human access and automated integrations at the worst possible moment — when an ETL pipeline has failed and your team needs to query run history to understand the scope of the warehouse data gap:

openssl s_client -connect api.etleap.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: api.etleap.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for any custom domains that front your webhook receivers or internal tooling that integrates with the Etleap API. Etleap-managed certificates on api.etleap.com are renewed by Etleap, but custom webhook domains and internal proxies require your team's certificate management.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Etleap API | Non-200 or data missing | All pipeline management halted; escalate to data engineering team | | Pipeline status | Non-200 or data missing | Pipeline state invisible; orchestration tools blind to ETL health | | Run history | Non-200 or data missing | Run details unavailable; cannot confirm load completion | | Pipeline webhook | Non-200 or ok missing | Run events not being delivered; downstream automation stalled | | SSL certificate | < 30 days to expiry | Renew certificate; API access blocked on expiry |

Alert after: 1 consecutive failure for the Etleap API and pipeline status endpoint. 2 consecutive failures for the run history endpoint and webhook receiver to reduce noise from brief transient timeouts during Etleap platform maintenance windows.

Common Etleap Infrastructure Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Etleap platform outage | API monitor fires; all pipeline management halts | | Pipeline endpoint partial degradation | Pipeline monitor fires; orchestration tools lose status visibility | | Source database credential expiry | Runs fail silently; run history endpoint shows extraction errors | | Webhook receiver crash | Webhook monitor fires; pipeline events not delivered | | TLS certificate expired on webhook | SSL monitor alerts at 30-day threshold | | Control plane maintenance window | API monitor fires during window | | Authentication service degradation | API returns 401; API monitor fires | | Network path to Etleap API blocked | All API monitors fire with connection timeout | | Destination warehouse schema drift | Individual runs fail at load phase; run history shows schema errors | | Source table DDL change breaks extraction | Runs fail with schema detection error; run history shows error context |

Monitoring ETL Data Quality (Beyond Infrastructure)

Vigilmon monitors the infrastructure that Etleap depends on — the API control plane, the pipeline status surface, the run history layer, and your webhook receivers. The actual completeness, freshness, and fidelity of what lands in your warehouse tables require additional observability practices:

  • Row count verification: Cross-check Etleap's reported rows-loaded counts against the actual row counts in your warehouse destination tables. Infrastructure health does not tell you that an incremental load is running successfully but delivering fewer rows than expected because the source query window is misconfigured or a source table was truncated.
  • Data freshness tracking: Monitor the updated_at or _etleap_loaded_at timestamp columns in your warehouse tables to verify that data freshness matches the pipeline schedule. An Etleap run that completes with zero errors but loads data with stale source timestamps indicates a source-side problem that infrastructure monitoring alone cannot catch.
  • Schema evolution monitoring: Etleap handles many schema changes automatically, but some changes — column type conflicts, primary key changes, or source table renames — require intervention. Monitor the schema of warehouse tables that Etleap loads against your expected schema baseline and alert when unexpected column additions or type changes appear.

Vigilmon catches infrastructure failures — when the Etleap API is unavailable, the pipeline status surface is degraded, or the run history endpoint is rejecting requests. The completeness of extracted data, the accuracy of incremental load windows, and the freshness of warehouse table updates require domain-specific data quality monitoring layered on top of infrastructure health checks.

Etleap makes reliable cloud ETL accessible to data engineering teams who need continuous incremental loading without the operational burden of managing custom pipeline code, but the API infrastructure it runs on — the pipeline control plane, the run history layer, and your webhook receivers — can fail in ways that silently stall warehouse ingestion without surfacing visible errors to downstream analytics consumers. Vigilmon gives your data engineering team external visibility into that infrastructure: API availability, pipeline status reachability, run history accessibility, and SSL certificate validity, so you know the moment Etleap needs attention and can restore pipeline health before warehouse staleness propagates into broken dashboards and incorrect business metrics.

Start monitoring your Etleap cloud ETL infrastructure in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles