tutorial

Monitoring Portable with Vigilmon: Data Connector Pipeline Health, Sync API & Webhook Availability

How to monitor Portable data connector pipelines with Vigilmon — sync API availability, connector endpoint health, webhook receiver reachability, and SSL certificate validity.

Portable is a data connector platform that builds and maintains ETL pipelines from long-tail and niche data sources — hundreds of SaaS applications, databases, and APIs that major ETL vendors don't support — delivering that data into your cloud data warehouse so your analytics and data science teams can work with it alongside the rest of your operational data. Portable manages the connector code, handles API versioning changes from source vendors, and operates the pipeline infrastructure so your data team doesn't maintain custom scripts for each uncommon source. When the Portable API becomes unavailable, you cannot query pipeline status, trigger manual runs, or validate that your niche source connectors are actively delivering data to your warehouse. When a connector pipeline silently stalls — due to a source API version bump, an authentication expiry, or a schema change — your warehouse tables stop receiving updates without any visible alert unless external monitoring is in place. When the webhook receiver that captures Portable pipeline events is down, your data quality checks, downstream dbt runs, and orchestration triggers fire without knowing whether the upstream extraction even completed. Vigilmon gives your data engineering team external HTTP visibility into the Portable surfaces that keep your long-tail connector pipelines healthy — the sync API, the pipeline status endpoint, the connector health surface, and your webhook receivers — so you catch failures before they create silent data gaps in your warehouse.

What You'll Build

  • An HTTP monitor on the Portable API to detect platform outages before they block pipeline status queries and manual triggers
  • A monitor on the Portable pipeline status endpoint to catch degradation in connector execution and data delivery
  • A monitor on your Portable webhook receiver to verify pipeline completion and failure notifications are being delivered
  • A monitor on your warehouse loading endpoint to detect when Portable cannot reach the destination
  • An SSL certificate monitor for your Portable workspace domain
  • Alerting that distinguishes API outages from pipeline execution failures and notification delivery breakdowns

Prerequisites

  • A Portable account with one or more connector pipelines delivering data to a cloud data warehouse
  • A Portable API key for programmatic access to pipeline status and connector endpoints
  • An external health endpoint or webhook receiver that captures Portable pipeline events
  • A free account at vigilmon.online

Step 1: Understand Portable's Infrastructure Architecture

Portable pipelines run on managed infrastructure, but several surfaces are externally monitorable via the Portable REST API and your integration layer:

| Component | Surface | Role | |---|---|---| | Portable API | HTTPS REST (api.portable.io) | Central control plane; manages pipelines, connectors, runs | | Pipeline status endpoint | /pipelines | Lists configured pipelines and their current execution state | | Connector run endpoint | /runs | Tracks individual connector run history and error details | | Destination connections | /destinations | Reports on warehouse connectivity and credential validity | | Pipeline completion webhook | Webhook / Slack / PagerDuty | Delivers run success, failure, and schema change alerts | | Portable web UI | HTTPS | Engineer and admin access during pipeline incidents |

External monitoring focuses on the Portable API availability, the pipeline status endpoint, and your webhook receivers — these are where connector pipeline failures become externally observable before they propagate into stale warehouse tables and broken downstream analytics. An API outage halts all pipeline management; an authentication expiry on a source connector silently fails individual runs; a downed webhook receiver leaves your dbt orchestration and data quality checks operating without confirmation that upstream extraction completed.

Step 2: Monitor the Portable API

The Portable REST API is the control plane for all connector pipeline operations. Every pipeline status query, manual run trigger, connector validation, and destination health check flows through api.portable.io. When it becomes unavailable or returns errors, pipeline status cannot be read by external orchestration tools, run history cannot be inspected, and your team cannot validate whether the long-tail connector delivering critical source data is still active:

# Test Portable API availability
curl -H "Authorization: Bearer $PORTABLE_API_KEY" \
  https://api.portable.io/v1/pipelines
# Returns: {"data": [...]} when the API is healthy

# Verify your account and workspace
curl -H "Authorization: Bearer $PORTABLE_API_KEY" \
  https://api.portable.io/v1/account
# Returns account details when the API is responding
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://api.portable.io/v1/pipelines.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: data (present in every successful Portable API response).
  7. Label: Portable API.
  8. Click Save.

This monitor catches:

  • Full Portable platform outages that halt all pipeline status queries and management operations
  • Infrastructure incidents affecting the Portable API control plane
  • Authentication service failures that cause all API requests to return 401
  • Network path changes that have cut off your monitoring infrastructure from the Portable API

Alert sensitivity: Set to trigger after 1 consecutive failure. A Portable API outage means no pipeline can be queried, no run can be triggered manually, and no connector validation can be performed from any external tool.

Step 3: Monitor the Pipeline Status Endpoint

The pipelines endpoint is the specific surface that tracks which connector pipelines are active, which have failed their last run, and which are scheduled for the next extraction cycle. External orchestration tools, data observability platforms, and incident runbooks depend on this endpoint to understand the health of your long-tail source connectors and whether your warehouse tables are receiving fresh data:

# List all pipeline configurations and their current state
curl -H "Authorization: Bearer $PORTABLE_API_KEY" \
  https://api.portable.io/v1/pipelines
# Returns: {"data": [{"id": "...", "name": "...", "status": "..."}]}

# Get run history for a specific pipeline
curl -H "Authorization: Bearer $PORTABLE_API_KEY" \
  https://api.portable.io/v1/pipelines/YOUR_PIPELINE_ID/runs
# Returns run history with status, row counts, and error messages
  1. Add Monitor → HTTP.
  2. URL: https://api.portable.io/v1/pipelines.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: data (present in every successful pipeline list response).
  7. Label: Portable pipeline status.
  8. Click Save.

Why monitor the pipeline endpoint separately from the account API? Portable may experience partial degradation where the account endpoint returns healthy responses while the pipeline management surface encounters errors. The pipelines endpoint can be degraded while general API responses appear healthy, leaving pipeline status invisible to external tools even though the API appears reachable.

Step 4: Monitor the Run History Endpoint

The runs endpoint exposes the execution history of individual connector runs — row counts extracted and loaded, error messages from source API failures, schema change events, and authentication expiry errors. External data observability tools, freshness monitors, and incident response runbooks query this endpoint to distinguish between a healthy pipeline run and a silent zero-row delivery that indicates the source connector is broken:

# List recent connector runs across all pipelines
curl -H "Authorization: Bearer $PORTABLE_API_KEY" \
  https://api.portable.io/v1/runs
# Returns: {"data": [{"id": "...", "pipeline_id": "...", "status": "..."}]}

# Get details of a specific run including error context
curl -H "Authorization: Bearer $PORTABLE_API_KEY" \
  https://api.portable.io/v1/runs/YOUR_RUN_ID
# Returns full run details with extraction counts and error descriptions
  1. Add Monitor → HTTP.
  2. URL: https://api.portable.io/v1/runs.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: data.
  7. Label: Portable run history.
  8. Click Save.

This monitor detects availability failures on the run history surface, which external orchestration and data quality tools rely on to confirm that connector extractions completed before triggering downstream dbt transformations and BI refreshes.

Step 5: Monitor Your Pipeline Completion Webhook Receiver

Portable emits events — run completions, run failures, schema changes, and authentication errors — to external webhooks that feed your incident management, data orchestration, or notification systems. If the webhook receiver is down when a Portable run fails, your downstream automation — dbt Cloud job triggers, Slack alerts for connector failures, or data freshness monitors — receives no signal and operates on stale or incomplete warehouse data:

curl https://your-portable-webhook.example.com/health
# Returns: {"status": "ok"} when the webhook receiver is operational
  1. Add Monitor → HTTP.
  2. URL: https://your-portable-webhook.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Portable pipeline webhook.
  8. Click Save.

For Slack-based Portable failure notifications, verify https://slack.com/api/api.test to confirm Slack API reachability from your notification infrastructure. For PagerDuty-routed connector failures, monitor https://api.pagerduty.com/ability to verify the Events API is accessible.

Step 6: Monitor SSL Certificates

The Portable API is accessed by data engineers during incident response and by automated orchestration tools over TLS. An expired certificate on api.portable.io or any custom domain fronting your webhook receiver blocks both human access and automated integrations at the worst possible moment — when a connector has failed and your team needs to query run history to understand why the warehouse table stopped receiving updates:

openssl s_client -connect api.portable.io:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: api.portable.io.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for any custom domains that front your webhook receivers or internal tooling that integrates with the Portable API. Portable-managed certificates on api.portable.io are renewed by Portable, but custom webhook domains and internal proxies require your team's certificate management.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Portable API | Non-200 or data missing | All pipeline management halted; escalate to data engineering team | | Pipeline status | Non-200 or data missing | Pipeline state invisible; orchestration tools blind | | Run history | Non-200 or data missing | Run details unavailable; cannot confirm extraction completion | | Pipeline webhook | Non-200 or ok missing | Run events not being delivered; downstream automation stalled | | SSL certificate | < 30 days to expiry | Renew certificate; API access blocked on expiry |

Alert after: 1 consecutive failure for the Portable API and pipeline status endpoint. 2 consecutive failures for the run history endpoint and webhook receiver to reduce noise from brief transient timeouts during Portable platform maintenance windows.

Common Portable Infrastructure Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Portable platform outage | API monitor fires; all pipeline management halts | | Pipeline endpoint partial degradation | Pipeline monitor fires; orchestration tools lose status visibility | | Source API authentication expiry | Runs fail silently; run history monitor detects endpoint availability | | Webhook receiver crash | Webhook monitor fires; pipeline events not delivered | | TLS certificate expired on webhook | SSL monitor alerts at 30-day threshold | | Control plane maintenance window | API monitor fires during window | | Authentication service degradation | API returns 401; API monitor fires | | Network path to Portable API blocked | All API monitors fire with connection timeout | | Source vendor API version change | Individual runs fail; run history shows extraction errors | | Destination warehouse unreachable | Runs fail at load phase; pipeline status shows errors |

Monitoring Connector Data Quality (Beyond Infrastructure)

Vigilmon monitors the infrastructure that Portable depends on — the API control plane, the pipeline status surface, the run history layer, and your webhook receivers. The actual completeness, freshness, and schema correctness of what lands in your warehouse tables require additional observability practices:

  • Row count freshness: Cross-check Portable's reported extraction row counts against the actual row counts in your warehouse destination tables. Infrastructure health does not tell you that a connector is running successfully but extracting zero rows because the source API changed its pagination behavior.
  • Schema drift detection: Portable handles schema changes from source vendors, but warehouse table schemas can lag if a schema change event is missed. Monitor the column counts and data types of warehouse tables that Portable loads against your expected schema baseline.
  • Extraction latency tracking: Monitor the elapsed time between a scheduled run start and its COMPLETED state using the Portable runs API. A healthy API does not guarantee that individual connector runs are completing within the SLA windows your downstream dbt models depend on for timely transformation.

Vigilmon catches infrastructure failures — when the Portable API is unavailable, the pipeline status surface is degraded, or the run history endpoint is rejecting requests. The completeness of extracted data, the accuracy of schema mapping, and the freshness of warehouse table updates require domain-specific data quality monitoring layered on top of infrastructure health checks.

Portable solves the long-tail connector problem for data teams who need niche and uncommon source data flowing into their warehouse without maintaining custom ETL scripts, but the API infrastructure it runs on — the pipeline control plane, the run history layer, and your webhook receivers — can fail in ways that silently stall warehouse ingestion without surfacing visible errors to downstream consumers. Vigilmon gives your data engineering team external visibility into that infrastructure: API availability, pipeline status reachability, run history accessibility, and SSL certificate validity, so you know the moment Portable needs attention and can restore connector pipelines before warehouse staleness propagates into broken dashboards and incorrect model outputs.

Start monitoring your Portable data connector infrastructure in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles