tutorial

Monitoring Panoply with Vigilmon: Cloud Data Warehouse Health, API Availability & Ingestion Pipeline Uptime

How to monitor Panoply cloud data warehouse infrastructure with Vigilmon — API availability, data ingestion endpoint health, query service reachability, webhook receiver uptime, and SSL certificate validity.

Panoply is a cloud data warehouse platform built on Amazon Redshift that combines managed warehouse infrastructure with automated data ingestion from popular SaaS sources, databases, and event streams. Panoply handles warehouse provisioning, query optimization, and incremental data loading from connected sources, giving data and analytics teams a self-service path to a fully managed warehouse without requiring deep database administration expertise. When the Panoply API becomes unavailable, you cannot query ingestion job status, manage data source connections, or trigger manual data refreshes programmatically. When a Panoply ingestion pipeline silently stalls — due to a source authentication expiry, a schema change at the source, or a capacity constraint on the underlying Redshift cluster — your warehouse tables stop receiving fresh data and downstream analysts, dashboards, and models operate on increasingly stale inputs without any external alert. When the webhook receiver that captures Panoply ingestion events is down, your orchestration tools and data quality monitors have no confirmation that the upstream ingestion cycle completed before triggering dependent transformations. Vigilmon gives your data and analytics engineering team external HTTP visibility into the Panoply surfaces that keep your managed warehouse healthy — the management API, the data source status endpoint, the query service, and your event webhook receivers — so you catch failures before they propagate into stale warehouse data and broken downstream analytics.

What You'll Build

  • An HTTP monitor on the Panoply API to detect platform outages before they block warehouse management and ingestion status queries
  • A monitor on the Panoply data sources endpoint to catch degradation in ingestion pipeline reporting
  • A monitor on your Panoply webhook receiver to confirm ingestion completion events are being delivered
  • A monitor on the Panoply query service endpoint to verify analyst access to the warehouse is functional
  • An SSL certificate monitor for your Panoply workspace domain
  • Alerting that distinguishes API outages from ingestion failures and warehouse query service degradation

Prerequisites

  • A Panoply account with one or more data sources configured for automated ingestion
  • A Panoply API token for programmatic access to ingestion status and data source management endpoints
  • An external health endpoint or webhook receiver that captures Panoply ingestion events
  • A free account at vigilmon.online

Step 1: Understand Panoply's Infrastructure Architecture

Panoply runs on managed Amazon Redshift infrastructure, but several surfaces are externally monitorable via the Panoply REST API and your integration layer:

| Component | Surface | Role | |---|---|---| | Panoply API | HTTPS REST (api.panoply.io) | Central control plane; manages sources, ingestion, warehouse config | | Data sources endpoint | /sources | Lists connected data sources and their current ingestion status | | Ingestion jobs endpoint | /jobs | Tracks individual ingestion job history with row counts and errors | | Warehouse query service | Redshift endpoint | SQL query access for analysts and BI tools | | Ingestion completion webhook | Webhook / Email / Slack | Delivers ingestion success, failure, and schema change alerts | | Panoply web UI | HTTPS | Analyst and engineer access during warehouse incidents |

External monitoring focuses on the Panoply API availability, the data sources endpoint, and your webhook receivers — these are where ingestion failures become externally observable before they propagate into stale warehouse tables and broken analyst workflows. An API outage halts all ingestion management; a source credential expiry silently stops incremental loads from that source; a downed webhook receiver leaves your orchestration and freshness monitoring operating without confirmation that ingestion completed.

Step 2: Monitor the Panoply API

The Panoply REST API is the control plane for all warehouse management and ingestion operations. Every source status query, manual ingestion trigger, schema inspection, and job history check flows through api.panoply.io. When it becomes unavailable or returns errors, ingestion status cannot be read by external data observability tools, manual refreshes cannot be triggered, and your team cannot validate whether data sources are actively delivering data to the warehouse:

# Test Panoply API availability
curl -H "Authorization: Token $PANOPLY_API_TOKEN" \
  https://api.panoply.io/sources
# Returns: [{"id": "...", "type": "...", "status": "..."}] when the API is healthy

# Verify account connectivity
curl -H "Authorization: Token $PANOPLY_API_TOKEN" \
  https://api.panoply.io/account
# Returns account details when the API is responding
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://api.panoply.io/sources.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: id (present in every successful Panoply source list response).
  7. Label: Panoply API.
  8. Click Save.

This monitor catches:

  • Full Panoply platform outages that halt all ingestion management and status queries
  • Infrastructure incidents affecting the Panoply API control plane
  • Authentication service failures that cause all API requests to return 401
  • Network path changes that have cut off your monitoring infrastructure from the Panoply API

Alert sensitivity: Set to trigger after 1 consecutive failure. A Panoply API outage means no ingestion can be managed, no source status can be queried, and no manual refresh can be triggered from any external tool.

Step 3: Monitor the Data Sources Endpoint

The data sources endpoint is the specific surface that reports whether each connected source — Salesforce, HubSpot, Stripe, Google Analytics, databases, or event streams — is actively ingesting, has errored, or has been paused. External data observability tools, freshness monitors, and incident runbooks depend on this endpoint to understand the health of your entire ingestion layer and confirm that warehouse tables are receiving data from all expected sources:

# List all data sources and their current ingestion status
curl -H "Authorization: Token $PANOPLY_API_TOKEN" \
  https://api.panoply.io/sources
# Returns: [{"id": "...", "type": "...", "status": "...", "lastCollect": "..."}]

# Get details for a specific data source
curl -H "Authorization: Token $PANOPLY_API_TOKEN" \
  https://api.panoply.io/sources/YOUR_SOURCE_ID
# Returns full source configuration, last ingestion timestamp, and error details
  1. Add Monitor → HTTP.
  2. URL: https://api.panoply.io/sources.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: id (present in every successful sources response).
  7. Label: Panoply data sources.
  8. Click Save.

Why monitor the sources endpoint separately from the account API? Panoply may experience partial degradation where the account endpoint returns healthy responses while the data source management surface encounters errors. The sources endpoint can be degraded while general API responses appear healthy, leaving ingestion status invisible to external tools even though the API appears reachable.

Step 4: Monitor the Ingestion Jobs Endpoint

The jobs endpoint exposes the execution history of individual ingestion runs — row counts collected and loaded, error messages from source API failures, schema change detections, and authentication error details. External data freshness monitors, completeness checks, and incident response playbooks query this endpoint to distinguish between a healthy incremental ingest and a silent failure that left the warehouse table without an expected update:

# List recent ingestion jobs across all sources
curl -H "Authorization: Token $PANOPLY_API_TOKEN" \
  https://api.panoply.io/jobs
# Returns: [{"id": "...", "sourceId": "...", "status": "...", "rowsCollected": ...}]

# Get jobs for a specific source
curl -H "Authorization: Token $PANOPLY_API_TOKEN" \
  "https://api.panoply.io/jobs?sourceId=YOUR_SOURCE_ID"
# Returns ingestion jobs filtered to that source with full error context
  1. Add Monitor → HTTP.
  2. URL: https://api.panoply.io/jobs.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: id.
  7. Label: Panoply ingestion jobs.
  8. Click Save.

This monitor detects availability failures on the job history surface, which external data quality and freshness tools rely on to confirm that ingestion cycles completed before triggering downstream dbt transformations and BI dashboard refreshes.

Step 5: Monitor Your Ingestion Event Webhook Receiver

Panoply emits events — ingestion completions, ingestion failures, and schema changes — to external webhooks that feed your incident management, orchestration, or notification systems. If the webhook receiver is down when a Panoply ingestion fails, your downstream automation — dbt Cloud jobs triggered on ingestion completion, Slack alerts for source failures, or data quality checks that run after each ingestion cycle — receives no signal and operates on stale or incomplete warehouse data:

curl https://your-panoply-webhook.example.com/health
# Returns: {"status": "ok"} when the webhook receiver is operational
  1. Add Monitor → HTTP.
  2. URL: https://your-panoply-webhook.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Panoply ingestion webhook.
  8. Click Save.

For Slack-based Panoply failure notifications, verify https://slack.com/api/api.test to confirm Slack API reachability from your notification infrastructure. For PagerDuty-routed ingestion failures, monitor https://api.pagerduty.com/ability to verify the Events API is accessible.

Step 6: Monitor SSL Certificates

The Panoply API is accessed by data engineers during incident response and by automated orchestration tools over TLS. An expired certificate on api.panoply.io or any custom domain fronting your webhook receiver blocks both human access and automated integrations at the worst possible moment — when an ingestion source has failed and your team needs to query job history to understand which warehouse tables are missing expected data updates:

openssl s_client -connect api.panoply.io:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: api.panoply.io.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for any custom domains that front your webhook receivers or internal tooling that integrates with the Panoply API. Panoply-managed certificates on api.panoply.io are renewed by Panoply, but custom webhook domains and internal proxies require your team's certificate management.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Panoply API | Non-200 or id missing | All ingestion management halted; escalate to data engineering team | | Data sources | Non-200 or id missing | Source status invisible; cannot confirm active ingestion | | Ingestion jobs | Non-200 or id missing | Job history unavailable; freshness monitoring blind | | Ingestion webhook | Non-200 or ok missing | Ingestion events not delivered; downstream automation stalled | | SSL certificate | < 30 days to expiry | Renew certificate; API access blocked on expiry |

Alert after: 1 consecutive failure for the Panoply API and data sources endpoint. 2 consecutive failures for the ingestion jobs endpoint and webhook receiver to reduce noise from brief transient timeouts during Panoply platform maintenance windows.

Common Panoply Infrastructure Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Panoply platform outage | API monitor fires; all ingestion management halts | | Sources endpoint partial degradation | Sources monitor fires; ingestion status becomes invisible | | Source credential expiry | Individual sources stop ingesting; job history shows auth errors | | Webhook receiver crash | Webhook monitor fires; ingestion events not delivered | | TLS certificate expired on webhook | SSL monitor alerts at 30-day threshold | | Control plane maintenance window | API monitor fires during window | | Authentication service degradation | API returns 401; API monitor fires | | Network path to Panoply API blocked | All API monitors fire with connection timeout | | Underlying Redshift cluster degradation | API may respond healthy but queries time out | | Source SaaS API rate limiting | Individual source ingestions throttle; job history shows partial runs |

Monitoring Warehouse Data Quality (Beyond Infrastructure)

Vigilmon monitors the infrastructure that Panoply depends on — the API control plane, the data sources management surface, the ingestion job history layer, and your webhook receivers. The actual completeness, freshness, and accuracy of what lands in your warehouse tables require additional observability practices:

  • Table freshness monitoring: Query your Panoply warehouse directly for the maximum updated_at or _sdc_received_at timestamp in tables that Panoply loads. Infrastructure health does not tell you that ingestion is running successfully but delivering data that is hours behind the source because the incremental replication window is misconfigured.
  • Row count anomaly detection: Track the daily row count delivered by each Panoply source over time and alert when a source delivers significantly fewer rows than its historical baseline. A source can appear healthy in the API while delivering zero incremental rows because a filter at the source level has changed.
  • Schema drift in warehouse tables: Panoply handles some schema changes automatically, but new columns added by source SaaS vendors may not appear in warehouse tables immediately. Monitor your warehouse table schemas against expected column definitions and alert when columns disappear or change types unexpectedly.

Vigilmon catches infrastructure failures — when the Panoply API is unavailable, the data sources surface is degraded, or the ingestion job history endpoint is rejecting requests. The completeness of collected data from each source, the accuracy of incremental replication windows, and the freshness of warehouse table updates require domain-specific data quality monitoring layered on top of infrastructure health checks.

Panoply makes managed cloud data warehousing and automated data ingestion accessible to data and analytics teams who need a fully managed warehouse without deep database administration expertise, but the API infrastructure it runs on — the ingestion management control plane, the data sources layer, and your webhook receivers — can fail in ways that silently stall warehouse ingestion without surfacing visible errors to downstream analysts and BI consumers. Vigilmon gives your data engineering team external visibility into that infrastructure: API availability, data source status reachability, ingestion job history accessibility, and SSL certificate validity, so you know the moment Panoply needs attention and can restore ingestion health before warehouse staleness propagates into broken dashboards and incorrect business metrics.

Start monitoring your Panoply cloud data warehouse infrastructure in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles