How to Monitor Fivetran Data Pipelines with Vigilmon

Fivetran automates the most tedious part of data engineering: keeping your warehouse in sync with hundreds of SaaS sources. But automated connectors are not self-healing — a connector can fail silently, sync delays can accumulate for hours, and API quota exhaustion on a source system can stall entire pipelines without a single alert reaching your team.

In this tutorial you'll set up end-to-end monitoring for Fivetran connectors and the downstream systems they feed using Vigilmon — free tier, no credit card.

Why Fivetran connectors need dedicated monitoring

Fivetran handles connection management, but it does not replace external monitoring:

• Connector failures — a schema change on the source, an expired OAuth token, or a rate-limit event can pause a connector for hours; Fivetran's built-in emails are easy to miss
• Silent sync delays — a connector shows "syncing" but the last successful row was 12 hours ago; your dashboards serve stale data without anyone knowing
• Destination write failures — Fivetran reaches the source fine but the warehouse rejects writes (permissions, quota, type mismatch), and the connector is marked failed only after the next sync attempt
• Webhook delivery failures — if you rely on Fivetran webhooks to trigger downstream dbt runs or alerting workflows, a missed delivery breaks your entire data stack

Vigilmon heartbeat and HTTP monitors let you close these gaps by asserting that data is flowing end-to-end.

What you'll need

• A Fivetran account with at least one active connector
• A free Vigilmon account (sign up takes 30 seconds)
• (Optional) Fivetran webhook support enabled on your plan

Step 1: Set up a heartbeat monitor for each critical connector

The most reliable way to monitor Fivetran is to hook into its webhook notifications and forward a ping to Vigilmon after each successful sync.

Configure a Fivetran webhook:

1. In the Fivetran dashboard, go to Settings → Webhooks
2. Click Add Webhook
3. Set the webhook URL to a small relay service (see Step 2) or directly to a Vigilmon heartbeat URL
4. Select the events: sync_end (successful completion)
5. Save the webhook

Relay script (Node.js/Express) to forward only successful syncs:

const express = require('express');
const fetch = require('node-fetch');
const app = express();

app.use(express.json());

// Map connector_id → Vigilmon heartbeat URL
const HEARTBEAT_MAP = {
  'your_connector_id_salesforce': 'https://vigilmon.online/api/heartbeats/HEARTBEAT_1/ping',
  'your_connector_id_stripe':     'https://vigilmon.online/api/heartbeats/HEARTBEAT_2/ping',
};

app.post('/fivetran-webhook', async (req, res) => {
  const { event, connector_id, status } = req.body;

  if (event === 'sync_end' && status === 'SUCCESSFUL') {
    const heartbeatUrl = HEARTBEAT_MAP[connector_id];
    if (heartbeatUrl) {
      await fetch(heartbeatUrl, { method: 'GET' });
      console.log(`Heartbeat sent for connector ${connector_id}`);
    }
  }

  res.sendStatus(200);
});

app.listen(3000, () => console.log('Fivetran webhook relay running on :3000'));

Deploy this relay to any server or serverless function, and point your Fivetran webhook at it.

Step 2: Create heartbeat monitors in Vigilmon

For each critical connector, create a Vigilmon heartbeat monitor:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose Heartbeat as the type
3. Name it after the connector, e.g. Fivetran: Salesforce → BigQuery
4. Set the expected interval to match your connector's sync frequency (e.g., 6 hours for a 6-hour sync)
5. Set the grace period to 30 minutes
6. Copy the heartbeat URL and add it to your relay map from Step 1
7. Save the monitor

Vigilmon will alert you the moment a connector's sync stops reporting success within the expected window.

Step 3: Monitor the Fivetran API status page

Fivetran publishes a platform status page. Add an HTTP monitor in Vigilmon to get alerted when Fivetran itself has an outage:

1. Go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to https://status.fivetran.com/ (or the Fivetran status API endpoint)
4. Set check interval to 5 minutes
5. Set expected status code to 200
6. Save the monitor

This gives you early warning on platform-wide Fivetran incidents before your connectors start failing.

Step 4: Monitor your data warehouse endpoint

Fivetran writes data to your warehouse — monitor the warehouse itself to catch destination failures:

-- Run this query on a schedule to verify freshness
SELECT
  connector_name,
  MAX(synced_at) AS last_synced,
  DATEDIFF(MINUTE, MAX(synced_at), CURRENT_TIMESTAMP) AS minutes_since_sync
FROM fivetran_log.connector_status
GROUP BY connector_name
ORDER BY minutes_since_sync DESC;

Wrap this in a small script and ping a Vigilmon heartbeat if all connectors are within your freshness threshold:

#!/usr/bin/env bash
set -euo pipefail

# Query BigQuery for staleness (adjust for your warehouse)
STALE_COUNT=$(bq query --format=csv --use_legacy_sql=false \
  "SELECT COUNT(*) FROM \`your_project.fivetran_log.connector_status\`
   WHERE TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), synced_at, MINUTE) > 360" \
  | tail -1)

if [ "$STALE_COUNT" -eq 0 ]; then
  curl -fsS --max-time 10 \
    "https://vigilmon.online/api/heartbeats/FRESHNESS_HEARTBEAT_ID/ping" \
    > /dev/null
  echo "All connectors fresh — heartbeat sent."
else
  echo "WARNING: $STALE_COUNT connectors are stale. Heartbeat NOT sent."
  exit 1
fi

Schedule this script every hour to get a freshness alarm separate from the sync-completion alarm.

Step 5: Configure alerts and create a status page

Alert channels:

1. Go to Alert Channels → Add Channel
2. Add a Slack or PagerDuty channel for your data engineering on-call
3. Assign the channel to all Fivetran monitors

Example Vigilmon incident alert payload:

{
  "monitor_name": "Fivetran: Salesforce → BigQuery",
  "status": "down",
  "type": "heartbeat",
  "last_ping_at": "2024-01-15T06:00:00Z",
  "expected_by": "2024-01-15T12:30:00Z",
  "message": "Heartbeat overdue by 2 hours 15 minutes"
}

Status page:

1. Go to Status Pages → New Status Page
2. Name it "Data Pipeline Health"
3. Add all your Fivetran connector heartbeat monitors
4. Publish the page and share it with your data consumers and BI team

Putting it all together

| Monitor | Type | What it catches | |---------|------|-----------------| | Fivetran: Salesforce → BigQuery | Heartbeat | Connector failures, sync delays, OAuth expiry | | Fivetran: Stripe → BigQuery | Heartbeat | Payment data pipeline failures | | Warehouse freshness check | Heartbeat | Data older than SLA threshold | | status.fivetran.com | HTTP | Fivetran platform outages |

What's next

• dbt run monitoring — after Fivetran syncs, your dbt jobs transform the data; add a heartbeat monitor for dbt Cloud job completions
• SSL expiry monitoring — if your warehouse or relay service sits behind HTTPS, Vigilmon monitors your TLS certificates automatically
• Multi-region checks — Vigilmon probes from multiple regions, so a regional network partition won't generate a false alarm

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.