tutorial

How to Monitor Airbyte: Sync Health, Scheduler Availability, and Pipeline SLAs with Vigilmon

Airbyte connector sync failures and scheduler outages silently stall your data pipelines. Learn how to monitor Airbyte sync health, source/destination failure rates, worker pod restarts, and enforce pipeline SLAs with Vigilmon heartbeat checks.

Airbyte is a cornerstone of the modern data stack — but when a connector sync silently fails, your downstream analytics, dbt models, and dashboards quietly go stale. Unlike HTTP APIs that return 500 errors, failed Airbyte syncs just stop delivering rows. Your data warehouse fills with yesterday's data, nobody notices until a business stakeholder asks why the numbers look wrong, and by then you've lost hours of pipeline SLA.

Vigilmon gives you external visibility into Airbyte health through HTTP probe monitoring of the Airbyte API and heartbeat monitoring for your sync pipeline SLAs. This tutorial covers both.

Why Airbyte Needs External Monitoring

Airbyte's built-in UI shows sync status and failure counts — but only if someone is watching. External monitoring with Vigilmon adds:

  • Proactive alerting when the Airbyte server or scheduler becomes unreachable
  • Sync failure rate tracking via a sidecar that polls the Airbyte API for recent job status
  • Worker pod restart detection so you catch infrastructure-level churn before it cascades to sync failures
  • Heartbeat SLA enforcement so downstream consumers know when a pipeline hasn't delivered fresh data on time

These layers work together: the Airbyte scheduler can be healthy while a specific connector is silently failing, and a connector can report success while the data it delivers is missing rows due to a source-side API rate limit.

Step 1: Build an Airbyte Health Endpoint

Airbyte exposes a REST API at http://localhost:8000/api/v1 (or your deployed Airbyte host). You need a sidecar that wraps the API and returns a simple HTTP health response for Vigilmon to probe.

Node.js Health Sidecar

// health/airbyte.js
const express = require('express');
const axios = require('axios');

const app = express();
const AIRBYTE_API = process.env.AIRBYTE_API_URL || 'http://localhost:8000/api/v1';
const AIRBYTE_TOKEN = process.env.AIRBYTE_API_TOKEN;

const headers = AIRBYTE_TOKEN
  ? { Authorization: `Bearer ${AIRBYTE_TOKEN}` }
  : {};

// Basic server health — is the Airbyte API reachable?
app.get('/health/airbyte', async (req, res) => {
  try {
    const response = await axios.get(`${AIRBYTE_API}/health`, { headers, timeout: 5000 });
    if (response.data.available) {
      return res.status(200).json({ status: 'ok', version: response.data.version });
    }
    return res.status(503).json({ status: 'degraded', detail: response.data });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Sync health — check recent job failures across all connections
app.get('/health/airbyte/syncs', async (req, res) => {
  try {
    const workspaceId = process.env.AIRBYTE_WORKSPACE_ID;
    const jobsRes = await axios.post(
      `${AIRBYTE_API}/jobs/list`,
      { configTypes: ['sync'], configId: workspaceId },
      { headers, timeout: 10000 }
    );

    const jobs = jobsRes.data.jobs || [];
    const recent = jobs.slice(0, 20);
    const failed = recent.filter(j => j.status === 'failed');
    const failureRate = recent.length > 0 ? (failed.length / recent.length) : 0;

    if (failureRate > 0.3) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'high_sync_failure_rate',
        failure_rate: failureRate,
        failed_jobs: failed.map(j => ({ id: j.id, created_at: j.createdAt })),
      });
    }

    return res.status(200).json({ status: 'ok', failure_rate: failureRate, checked: recent.length });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3007);

Python Health Sidecar

# health/airbyte_health.py
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)
AIRBYTE_API = os.environ.get("AIRBYTE_API_URL", "http://localhost:8000/api/v1")
AIRBYTE_TOKEN = os.environ.get("AIRBYTE_API_TOKEN")
HEADERS = {"Authorization": f"Bearer {AIRBYTE_TOKEN}"} if AIRBYTE_TOKEN else {}

@app.route("/health/airbyte")
def airbyte_health():
    try:
        r = requests.get(f"{AIRBYTE_API}/health", headers=HEADERS, timeout=5)
        data = r.json()
        if data.get("available"):
            return jsonify({"status": "ok", "version": data.get("version")}), 200
        return jsonify({"status": "degraded", "detail": data}), 503
    except Exception as e:
        return jsonify({"status": "down", "error": str(e)}), 503

@app.route("/health/airbyte/syncs")
def sync_health():
    try:
        workspace_id = os.environ["AIRBYTE_WORKSPACE_ID"]
        r = requests.post(
            f"{AIRBYTE_API}/jobs/list",
            json={"configTypes": ["sync"], "configId": workspace_id},
            headers=HEADERS,
            timeout=10,
        )
        jobs = r.json().get("jobs", [])[:20]
        failed = [j for j in jobs if j["status"] == "failed"]
        rate = len(failed) / len(jobs) if jobs else 0
        if rate > 0.3:
            return jsonify({"status": "degraded", "failure_rate": rate}), 503
        return jsonify({"status": "ok", "failure_rate": rate}), 200
    except Exception as e:
        return jsonify({"status": "down", "error": str(e)}), 503

if __name__ == "__main__":
    app.run(port=3007)

Step 2: Configure Vigilmon HTTP Monitors for Airbyte

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Airbyte health endpoint: https://your-host.example.com/health/airbyte
  4. Set the check interval to 2 minutes
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 8000ms
  6. Under Alert channels, assign your on-call Slack channel or PagerDuty integration
  7. Save the monitor

Add a second monitor for sync failure rate:

  • URL: https://your-host.example.com/health/airbyte/syncs
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes (sync job lists can take longer to query)
  • Alert channel: data engineering Slack channel

Vigilmon checks from multiple regions so a single network hiccup doesn't trigger false-positive alerts.

Step 3: Heartbeat Monitoring for Pipeline SLAs

Knowing the Airbyte scheduler is alive isn't enough. Your business stakeholders care about whether the salesforce → BigQuery sync delivered fresh data by 06:00 UTC. Vigilmon heartbeat monitors enforce these SLAs: your pipeline sends a ping after each successful sync, and if the ping stops arriving, Vigilmon alerts your team.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: airbyte-salesforce-bigquery-sync
  3. Set the expected interval: 24 hours (or match your sync schedule)
  4. Set the grace period: 30 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire Heartbeats Into Airbyte Webhooks

Airbyte supports webhooks on sync completion. Configure one per critical connection in Settings → Notifications → Webhooks:

{
  "notificationTrigger": "sync_success",
  "webhookUrl": "https://vigilmon.online/heartbeat/abc123xyz",
  "sendOnSuccess": true,
  "sendOnFailure": false
}

For failure alerting, create a separate failure webhook that calls a different Vigilmon monitor set to always alert:

{
  "notificationTrigger": "sync_failure",
  "webhookUrl": "https://your-host.example.com/notify/failure",
  "sendOnSuccess": false,
  "sendOnFailure": true
}

Poll-Based Heartbeat (Airbyte API)

If webhooks aren't available in your Airbyte deployment, use a cron job to poll for sync completion and ping Vigilmon:

#!/bin/bash
# check_airbyte_sync.sh — runs every 5 minutes via cron

CONNECTION_ID="${AIRBYTE_CONNECTION_ID}"
WORKSPACE_ID="${AIRBYTE_WORKSPACE_ID}"
API="${AIRBYTE_API_URL}/api/v1"
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"

LATEST=$(curl -s -X POST "${API}/jobs/list" \
  -H "Authorization: Bearer ${AIRBYTE_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"configTypes\":[\"sync\"],\"configId\":\"${CONNECTION_ID}\"}" \
  | jq -r '.jobs[0].status')

if [ "${LATEST}" = "succeeded" ]; then
  curl -s "${HEARTBEAT_URL}"
fi

Add to crontab:

*/5 * * * * /opt/scripts/check_airbyte_sync.sh

Step 4: Monitoring Worker Pod Restarts (Kubernetes)

Airbyte runs connector syncs as ephemeral Kubernetes pods (worker pods). Frequent pod restarts indicate OOM kills, image pull failures, or resource contention — all of which precede sync failures.

Expose a restart count endpoint from your cluster monitoring:

# k8s_health.py — queries Kubernetes API for worker pod restarts
from kubernetes import client, config
from flask import Flask, jsonify
import os

app = Flask(__name__)
config.load_incluster_config()
v1 = client.CoreV1Api()
NAMESPACE = os.environ.get("AIRBYTE_NAMESPACE", "airbyte")

@app.route("/health/airbyte/workers")
def worker_health():
    pods = v1.list_namespaced_pod(namespace=NAMESPACE, label_selector="app=airbyte-worker")
    high_restart = [
        {"name": p.metadata.name, "restarts": p.status.container_statuses[0].restart_count}
        for p in pods.items
        if p.status.container_statuses and p.status.container_statuses[0].restart_count > 3
    ]
    if high_restart:
        return jsonify({"status": "degraded", "workers": high_restart}), 503
    return jsonify({"status": "ok", "pod_count": len(pods.items)}), 200

Summary

Airbyte sync failures are silent. External monitoring with Vigilmon catches scheduler outages, high sync failure rates, and SLA breaches before downstream consumers notice stale data:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/airbyte | Airbyte scheduler availability, API reachability | | HTTP monitor on /health/airbyte/syncs | Connector sync failure rate across the workspace | | Heartbeat monitor | Pipeline SLA — fresh data delivered on schedule | | HTTP monitor on /health/airbyte/workers | Worker pod restart churn (Kubernetes deployments) |

Get started free at vigilmon.online — your first Airbyte monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles