How to Monitor Singer ETL Pipelines with Vigilmon

Singer is the open-source standard for writing scripts that move data — taps extract data from sources, targets load it into destinations, and the data flows between them through a simple JSON-based protocol. Teams run Singer pipelines on cron schedules, in containers, or as part of larger orchestration frameworks, and when a tap fails silently, data simply stops flowing. No error. No alert. Just stale tables.

In this tutorial you'll set up end-to-end monitoring for Singer pipelines using Vigilmon — free tier, no credit card.

Why Singer pipelines need dedicated monitoring

Singer's simplicity is also its blind spot for operations:

Silent tap failures — a tap exits with a non-zero code, the pipeline stops, but nothing downstream alerts on it
Schema drift — a source changes a field type and the target rejects records, causing partial loads without obvious errors
Credential expiry — API tokens rotate, taps start returning 401s, and no data flows for hours before anyone notices
Cron gaps — a scheduled Singer run is skipped or delayed (e.g., a locked process, a missed cron window), and staleness accumulates silently
Volume drops — a tap runs to completion but emits far fewer records than expected, indicating an upstream API degradation

Vigilmon's heartbeat monitors are the right tool here: your Singer pipeline checks in on success, and Vigilmon alerts you when the check-in stops arriving.

What you'll need

A Singer tap and target pipeline (we'll use tap-github and target-jsonl as examples)
A free Vigilmon account (sign up takes 30 seconds)

Step 1: Add a heartbeat ping to your Singer pipeline

Vigilmon heartbeat monitors work by expecting a periodic HTTP ping. When the ping stops arriving, Vigilmon fires an alert. The ideal place to send this ping is at the end of a successful Singer run.

Wrap your Singer pipeline in a shell script that pings Vigilmon on success:

#!/usr/bin/env bash
set -euo pipefail

# Run the Singer pipeline
tap-github --config tap_config.json --catalog catalog.json \
  | target-jsonl --config target_config.json

# Ping Vigilmon heartbeat on success
curl -fsS --max-time 10 \
  "https://vigilmon.online/api/heartbeats/YOUR_HEARTBEAT_ID/ping" \
  > /dev/null

echo "Pipeline complete, heartbeat sent."

The set -euo pipefail ensures the script exits immediately if any command fails — so the heartbeat ping is only sent on a fully successful run. If the tap or target fails, the ping never fires, and Vigilmon raises an incident.

Step 2: Create a heartbeat monitor in Vigilmon

Log in to vigilmon.online and go to Monitors → New Monitor
Choose Heartbeat as the type
Give the monitor a name, e.g. Singer: tap-github → target-jsonl
Set the expected interval to match your cron schedule — e.g., 1 hour for an hourly pipeline
Set the grace period to 5 minutes (Vigilmon waits this long past the deadline before triggering an incident)
Copy the generated heartbeat URL — it looks like https://vigilmon.online/api/heartbeats/abc123/ping
Save the monitor

Paste that URL into the curl command in your run script from Step 1.

Step 3: Monitor the pipeline endpoint if you expose a status API

Some Singer orchestration setups (like Meltano, which uses Singer under the hood) expose an HTTP status API. If yours does, set up an HTTP monitor in addition to the heartbeat:

Go to Monitors → New Monitor
Choose HTTP / HTTPS
Set the URL to your orchestration API's health endpoint, e.g. https://meltano.your-company.com/health
Set check interval to 1 minute
Set expected status code to 200
Save the monitor

For a standalone Singer pipeline without an HTTP interface, the heartbeat monitor from Step 2 is sufficient.

Step 4: Schedule your pipeline and wire up alerts

A typical cron entry for an hourly Singer pipeline:

0 * * * * /opt/singer/run_pipeline.sh >> /var/log/singer/pipeline.log 2>&1

Configure alert channels in Vigilmon so you're notified the moment a heartbeat is missed:

Go to Alert Channels → Add Channel
Choose Email, Slack, or Webhook
For a Slack webhook, paste your incoming webhook URL
Assign the alert channel to your Singer heartbeat monitor

Example webhook payload Vigilmon sends when a heartbeat is missed:

{
  "monitor_name": "Singer: tap-github → target-jsonl",
  "status": "down",
  "type": "heartbeat",
  "last_ping_at": "2024-01-15T09:00:00Z",
  "expected_by": "2024-01-15T10:05:00Z",
  "message": "Heartbeat overdue by 12 minutes"
}

Step 5: Create a status page for your data pipelines

If you run multiple Singer pipelines feeding dashboards or downstream systems, a status page gives your data consumers visibility into pipeline health.

In Vigilmon, go to Status Pages → New Status Page
Name it something like "Data Pipeline Health"
Add your Singer heartbeat monitors and any HTTP monitors
Publish the page

You get a public URL like https://status.vigilmon.online/your-page that shows live pipeline status — share it with your data team or embed it in your internal data portal.

Putting it all together

Here's the complete monitoring setup for a Singer pipeline:

#!/usr/bin/env bash
# /opt/singer/run_pipeline.sh
set -euo pipefail

HEARTBEAT_URL="https://vigilmon.online/api/heartbeats/YOUR_HEARTBEAT_ID/ping"
LOG_FILE="/var/log/singer/pipeline.log"

echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Starting Singer pipeline" >> "$LOG_FILE"

tap-github --config /etc/singer/tap_config.json \
           --catalog /etc/singer/catalog.json \
  | target-jsonl --config /etc/singer/target_config.json

echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Pipeline succeeded" >> "$LOG_FILE"

curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null
echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Heartbeat sent" >> "$LOG_FILE"

And the corresponding Vigilmon monitors:

| Monitor | Type | What it catches | |---------|------|-----------------| | Singer: tap-github → target-jsonl | Heartbeat | Pipeline failure, cron gaps, credential expiry | | Meltano health endpoint (if applicable) | HTTP | Orchestration API failures |

What's next

Multiple taps — create one heartbeat monitor per tap-target pair so you can pinpoint exactly which pipeline failed
SSL expiry monitoring — if your Singer orchestration runs behind HTTPS, add a certificate expiry monitor in Vigilmon
Heartbeat volume thresholds — combine heartbeats with a lightweight record-count check in your run script to alert on unexpectedly small syncs

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.