How to Monitor dbt Core and dbt Cloud: Job Runs, Model Tests, and Artifact Freshness with Vigilmon

dbt is the transformation layer that keeps your analytics data trustworthy — but when a dbt job silently fails at 03:00 UTC, your BI dashboards serve yesterday's numbers to executives making today's decisions. Unlike web application failures that produce HTTP errors, dbt failures produce nothing: the stale model just sits in your warehouse, indistinguishable from fresh data without external monitoring.

Vigilmon gives you external visibility into dbt health through heartbeat monitoring for scheduled job runs, HTTP probe monitoring for dbt Cloud's API, and artifact freshness checks for dbt Core deployments. This tutorial covers both deployment modes.

Why dbt Needs External Monitoring

dbt's built-in tooling (dbt Cloud job history, Airflow task logs, artifact metadata) provides rich job-level detail — but requires someone to be watching. External monitoring with Vigilmon adds:

• Heartbeat SLA enforcement so you know when a dbt job didn't complete on schedule, even if the orchestrator thinks it succeeded
• Model test failure alerting via a post-run hook that pings Vigilmon only when all tests pass
• Artifact freshness checks so downstream consumers know when manifest.json is older than expected
• dbt Cloud API availability monitoring for teams using the managed service

These layers work together: a dbt Cloud job can show "succeeded" in the UI while upstream source freshness tests fail, and an Airflow-orchestrated dbt Core run can appear healthy while writing zero rows to the target model.

Step 1: Heartbeat Monitoring for dbt Job Runs

The most reliable way to detect dbt job failures is the inverse approach: your dbt job sends a heartbeat ping to Vigilmon on successful completion. If the ping doesn't arrive within the expected window, Vigilmon alerts your team.

Set Up the Heartbeat Monitor

1. Log in to vigilmon.online and go to Monitors → New Monitor → Heartbeat
2. Set the name: dbt-nightly-transform
3. Set the expected interval: 24 hours (match your job schedule)
4. Set the grace period: 30 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

dbt Core — Post-hook in dbt_project.yml

For dbt Core, add an on-run-end hook that pings Vigilmon only when the run succeeds with zero test failures:

# dbt_project.yml
on-run-end:
  - "{{ log_run_results() }}"
  - >
    {% if results | selectattr('status', 'eq', 'fail') | list | length == 0 %}
      {% set _ = run_query(
        'SELECT 1'
      ) %}
    {% endif %}

A cleaner approach is a shell wrapper script around dbt run && dbt test:

#!/bin/bash
# run_dbt.sh — wraps dbt run + test and pings Vigilmon on success

set -e

VIGILMON_URL="${VIGILMON_HEARTBEAT_URL:?'VIGILMON_HEARTBEAT_URL not set'}"
DBT_PROFILES_DIR="${DBT_PROFILES_DIR:-~/.dbt}"

echo "Starting dbt run..."
dbt run --profiles-dir "${DBT_PROFILES_DIR}" --target prod

echo "Running dbt tests..."
dbt test --profiles-dir "${DBT_PROFILES_DIR}" --target prod

echo "All models ran and tests passed — pinging Vigilmon..."
curl -fsS "${VIGILMON_URL}" > /dev/null

echo "Done."

If dbt run or dbt test exits non-zero, set -e halts the script before the Vigilmon ping. Missing heartbeats surface failed runs.

dbt Cloud — Webhook to Vigilmon

dbt Cloud supports webhooks on job completion. Go to Account Settings → Webhooks → New Webhook:

{
  "name": "vigilmon-nightly-transform",
  "clientUrl": "https://vigilmon.online/heartbeat/abc123xyz",
  "eventTypes": ["job.run.completed"],
  "jobIds": ["your-job-id"],
  "active": true
}

For conditional pinging (only on success), put a thin proxy in front:

// webhook-proxy/index.js — only pings Vigilmon on successful dbt Cloud runs
const express = require('express');
const axios = require('axios');
const app = express();

app.use(express.json());

app.post('/webhook/dbt', async (req, res) => {
  const { runStatus, runId } = req.body;
  if (runStatus === 'Success') {
    await axios.get(process.env.VIGILMON_HEARTBEAT_URL).catch(console.error);
    console.log(`Heartbeat sent for run ${runId}`);
  }
  res.sendStatus(200);
});

app.listen(3008);

Step 2: Model Test Failure Rate Monitoring

dbt tests catch data quality issues — but only if someone notices when they fail. Build a health endpoint that queries your test results:

# health/dbt_health.py
import os, json, glob
from flask import Flask, jsonify
from datetime import datetime, timezone

app = Flask(__name__)
DBT_TARGET_DIR = os.environ.get("DBT_TARGET_DIR", "./target")

@app.route("/health/dbt/tests")
def test_health():
    try:
        # Read dbt run_results.json artifact
        artifact_path = os.path.join(DBT_TARGET_DIR, "run_results.json")
        if not os.path.exists(artifact_path):
            return jsonify({"status": "unknown", "reason": "no_artifact"}), 404

        with open(artifact_path) as f:
            results = json.load(f)

        # Check artifact freshness
        generated_at = datetime.fromisoformat(
            results["metadata"]["generated_at"].replace("Z", "+00:00")
        )
        age_hours = (datetime.now(timezone.utc) - generated_at).total_seconds() / 3600
        if age_hours > 25:
            return jsonify({
                "status": "degraded",
                "reason": "stale_artifact",
                "age_hours": round(age_hours, 1),
            }), 503

        # Count test failures
        test_results = [r for r in results["results"] if r.get("unique_id", "").startswith("test.")]
        failures = [r for r in test_results if r["status"] == "fail"]
        failure_rate = len(failures) / len(test_results) if test_results else 0

        if failures:
            return jsonify({
                "status": "degraded",
                "reason": "test_failures",
                "failure_count": len(failures),
                "failure_rate": round(failure_rate, 3),
                "failed_tests": [r["unique_id"] for r in failures[:10]],
            }), 503

        return jsonify({
            "status": "ok",
            "test_count": len(test_results),
            "artifact_age_hours": round(age_hours, 1),
        }), 200

    except Exception as e:
        return jsonify({"status": "down", "error": str(e)}), 503

if __name__ == "__main__":
    app.run(port=3008)

Step 3: Configure Vigilmon HTTP Monitors for dbt

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your dbt health endpoint: https://your-host.example.com/health/dbt/tests
4. Set the check interval to 30 minutes (after scheduled job windows)
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 10000ms
6. Under Alert channels, assign your data engineering Slack channel
7. Save the monitor

For dbt Cloud API availability, add a second monitor:

• URL: https://cloud.getdbt.com/api/v2/accounts/{account_id}/ (with Authorization header)
• Method: GET
• Expected: 200
• Interval: 5 minutes

Step 4: Source Freshness Monitoring

dbt's source freshness command checks whether your source tables are up to date. Integrate it into the same health sidecar:

#!/bin/bash
# check_freshness.sh — runs dbt source freshness and pings on success

dbt source freshness --output json --profiles-dir "${DBT_PROFILES_DIR}" \
  > /tmp/freshness_results.json 2>&1

# Check exit code — dbt exits non-zero when sources are stale
if [ $? -eq 0 ]; then
  curl -fsS "${VIGILMON_SOURCE_FRESHNESS_HEARTBEAT_URL}" > /dev/null
  echo "Source freshness OK — heartbeat sent"
else
  echo "Source freshness FAILED — heartbeat withheld"
  exit 1
fi

Create a dedicated Vigilmon heartbeat monitor for source freshness with the same interval as your freshness check schedule (commonly hourly or every 6 hours).

Summary

dbt failures are silent by default. External monitoring with Vigilmon catches job run gaps, model test failures, stale artifacts, and source freshness violations before they contaminate downstream analytics:

| Monitor Type | What It Covers | |---|---| | Heartbeat monitor (job) | dbt run + test completed on schedule | | HTTP monitor on /health/dbt/tests | Test failure rate, artifact freshness | | Heartbeat monitor (source freshness) | Upstream source data delivered on time | | HTTP monitor on dbt Cloud API | dbt Cloud service availability |

Get started free at vigilmon.online — your first dbt monitor is running in under two minutes.