Monitoring Azure Data Factory Pipelines with Vigilmon

Azure Data Factory is Microsoft's cloud ETL and data orchestration service. It runs pipelines on triggers — scheduled, event-based, or on-demand — and integrates with hundreds of data sources. But when an ADF pipeline fails or a trigger fires without a run completing, the alerts are opt-in and often misconfigured. Vigilmon gives you a dead-simple external health check that confirms your pipelines are actually finishing on schedule.

What You'll Set Up

• Cron heartbeat monitors for each scheduled ADF pipeline
• HTTP endpoint monitor backed by an Azure Function that checks pipeline run status
• Alert channels for pipeline failures and missed schedule windows

Prerequisites

• An Azure subscription with at least one Azure Data Factory instance
• An ADF pipeline with a schedule trigger configured
• Azure CLI installed, or access to the Azure portal
• A free Vigilmon account

Step 1: Add a Heartbeat Monitor for Each Scheduled Pipeline

ADF pipelines run on a trigger schedule. If a pipeline run fails, times out, or the trigger stops firing, there's no external signal by default. Vigilmon's cron heartbeat lets you verify completion from outside Azure.

1. Log in to vigilmon.online and click Add Monitor.
2. Select Cron Heartbeat as the monitor type.
3. Name the monitor after your pipeline: e.g., adf-sales-etl-daily.
4. Set the expected ping interval to match your trigger schedule (e.g., 1440 minutes for a daily pipeline).
5. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.
6. Click Save.

If the pipeline runs but never pings, Vigilmon alerts you — catching both failed runs and trigger configuration drift.

Step 2: Ping Vigilmon at the End of Each ADF Pipeline

ADF pipelines support a Web Activity that makes an HTTP call to any URL. Add a Web Activity as the last step in each pipeline to ping Vigilmon on successful completion.

1. Open your pipeline in the ADF Studio.
2. Drag a Web activity onto the canvas and connect it as the final step.
3. Configure the Web Activity:
   • URL: https://vigilmon.online/heartbeat/abc123
   • Method: GET
   • Timeout: 00:00:10
4. Publish the pipeline changes.

For pipelines with a failure path (failure branch after a copy activity), do not connect the Web Activity to the failure branch — only wire it to the success path. Vigilmon fires an alert if the success ping never arrives.

For conditional pipelines, add the Web Activity inside an If Condition activity's true branch, after all transformations succeed:

[Source Copy Activity] → [Transform Activity] → [Success Web Activity → Vigilmon]
                                ↓
                         [Failure branch → Log failure to storage]

Step 3: Deploy an Azure Function for Pipeline Status Checks

For more detailed monitoring, deploy a lightweight Azure Function that queries the ADF REST API and returns 200 when the last pipeline run succeeded, or 503 when it failed.

Create a Python Azure Function:

import os
import logging
import azure.functions as func
from azure.identity import DefaultAzureCredential
from azure.mgmt.datafactory import DataFactoryManagementClient

SUBSCRIPTION_ID = os.environ["AZURE_SUBSCRIPTION_ID"]
RESOURCE_GROUP = os.environ["ADF_RESOURCE_GROUP"]
FACTORY_NAME = os.environ["ADF_FACTORY_NAME"]

def main(req: func.HttpRequest) -> func.HttpResponse:
    pipeline_name = req.params.get("pipeline")
    if not pipeline_name:
        return func.HttpResponse("pipeline param required", status_code=400)

    try:
        credential = DefaultAzureCredential()
        client = DataFactoryManagementClient(credential, SUBSCRIPTION_ID)

        runs = client.pipeline_runs.query_by_factory(
            RESOURCE_GROUP,
            FACTORY_NAME,
            {
                "lastUpdatedAfter": "2000-01-01T00:00:00Z",
                "lastUpdatedBefore": "2099-01-01T00:00:00Z",
                "filters": [
                    {"operand": "PipelineName", "operator": "Equals", "values": [pipeline_name]}
                ],
            }
        )
        run_list = list(runs.value)
        if not run_list:
            return func.HttpResponse("No runs found", status_code=503)

        latest = sorted(run_list, key=lambda r: r.run_start, reverse=True)[0]

        if latest.status == "Succeeded":
            return func.HttpResponse(f"OK: last run succeeded at {latest.run_end}", status_code=200)
        else:
            return func.HttpResponse(f"FAIL: last run status is {latest.status}", status_code=503)

    except Exception as e:
        logging.error(str(e))
        return func.HttpResponse("Error querying ADF", status_code=503)

Deploy with:

az functionapp create \
  --resource-group myResourceGroup \
  --consumption-plan-location eastus \
  --runtime python \
  --runtime-version 3.11 \
  --functions-version 4 \
  --name adf-status-api \
  --storage-account mystorageacct

func azure functionapp publish adf-status-api

Add a Vigilmon HTTP monitor pointing to your Function App URL:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: https://adf-status-api.azurewebsites.net/api/pipeline-health?pipeline=sales-etl.
3. Set Expected HTTP status to 200.
4. Set Check interval to 5 minutes.
5. Click Save.

Step 4: Monitor the ADF Integration Runtime

Self-hosted Integration Runtimes (SHIR) run on on-premises or VM infrastructure. If the SHIR node goes offline, all pipelines using it will fail. Add TCP and HTTP monitors for the SHIR host:

1. Click Add Monitor → TCP Port.
2. Enter the SHIR host and port: shir-host.yourdomain.com:8060.
3. Set Check interval to 1 minute.
4. Click Save.

For VM-hosted SHIRs, also monitor the VM's HTTP health endpoint if you have one, or simply watch the VM's SSH port (22 or 3389) as a liveness check.

Step 5: Configure Alert Channels and Maintenance Windows

1. Go to Alert Channels in Vigilmon and add Slack, Teams, or email.
2. For pipelines that feed reports due first thing in the morning, set Consecutive failures before alert to 1 — any missed run creates a data gap that someone will notice.
3. Suppress alerts during Azure maintenance windows or planned pipeline pauses:

# Create a maintenance window for a 2-hour ADF migration
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 120}'

Step 6: Verify End-to-End

Trigger your ADF pipeline manually and confirm the Vigilmon heartbeat updates:

# Trigger via Azure CLI
az datafactory pipeline create-run \
  --resource-group myResourceGroup \
  --factory-name myADF \
  --name sales-etl

# Watch the run status
az datafactory pipeline-run show \
  --resource-group myResourceGroup \
  --factory-name myADF \
  --run-id <runId>

After the run completes, open the Vigilmon dashboard and confirm the heartbeat timestamp updated. If it didn't, check the ADF pipeline's Web Activity output in the ADF monitoring view for connectivity errors.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat | Each ADF pipeline (via Web Activity) | Failed run, missed trigger, timeout | | HTTP endpoint | Azure Function status API | Last run in Failed / Cancelled state | | TCP port | Self-hosted Integration Runtime | SHIR node offline |

Azure Data Factory makes complex data orchestration manageable — but it doesn't call you when a daily ETL quietly fails at 2 AM. With Vigilmon heartbeats wired into each pipeline's success path and a status API watching the last run state, you get confident visibility into your entire ADF estate without digging through portal alerts every morning.