Monitoring AWS Glue Jobs with Vigilmon

AWS Glue is Amazon's serverless ETL service for running data transformation jobs without managing infrastructure. It's powerful — but Glue jobs can fail silently, crawlers can stall for hours, and the built-in CloudWatch alerts require manual wiring that teams often skip. Vigilmon gives you a simple, external check to confirm your Glue pipelines are actually completing on schedule.

What You'll Set Up

• Cron heartbeat monitors to confirm Glue jobs complete on schedule
• HTTP endpoint monitor for a lightweight job-status API
• Alert channels for job failure and stalled crawler notifications
• Maintenance windows to suppress alerts during intentional ETL pauses

Prerequisites

• An AWS account with at least one AWS Glue ETL job or crawler configured
• AWS CLI configured, or access to the Lambda console
• A free Vigilmon account

Step 1: Add a Heartbeat Monitor for Each Glue Job

AWS Glue jobs run on a schedule — hourly, daily, or triggered by upstream events. If a job fails or is never triggered, there is no external signal unless you wire one up. Vigilmon's cron heartbeat gives you that signal.

1. Log in to vigilmon.online and click Add Monitor.
2. Select Cron Heartbeat as the monitor type.
3. Name the monitor after your job: e.g., glue-orders-etl-daily.
4. Set the expected ping interval to match your job schedule (e.g., 1440 minutes for a daily job).
5. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.
6. Click Save.

If your job runs but never pings, Vigilmon alerts after the expected interval passes — catching both failed runs and missed triggers.

Step 2: Ping Vigilmon at the End of Each Glue Job

AWS Glue jobs are Python (PySpark or Python Shell) scripts. Add a Vigilmon ping at the end of each job's success path:

Python Shell job

import urllib.request

def run_etl():
    # ... your ETL logic ...
    pass

if __name__ == "__main__":
    run_etl()
    # Ping Vigilmon on successful completion
    urllib.request.urlopen(
        "https://vigilmon.online/heartbeat/abc123",
        timeout=5
    )

PySpark (Glue ETL) job

import sys
import urllib.request
from awsglue.context import GlueContext
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)

# ... your transformation logic ...

# Signal success to Vigilmon
try:
    urllib.request.urlopen(
        "https://vigilmon.online/heartbeat/abc123",
        timeout=5
    )
except Exception:
    pass  # Don't fail the job if monitoring ping fails

Place the ping after the final commit or write operation. If the job crashes before reaching that line, Vigilmon never receives the ping and triggers an alert.

Step 3: Monitor Glue Crawlers via a Lambda Proxy

AWS Glue Crawlers don't run user code, so you can't embed a heartbeat directly. Instead, use a lightweight AWS Lambda function that runs after each crawler completes (via EventBridge) and pings Vigilmon.

Create a Lambda function named glue-crawler-vigilmon-ping:

import urllib.request

HEARTBEAT_URL = "https://vigilmon.online/heartbeat/crawler-abc456"

def handler(event, context):
    detail = event.get("detail", {})
    state = detail.get("state", "")

    if state == "Succeeded":
        urllib.request.urlopen(HEARTBEAT_URL, timeout=5)

    return {"status": "ok", "crawler_state": state}

Wire the EventBridge rule to trigger on crawler completion:

{
  "source": ["aws.glue"],
  "detail-type": ["Glue Crawler State Change"],
  "detail": {
    "crawlerName": ["my-s3-crawler"],
    "state": ["Succeeded"]
  }
}

Set the Vigilmon heartbeat interval to slightly more than your crawler's typical run time plus schedule interval. A crawler that runs every 6 hours should have a 360-minute heartbeat interval (plus a grace buffer of 30–60 minutes).

Step 4: Expose a Job Status HTTP Endpoint

For more detailed monitoring, deploy a small API Gateway + Lambda endpoint that queries the AWS Glue API for recent job run status and returns an HTTP 200 if healthy or 503 if the last run failed:

import boto3

glue = boto3.client("glue")

def handler(event, context):
    job_name = event.get("pathParameters", {}).get("job_name", "")

    response = glue.get_job_runs(JobName=job_name, MaxResults=1)
    runs = response.get("JobRuns", [])

    if not runs:
        return {"statusCode": 503, "body": "No runs found"}

    last_run = runs[0]
    job_state = last_run.get("JobRunState", "UNKNOWN")

    if job_state in ("SUCCEEDED",):
        return {"statusCode": 200, "body": f"Last run: {job_state}"}
    else:
        return {"statusCode": 503, "body": f"Last run: {job_state}"}

Once deployed, add an HTTP monitor in Vigilmon:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter the API Gateway URL: https://abc123.execute-api.us-east-1.amazonaws.com/prod/glue-status/orders-etl.
3. Set Expected HTTP status to 200.
4. Set Check interval to 5 minutes.
5. Click Save.

Step 5: Configure Alert Channels and Escalation

1. Go to Alert Channels in Vigilmon and add Slack, PagerDuty, or email.
2. For jobs that feed downstream reports or dashboards, set Consecutive failures before alert to 1 — a single missed ETL run can cascade.
3. For less critical jobs, set it to 2 to absorb transient Glue capacity delays.
4. Use Maintenance windows when you intentionally pause ETL pipelines:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 120}'

Step 6: Validate Your Setup End-to-End

Run a test Glue job and confirm the ping arrives:

# Trigger a Glue job via CLI
aws glue start-job-run --job-name orders-etl

# Watch the run status
aws glue get-job-runs --job-name orders-etl \
  --query 'JobRuns[0].{State:JobRunState,Start:StartedOn}'

While the job runs, open the Vigilmon dashboard and check that the heartbeat timestamp updates after the job completes. If it doesn't, check your Glue job logs in CloudWatch for Python errors near the urlopen call.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat | Each Glue ETL job | Failed run, missed trigger, stalled job | | Cron heartbeat (via Lambda) | Each Glue Crawler | Crawler failure, EventBridge misconfiguration | | HTTP endpoint | Job status Lambda | Last run in FAILED / STOPPED state |

AWS Glue handles the heavy lifting of serverless ETL, but it won't tell you when something silently stops working. With Vigilmon heartbeats wired into each job and a status API for quick checks, you get visibility into your entire data pipeline without instrumenting every CloudWatch metric manually.