tutorial

How to Monitor Apache Hudi Data Lake Jobs with Vigilmon

Apache Hudi (Hadoop Upserts Deletes and Incrementals) brings ACID transactions, record-level updates, and incremental processing to your data lake. Whether y...

Apache Hudi (Hadoop Upserts Deletes and Incrementals) brings ACID transactions, record-level updates, and incremental processing to your data lake. Whether you run Hudi on AWS EMR, Databricks, or a self-managed Spark cluster, the jobs that write and compact Hudi tables are critical infrastructure — a failed upsert pipeline means your downstream analytics read stale or corrupt data without knowing it.

This tutorial shows you how to monitor Apache Hudi ingestion pipelines, table services (compaction, clustering, cleaning), and the REST-based timeline server using Vigilmon.

Why Hudi pipelines need external monitoring

Hudi jobs fail silently in ways that surprise teams moving from traditional RDBMS:

  • Failed upsert jobs — the Spark job exits with code 0 but writes zero rows because a precondition check failed; no one notices until a BI report shows yesterday's data
  • Compaction lag — merge-on-read (MOR) tables accumulate delta log files faster than compaction jobs run; query latency balloons over days
  • Cleaning job skips — old file versions pile up on S3/HDFS, storage costs spike, and the next compaction OOMs
  • Timeline server down — Hudi's embedded timeline server (used for conflict detection in multi-writer scenarios) goes unreachable and all concurrent writers fall back to serial mode, halving throughput

External monitoring — an independent probe that checks outcomes and endpoints — catches all of these before they show up in SLA misses.

What you'll need

  • An Apache Hudi pipeline (Spark on EMR, Databricks, or standalone)
  • A publicly reachable health endpoint or webhook callback (details below)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose a health endpoint for Hudi pipeline status

Hudi doesn't ship a built-in HTTP health endpoint, but you can add one in two common ways.

Option A: A lightweight sidecar HTTP server

Run a tiny HTTP server alongside your Hudi Spark driver that reads the last-write timestamp from the Hudi .hoodie metadata directory:

# hudi_health_server.py — run as a background thread inside your Spark driver
from http.server import BaseHTTPRequestHandler, HTTPServer
import json, time, threading
from pyspark.sql import SparkSession

LAST_COMMIT_TS = {"value": None, "status": "starting"}

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            age_seconds = None
            if LAST_COMMIT_TS["value"]:
                age_seconds = int(time.time() - LAST_COMMIT_TS["value"])
            body = json.dumps({
                "status": LAST_COMMIT_TS["status"],
                "last_commit_age_seconds": age_seconds
            })
            code = 200 if LAST_COMMIT_TS["status"] == "ok" else 503
            self.send_response(code)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(body.encode())
        else:
            self.send_response(404)
            self.end_headers()
    def log_message(self, *args):
        pass  # suppress request logs

def start_health_server(port=8765):
    server = HTTPServer(("0.0.0.0", port), HealthHandler)
    t = threading.Thread(target=server.serve_forever, daemon=True)
    t.start()

Call start_health_server() at the top of your Spark application. After each successful Hudi commit, update LAST_COMMIT_TS:

start_health_server(port=8765)

# ... your Hudi write logic ...
df.write.format("hudi") \
    .options(**hudi_options) \
    .mode("append") \
    .save(base_path)

# Mark success
import time
LAST_COMMIT_TS["value"] = time.time()
LAST_COMMIT_TS["status"] = "ok"

Option B: Write a status file to S3 and serve it via API Gateway

If your Spark drivers run on ephemeral EMR clusters with no stable IP, write a JSON status file to S3 after each run and expose it via a Lambda + API Gateway:

# Called after each successful Hudi commit
import boto3, json, time

s3 = boto3.client("s3")
payload = {
    "status": "ok",
    "last_commit": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    "table": "events_hudi",
    "commit_count": commit_count
}
s3.put_object(
    Bucket="your-monitoring-bucket",
    Key="hudi/status.json",
    Body=json.dumps(payload),
    ContentType="application/json"
)

A Lambda function reads this file and returns it as an HTTP response — giving Vigilmon a stable URL to probe even as your EMR clusters come and go.

Step 2: Monitor the Hudi REST Timeline Server

If you run Hudi in multi-writer mode, the embedded timeline server is critical for conflict detection. It exposes a REST endpoint at port 26754 by default:

GET http://<driver-host>:26754/v1/hoodie/timeline/listAllInstants

Add a TCP port monitor for it in Vigilmon:

  1. Log in to vigilmon.online → Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your driver hostname and port 26754
  4. Set check interval to 1 minute
  5. Save

For HTTP-level monitoring of the timeline API:

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://<driver-host>:26754/v1/hoodie/timeline/listAllInstants
  3. Expected status: 200
  4. Save

Step 3: Set up Vigilmon HTTP monitoring for the pipeline health endpoint

With your /health endpoint running on the Spark driver:

  1. Go to Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://<spark-driver-public-ip>:8765/health
  3. Expected status: 200
  4. (Optional) Response body keyword: "status":"ok"
  5. Check interval: 1 minute
  6. Save

Vigilmon now probes your Hudi pipeline health from multiple regions every minute. If the job crashes, times out, or starts returning 503, an incident is opened immediately.

Step 4: Monitor Hudi table storage metrics via a custom endpoint

Track compaction lag and storage growth by exposing table-level metrics:

from pyspark.sql import SparkSession
from pyhudi.hudi_commit_metadata import HoodieCommitMetadata

def get_table_metrics(spark, base_path):
    """Return dict of compaction lag and pending delta files."""
    # Read active timeline
    fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(
        spark._jsc.hadoopConfiguration()
    )
    client = spark._jvm.org.apache.hudi.client.common.HoodieSparkEngineContext(
        spark._jsc
    )
    # Simplified: check .hoodie directory for pending compaction instants
    hoodie_path = f"{base_path}/.hoodie"
    pending_compactions = len([
        f for f in fs.listStatus(
            spark._jvm.org.apache.hadoop.fs.Path(hoodie_path)
        )
        if str(f.getPath().getName()).endswith(".compaction.requested")
    ])
    return {"pending_compactions": pending_compactions}

Expose these via your health server:

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/metrics":
            metrics = get_table_metrics(spark, base_path)
            body = json.dumps(metrics)
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(body.encode())

Add a Vigilmon keyword monitor on /metrics that alerts if "pending_compactions" exceeds a threshold — a leading indicator of compaction lag before query performance degrades.

Step 5: Configure webhook alerts for Hudi job failures

When a Hudi ingestion job fails, you want an alert in seconds.

  1. In Vigilmon, go to Alert Channels → Add Channel → Webhook
  2. Enter your Slack or PagerDuty webhook URL
  3. Assign the channel to your Hudi monitors

You can also call Vigilmon's webhook directly from your Spark job's exception handler:

import requests

def alert_vigilmon(monitor_id: str, api_key: str, message: str):
    """Push an immediate alert to Vigilmon on pipeline failure."""
    requests.post(
        f"https://vigilmon.online/api/monitors/{monitor_id}/alert",
        headers={"Authorization": f"Bearer {api_key}"},
        json={"message": message, "severity": "critical"},
        timeout=5
    )

try:
    run_hudi_ingestion(spark, df, base_path, hudi_options)
except Exception as e:
    alert_vigilmon(MONITOR_ID, API_KEY, f"Hudi ingest failed: {e}")
    raise

Step 6: Create a status page for your data platform

If multiple teams depend on your Hudi tables, a public status page reduces support noise when something is genuinely down.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name it (e.g. "Data Platform")
  3. Add monitors: pipeline health, timeline server TCP, metrics endpoint
  4. Publish

Share the URL in your data team's Slack channel and internal wiki so analysts self-serve instead of opening tickets.

Monitoring checklist for Apache Hudi

| What to monitor | Vigilmon monitor type | Alert threshold | |---|---|---| | Pipeline /health endpoint | HTTP | Non-200 response or timeout | | Timeline server (port 26754) | TCP Port | Connection refused | | Timeline server REST API | HTTP | Non-200 response | | Pending compactions metric | HTTP keyword | pending_compactions > 5 | | Last commit age | HTTP keyword | last_commit_age_seconds > 3600 |

Summary

Apache Hudi brings database semantics to your data lake — but the jobs maintaining those semantics are as critical as any production service. With Vigilmon you get:

  • 1-minute HTTP probes on your pipeline health endpoint
  • TCP monitoring of the Hudi timeline server
  • Webhook alerts delivered in seconds when ingestion fails
  • A public status page that keeps stakeholders informed

Start monitoring your Hudi pipelines free →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles