tutorial

How to Monitor Apache Iceberg Table Services with Vigilmon

Apache Iceberg is the open table format that brings snapshot isolation, hidden partitioning, and schema evolution to cloud data lakes. Used by Netflix, Apple...

Apache Iceberg is the open table format that brings snapshot isolation, hidden partitioning, and schema evolution to cloud data lakes. Used by Netflix, Apple, and thousands of teams running Trino, Spark, Flink, and DuckDB, Iceberg tables are central data infrastructure — and when the catalog service, REST catalog, or compaction jobs go down, query engines silently fail or return wrong results.

This tutorial shows you how to monitor Apache Iceberg REST catalogs, compaction services, Flink streaming ingestion, and table freshness using Vigilmon.

Why Iceberg infrastructure needs external monitoring

Iceberg's architecture introduces failure modes that standard job schedulers miss:

  • REST catalog unavailability — when the Iceberg REST catalog goes down, Trino and Spark can't commit new snapshots or read table metadata; jobs fail with RESTException: Service Unavailable but no alert fires until queries start returning errors
  • Metadata table bloat — the metadata/ prefix accumulates snapshot files; without regular expiration jobs, metadata reads slow to a crawl over weeks
  • Orphaned files — failed writes leave data files on S3 not referenced by any snapshot; storage costs grow invisibly
  • Flink streaming job stall — a Flink job that writes to Iceberg via the Flink sink stops checkpointing; data stops arriving but the job appears "running" in the Flink dashboard
  • Catalog backend failures — Iceberg catalogs backed by AWS Glue, Hive Metastore, or JDBC go unhealthy; all table operations fail while the Spark/Trino services themselves appear healthy

External monitoring that probes the REST catalog and checks table freshness catches these before they appear in analyst support tickets.

What you'll need

  • An Iceberg deployment with a REST catalog (Polaris, Nessie, Iceberg REST, or AWS Glue)
  • Python 3.9+ for custom health endpoints
  • A free Vigilmon account

Step 1: Monitor the Iceberg REST Catalog

The Iceberg REST catalog specification defines a /v1/config endpoint that returns catalog configuration. This is the best health check: if the catalog returns 200, it's serving metadata.

Vigilmon can probe this directly:

  1. Log in to vigilmon.online → Monitors → New Monitor → HTTP / HTTPS
  2. URL: https://your-iceberg-catalog.example.com/v1/config
  3. Expected status: 200
  4. Check interval: 1 minute
  5. Save

For Polaris (Apache Polaris REST catalog), the endpoint is at:

GET https://your-polaris-host/api/catalog/v1/config

For Nessie, monitor the v2 API:

GET https://your-nessie-host/api/v2/config

If your catalog requires authentication, add an Authorization header in Vigilmon's monitor settings.

Step 2: Monitor catalog namespace and table listing

Beyond the config endpoint, verify that the catalog can actually serve table metadata:

# iceberg_catalog_health.py — lightweight Flask service
from flask import Flask, jsonify
from pyiceberg.catalog.rest import RestCatalog
import os, time

app = Flask(__name__)

catalog = RestCatalog(
    name="prod",
    **{
        "uri": os.environ["ICEBERG_REST_URI"],
        "credential": os.environ["ICEBERG_CREDENTIAL"],
        "warehouse": os.environ["ICEBERG_WAREHOUSE"],
    }
)

@app.route("/health")
def health():
    try:
        start = time.time()
        namespaces = catalog.list_namespaces()
        latency_ms = int((time.time() - start) * 1000)
        return jsonify({
            "status": "ok",
            "namespace_count": len(namespaces),
            "catalog_latency_ms": latency_ms
        }), 200
    except Exception as e:
        return jsonify({"status": "error", "error": str(e)}), 503

@app.route("/table-health/<namespace>/<table_name>")
def table_health(namespace, table_name):
    try:
        table = catalog.load_table((namespace, table_name))
        snapshot = table.current_snapshot()
        if snapshot is None:
            return jsonify({"status": "no_snapshot"}), 503

        snapshot_age_seconds = int(
            (time.time() * 1000 - snapshot.timestamp_ms) / 1000
        )
        max_age = int(os.environ.get("MAX_SNAPSHOT_AGE_SECONDS", 3600))
        status = "ok" if snapshot_age_seconds < max_age else "stale"
        code = 200 if status == "ok" else 503

        return jsonify({
            "status": status,
            "snapshot_id": snapshot.snapshot_id,
            "snapshot_age_seconds": snapshot_age_seconds,
            "record_count": snapshot.summary.get("added-records", "unknown")
        }), code
    except Exception as e:
        return jsonify({"status": "error", "error": str(e)}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8090)

Deploy this service alongside your catalog. Then add monitors in Vigilmon:

  • /health — catalog connectivity check
  • /table-health/analytics/events — freshness check per critical table

Step 3: Monitor Flink Iceberg streaming jobs

Flink jobs writing to Iceberg via the flink-connector-iceberg fail silently when the job manager loses task managers. The Flink REST API exposes job health:

GET http://<flink-jobmanager>:8081/jobs/overview

Build a thin wrapper that checks for stuck or failed jobs:

# flink_iceberg_health.py
from flask import Flask, jsonify
import requests, os, time

app = Flask(__name__)
FLINK_BASE = os.environ.get("FLINK_JOBMANAGER_URL", "http://flink-jobmanager:8081")
ICEBERG_JOB_NAME = os.environ.get("FLINK_JOB_NAME", "iceberg-events-sink")

@app.route("/flink-health")
def flink_health():
    try:
        resp = requests.get(f"{FLINK_BASE}/jobs/overview", timeout=5)
        resp.raise_for_status()
        jobs = resp.json().get("jobs", [])

        target = next(
            (j for j in jobs if ICEBERG_JOB_NAME in j.get("name", "")),
            None
        )
        if target is None:
            return jsonify({"status": "job_not_found", "name": ICEBERG_JOB_NAME}), 503

        state = target.get("state", "UNKNOWN")
        if state != "RUNNING":
            return jsonify({"status": "not_running", "state": state}), 503

        duration_ms = target.get("duration", 0)
        return jsonify({
            "status": "ok",
            "job_state": state,
            "duration_hours": round(duration_ms / 3_600_000, 2)
        }), 200
    except requests.RequestException as e:
        return jsonify({"status": "flink_unreachable", "error": str(e)}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8091)

Add this as a Vigilmon HTTP monitor on /flink-health.

Step 4: Monitor Iceberg compaction jobs

Iceberg uses compaction (rewriting data files) to maintain read performance. Monitor your compaction service with a status endpoint:

# In your compaction runner script
import boto3, json, time

def update_compaction_status(table_name: str, status: str, files_compacted: int):
    """Write compaction status to S3 for Vigilmon-accessible Lambda to serve."""
    s3 = boto3.client("s3")
    s3.put_object(
        Bucket=os.environ["STATUS_BUCKET"],
        Key=f"iceberg/compaction/{table_name}.json",
        Body=json.dumps({
            "status": status,
            "files_compacted": files_compacted,
            "completed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
        }),
        ContentType="application/json"
    )

A companion Lambda reads the S3 object and returns it as HTTP — Vigilmon probes this URL to confirm compaction is running regularly.

Step 5: Configure Vigilmon alerts

Set up alert channels so incidents reach your team immediately:

  1. Go to Alert Channels → Add Channel
  2. Choose Webhook for Slack or PagerDuty: 
    {
  "monitor_name": "Iceberg REST Catalog /v1/config",
  "status": "down",
  "url": "https://your-catalog.example.com/v1/config",
  "started_at": "2026-07-01T14:00:00Z"
}
  3. Or choose Email for on-call distribution
  4. Assign the alert channel to all Iceberg monitors

Add Vigilmon alerts directly from your compaction job's failure handler:

import requests

def alert_on_compaction_failure(table: str, error: Exception):
    api_key = os.environ["VIGILMON_API_KEY"]
    monitor_id = os.environ["VIGILMON_COMPACTION_MONITOR_ID"]
    try:
        requests.post(
            f"https://vigilmon.online/api/monitors/{monitor_id}/incidents",
            headers={"Authorization": f"Bearer {api_key}"},
            json={
                "message": f"Iceberg compaction failed for {table}: {error}",
                "severity": "high"
            },
            timeout=5
        )
    except Exception:
        pass

Step 6: Monitor table snapshot expiration jobs

Old Iceberg snapshots accumulate until the expireSnapshots procedure runs. Track expiration job health:

@app.route("/expiration-health")
def expiration_health():
    """Return status of last snapshot expiration run."""
    try:
        # Read from a status file your expiration job writes
        with open("/var/run/iceberg/last_expiration.json") as f:
            data = json.load(f)

        last_run = data.get("completed_at")
        if last_run:
            from datetime import datetime, timezone
            dt = datetime.fromisoformat(last_run.replace("Z", "+00:00"))
            age_hours = (datetime.now(timezone.utc) - dt).total_seconds() / 3600
            if age_hours > 24:
                return jsonify({"status": "stale", "last_run_hours_ago": age_hours}), 503
        return jsonify({"status": "ok", **data}), 200
    except FileNotFoundError:
        return jsonify({"status": "no_status_file"}), 503

Step 7: Create a data platform status page

  1. In Vigilmon → Status Pages → New Status Page
  2. Add monitors:
    • Iceberg REST Catalog /v1/config
    • Catalog health endpoint
    • Table freshness per critical table
    • Flink streaming job health
    • Compaction service status
  3. Publish and share with your data consumers

Monitoring checklist for Apache Iceberg

| What to monitor | Vigilmon type | Alert condition | |---|---|---| | REST catalog /v1/config | HTTP | Non-200 response | | Catalog namespace listing | HTTP | Non-200 or high latency | | Table snapshot freshness | HTTP | "status":"stale" or non-200 | | Flink job state | HTTP | state != RUNNING | | Compaction service | HTTP | Non-200 or stale timestamp | | Snapshot expiration job | HTTP | Last run > 24 hours ago | | Catalog backend TCP port | TCP Port | Connection refused |

Summary

Apache Iceberg provides the metadata layer that makes your data lake queryable at scale — but the catalog, streaming writers, and maintenance jobs that keep it healthy need their own monitoring. With Vigilmon you get:

  • 1-minute REST catalog probes that catch catalog downtime before query engines fail
  • Table freshness checks that alert before analysts notice stale data
  • Flink streaming job health monitored independently of the Flink dashboard
  • Instant alerts routed to Slack, PagerDuty, or email

Start monitoring your Iceberg tables for free →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles