Monitoring Project Nessie with Vigilmon

Project Nessie brings Git-style branch, tag, and merge semantics to your data lake. Teams use it to create isolated feature branches for data experiments, roll back bad schema changes, and maintain separate development, staging, and production environments — all pointing to the same object storage. But when Nessie goes down, every query engine connected to it (Spark, Trino, Flink, Dremio) loses access to its Iceberg tables immediately.

Vigilmon gives you external visibility into Nessie's REST API availability, version store backend health, commit throughput, and garbage collection before a silent failure becomes a data pipeline outage.

Why Nessie Monitoring Matters

Nessie failures are invisible to end users until the first query fails. A Nessie outage means:

• All Iceberg reads and writes fail — query engines cannot resolve table locations without the catalog
• DDL operations block — CREATE TABLE, ALTER TABLE, and DROP TABLE can no longer commit
• Branch proliferation goes undetected — teams forget to merge or delete feature branches, bloating the version store
• GC lag accumulates — expired content references pile up in object storage if the GC job silently fails

External monitoring with Vigilmon catches these failure modes before your data engineers start filing tickets asking why Trino is returning "table not found."

What You'll Set Up

• HTTP uptime monitor for the Nessie REST API (port 19120)
• Version store backend health check (RocksDB or PostgreSQL)
• Commit throughput and latency health endpoint
• GC job completion heartbeat
• Active branch count alert
• Alert routing for catalog and backend failures

Step 1: Monitor the Nessie REST API

Nessie exposes a health-checkable config endpoint at GET /api/v1/config. This endpoint returns Nessie's cluster configuration and is a reliable liveness probe — if it responds 200, the catalog API is accepting connections.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Nessie API URL: http://your-nessie-host:19120/api/v1/config
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Response body contains, enter "defaultBranch" (this field is always present in a valid Nessie config response).
7. Set Response time threshold to 5000ms.
8. Click Save.

If Nessie runs behind a reverse proxy or TLS terminator, use the public HTTPS URL instead. The /api/v1/config endpoint requires no authentication in default Nessie configurations.

Step 2: Build a Version Store Backend Health Endpoint

Nessie's pluggable version store backend (RocksDB for single-node, PostgreSQL for production HA) is the most common source of silent failures. Add a sidecar health endpoint that probes backend connectivity:

Python Health Sidecar (FastAPI)

# nessie_health.py
import os
import httpx
import psycopg2
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

NESSIE_API = os.environ.get("NESSIE_API_URL", "http://localhost:19120")
PG_DSN = os.environ.get("NESSIE_PG_DSN", "")  # only set in PostgreSQL deployments

@app.get("/health/nessie")
async def nessie_health():
    errors = []

    # Check Nessie REST API
    try:
        async with httpx.AsyncClient(timeout=5) as client:
            resp = await client.get(f"{NESSIE_API}/api/v1/config")
            if resp.status_code != 200:
                errors.append(f"nessie_api: HTTP {resp.status_code}")
    except Exception as e:
        errors.append(f"nessie_api: {e}")

    # Check PostgreSQL backend (skip for RocksDB single-node)
    if PG_DSN:
        try:
            conn = psycopg2.connect(PG_DSN, connect_timeout=3)
            cur = conn.cursor()
            cur.execute("SELECT 1")
            conn.close()
        except Exception as e:
            errors.append(f"postgres_backend: {e}")

    if errors:
        return JSONResponse(status_code=503, content={"status": "down", "errors": errors})
    return {"status": "ok", "backend": "postgres" if PG_DSN else "rocksdb"}


@app.get("/health/nessie/commits")
async def commit_health():
    """Check that Nessie can perform a catalog read (proxy for commit path health)."""
    try:
        async with httpx.AsyncClient(timeout=10) as client:
            resp = await client.get(f"{NESSIE_API}/api/v1/trees")
            if resp.status_code != 200:
                return JSONResponse(status_code=503, content={
                    "status": "down", "reason": f"trees_api_http_{resp.status_code}"
                })
            data = resp.json()
            branch_count = len(data.get("references", []))
            return {"status": "ok", "branch_count": branch_count}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Run this alongside your Nessie process on port 8099:

uvicorn nessie_health:app --host 0.0.0.0 --port 8099

Add a Vigilmon monitor for each endpoint:

• http://your-nessie-host:8099/health/nessie — API + backend connectivity
• http://your-nessie-host:8099/health/nessie/commits — branch/commit path health

Step 3: Heartbeat Monitoring for Nessie GC Jobs

Nessie's garbage collector removes expired content references from the version store and prevents orphaned data accumulating in object storage. GC jobs typically run as a scheduled task (cron or Nessie's built-in GC CLI). If GC silently stops running, object storage bills climb and version store cleanup halts.

Set up a heartbeat monitor to verify GC completion:

1. In Vigilmon, click Add Monitor → Heartbeat.
2. Set the name: nessie-gc-job.
3. Set the expected interval to match your GC schedule (e.g. 1440 minutes for daily GC).
4. Set grace period to 120 minutes.
5. Save and copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.

Wire the ping into your GC script:

#!/bin/bash
# nessie-gc.sh — run from cron or a scheduler

java -jar nessie-gc.jar \
  --nessie-uri http://localhost:19120/api/v1 \
  --iceberg-catalog nessie \
  --output-dir /tmp/nessie-gc-output

# Signal success to Vigilmon
if [ $? -eq 0 ]; then
  curl -s "${VIGILMON_HEARTBEAT_URL}" > /dev/null
else
  echo "Nessie GC failed — heartbeat NOT sent" >&2
  exit 1
fi

If the GC job crashes or is skipped, the heartbeat stops arriving and Vigilmon fires an alert after the grace period.

Step 4: Monitor Branch Count for Branch Explosion

Open branches in Nessie consume version store space and increase merge conflict surface area. Alert when the active branch count grows beyond a threshold your team sets during initial deployment.

Add a branch count check to your health sidecar:

@app.get("/health/nessie/branches")
async def branch_health():
    max_branches = int(os.environ.get("MAX_NESSIE_BRANCHES", "50"))
    try:
        async with httpx.AsyncClient(timeout=5) as client:
            resp = await client.get(f"{NESSIE_API}/api/v1/trees")
            data = resp.json()
            branches = [r for r in data.get("references", []) if r.get("type") == "BRANCH"]
            count = len(branches)
            if count > max_branches:
                return JSONResponse(status_code=503, content={
                    "status": "degraded",
                    "reason": "branch_explosion",
                    "branch_count": count,
                    "threshold": max_branches,
                })
            return {"status": "ok", "branch_count": count}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Add a Vigilmon monitor for http://your-nessie-host:8099/health/nessie/branches with expected status 200 and body containing "status":"ok".

Step 5: Configure Alert Channels and Routing

Open Vigilmon's Alert Channels settings and configure routing by severity:

| Monitor | Alert Channel | Suggested Priority | |---|---|---| | Nessie REST API /api/v1/config | Slack + PagerDuty | P1 — all catalog reads fail | | Backend health /health/nessie | Slack + PagerDuty | P1 — version store connectivity | | Commit path /health/nessie/commits | Slack | P2 — DDL operations failing | | Branch count /health/nessie/branches | Slack | P3 — hygiene alert | | GC heartbeat | Email | P3 — object storage accumulation |

Recommended settings for each monitor:

• Consecutive failures before alert: 2 — avoids false positives from brief network hiccups
• Response time threshold: 5000ms for API monitors (version store queries under load can be slow)
• Maintenance windows: suppress alerts during Nessie version upgrades and backend migrations

Step 6: TCP Port Monitor for Nessie API

Add a TCP-level monitor as a fast-path check that catches Nessie process crashes before the HTTP probe fires:

1. In Vigilmon, click Add Monitor → TCP Port.
2. Host: your-nessie-host, Port: 19120.
3. Interval: 1 minute.
4. Alert channel: same P1 channel as the REST API monitor.

A TCP monitor fires in under 60 seconds when the Nessie JVM crashes — faster than an HTTP probe that must wait for a full timeout.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP — REST API | :19120/api/v1/config | Nessie process down, port unreachable | | HTTP — backend health | :8099/health/nessie | RocksDB or PostgreSQL backend failure | | HTTP — commit path | :8099/health/nessie/commits | DDL operation failures, version store slowness | | HTTP — branch count | :8099/health/nessie/branches | Branch explosion from unmaintained feature branches | | Heartbeat — GC job | Heartbeat URL | GC job failure, orphaned object storage accumulation | | TCP — API port | :19120 | JVM crash, fast-path detection |

Project Nessie makes data lake table management as robust as Git — but like any infrastructure, it needs monitoring to stay healthy. With Vigilmon watching the REST API, version store backend, commit path, and GC jobs, your Spark and Trino jobs keep running even when something goes wrong upstream.

Get started free at vigilmon.online — your first Nessie monitor is running in under two minutes.