tutorial

How to Monitor LakeFS with Vigilmon (Data Versioning)

LakeFS brings Git-like version control to your data lake — branching, committing, merging, and rolling back petabytes of data on S3, GCS, or Azure Blob Stora...

LakeFS brings Git-like version control to your data lake — branching, committing, merging, and rolling back petabytes of data on S3, GCS, or Azure Blob Storage. Data engineering teams depend on LakeFS for reproducible pipelines, safe data migrations, and isolated experiment environments. When LakeFS is down, those pipelines fail silently or with confusing errors, and engineers can't commit or merge data branches.

In this tutorial you'll set up external uptime monitoring for your LakeFS deployment using Vigilmon — free tier, no credit card required.

Why LakeFS needs external monitoring

LakeFS runs as a stateful API service backed by a database (PostgreSQL) and an object store. Its internal health endpoint tells you if the process is alive. But there's a class of failures that the process itself can't detect:

  • Database connection pool exhausted — LakeFS responds to /api/v1/healthcheck but all DB connections are held; metadata operations hang indefinitely
  • Object store credentials expired — the LakeFS process is healthy, but any operation that touches S3 returns errors; /healthcheck still returns 200
  • Reverse proxy or load balancer routing breaks — LakeFS is fine but traffic can't reach it; pipelines fail at the client level
  • SSL certificate expires — clients see TLS errors while the LakeFS process is running normally
  • Database replica lag causes stale reads — reads succeed but return outdated commit metadata, causing pipeline reproducibility issues without obvious errors
  • Kubernetes service de-registers endpoints after a rolling deploy — old pod is gone, new pod isn't ready yet, but the service thinks everything is fine for a window

External monitoring from Vigilmon closes these gaps by checking what your data pipelines actually see when they reach out to LakeFS.

What you'll need

  • A running LakeFS deployment (Docker, Kubernetes, or managed)
  • The LakeFS API endpoint accessible over HTTP/HTTPS
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Verify your LakeFS health endpoint

LakeFS exposes a health check endpoint at /api/v1/healthcheck:

curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/api/v1/healthcheck
# Returns 204 No Content when healthy

Note: LakeFS health returns 204 (No Content), not 200. Make sure your monitoring tool is configured to accept 204 as a success.

You can also check storage connectivity:

curl -s http://localhost:8000/api/v1/setup_state | python3 -m json.tool
# Returns the current setup/initialization state

Starting LakeFS with Docker

docker run \
  --name lakefs \
  -p 8000:8000 \
  -e LAKEFS_DATABASE_TYPE=postgres \
  -e LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING="postgres://lakefs:password@postgres/lakefs" \
  -e LAKEFS_AUTH_ENCRYPT_SECRET_KEY="your-32-char-secret-key-here" \
  -e LAKEFS_BLOCKSTORE_TYPE=s3 \
  -e LAKEFS_BLOCKSTORE_S3_REGION=us-east-1 \
  -e AWS_ACCESS_KEY_ID=your-key \
  -e AWS_SECRET_ACCESS_KEY=your-secret \
  treeverse/lakefs:latest \
  run

Once running, verify:

curl http://localhost:8000/api/v1/healthcheck  # Should return 204

Step 2: Create an HTTP monitor in Vigilmon

Log in to Vigilmon and create your primary health monitor:

  1. Click Monitors → New Monitor
  2. Set the Type to HTTP
  3. Enter your LakeFS URL: https://lakefs.yourdomain.com/api/v1/healthcheck
  4. Set Check interval to 1 minute
  5. Set Expected status to 204
  6. Set Alert after to 1 failure
  7. Click Save

What this monitor catches

| Failure scenario | Vigilmon response | |-----------------|------------------| | LakeFS process crashed | Connection refused → alert | | PostgreSQL connection failure | Depends on LakeFS error handling — may return 500 | | Reverse proxy misconfigured | HTTP 502/504 → alert | | OOM kill | Connection refused → alert | | DNS failure | DNS error → alert | | TLS certificate expired | SSL handshake failure → alert |

Step 3: Add TCP monitoring for the database port

If your PostgreSQL database is on a separate host, monitor the database port directly to distinguish app-level failures from database-level failures:

  1. Click Monitors → New Monitor
  2. Set the Type to TCP
  3. Enter: postgres.yourdomain.com:5432
  4. Set Check interval: 1 minute
  5. Click Save

When a LakeFS alert fires, you can immediately cross-reference the database TCP monitor to know if the database is the root cause.

Step 4: Monitor the LakeFS UI availability

LakeFS ships with a web UI on the same port as the API. If your team uses the UI for branch management and data exploration, add a separate monitor:

  1. Click Monitors → New Monitor
  2. Set the Type to HTTP
  3. Enter: https://lakefs.yourdomain.com/
  4. Set Expected status: 200
  5. Set Keyword must contain: lakeFS (the UI's HTML title)
  6. Click Save

This catches cases where the API is healthy but the static assets fail to serve — a common failure mode when Nginx misconfiguration blocks frontend routes but passes API routes.

Step 5: Monitor repository and storage connectivity

LakeFS's /api/v1/repositories endpoint lists accessible repositories and requires both database and object store connectivity. Use it as a deeper health check:

  1. Click Monitors → New Monitor
  2. Set the Type to HTTP
  3. Enter: https://lakefs.yourdomain.com/api/v1/repositories
  4. Set HTTP method: GET
  5. Add a Basic auth header (or bearer token) for your LakeFS admin user
  6. Set Expected status: 200
  7. Set Keyword must contain: results (the JSON response field)
  8. Click Save

This monitor will fail if:

  • Object store credentials expire (LakeFS can list repos from DB but can't verify storage access)
  • Database has data consistency issues
  • Your LakeFS admin account credentials have changed

Step 6: Configure alerts

Slack alerts — navigate to Alerts → Slack and configure an incoming webhook:

# Alert to #data-platform or #lakefs-ops channel
# Webhook URL format: https://hooks.slack.com/services/T.../B.../...

PagerDuty — for production LakeFS deployments where pipeline failures have SLA implications:

{
  "routing_key": "YOUR_PD_INTEGRATION_KEY",
  "event_action": "trigger",
  "payload": {
    "summary": "LakeFS API is unreachable",
    "severity": "critical",
    "source": "vigilmon",
    "component": "lakefs",
    "group": "data-platform"
  }
}

Example Vigilmon alert payload:

{
  "monitor_name": "LakeFS healthcheck",
  "status": "down",
  "url": "https://lakefs.yourdomain.com/api/v1/healthcheck",
  "started_at": "2026-04-20T08:15:00Z",
  "duration_seconds": 300
}

Step 7: Triage runbook when alerts fire

# 1. Is the LakeFS process running?
docker ps | grep lakefs
# or systemd:
systemctl status lakefs

# 2. Check LakeFS logs for database and storage errors
docker logs lakefs --tail 100 | grep -E "error|ERROR|failed|FATAL" 

# 3. Test database connectivity from the LakeFS host
psql -h postgres.yourdomain.com -U lakefs -d lakefs -c "SELECT 1"

# 4. Test S3 connectivity
aws s3 ls s3://your-lakefs-bucket/ --region us-east-1

# 5. Test the health endpoint directly (bypass proxy)
curl -v http://localhost:8000/api/v1/healthcheck

# 6. Check for OOM kills
dmesg | grep -i "killed process" | tail -10
journalctl --since "30 minutes ago" | grep -i "oom"

# 7. Test the full API chain
curl -s -u admin:password http://localhost:8000/api/v1/repositories

Decision tree:

  • Process not running → OOM kill or crash; check dmesg and LakeFS logs; restart and investigate memory usage
  • Process running, direct health check fails → LakeFS internal error; check logs for DB connectivity or S3 errors
  • Direct health check passes, Vigilmon shows down → Proxy or network issue; check nginx/load balancer configuration
  • Health passes, repositories endpoint fails → Object store connectivity issue (expired credentials, IAM role change, S3 bucket policy change)

Step 8: Kubernetes deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lakefs
spec:
  replicas: 2
  selector:
    matchLabels:
      app: lakefs
  template:
    metadata:
      labels:
        app: lakefs
    spec:
      containers:
        - name: lakefs
          image: treeverse/lakefs:latest
          args: ["run"]
          ports:
            - containerPort: 8000
          env:
            - name: LAKEFS_DATABASE_TYPE
              value: postgres
            - name: LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING
              valueFrom:
                secretKeyRef:
                  name: lakefs-secrets
                  key: db-connection-string
            - name: LAKEFS_BLOCKSTORE_TYPE
              value: s3
          livenessProbe:
            httpGet:
              path: /api/v1/healthcheck
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /api/v1/healthcheck
              port: 8000
            initialDelaySeconds: 15
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: lakefs
spec:
  selector:
    app: lakefs
  ports:
    - port: 80
      targetPort: 8000
  type: LoadBalancer

For Kubernetes LakeFS, monitor the LoadBalancer IP/hostname externally — the internal readiness probe only tells the scheduler whether to route traffic; Vigilmon tells you whether that traffic is actually succeeding for external clients.

Step 9: Create a status page for your data platform

  1. Go to Status Pages → New Status Page
  2. Name it: "Data Platform Status"
  3. Add monitors grouped by component:
    • LakeFS API: healthcheck + repositories endpoint
    • Database: PostgreSQL TCP monitor
    • Storage: (if you expose a storage health endpoint)
  4. Publish the status page

Share with:

  • Data engineering team members who run pipelines against LakeFS
  • Data scientists whose experiments depend on LakeFS branches
  • Platform engineering leadership tracking data infrastructure SLAs

LakeFS monitoring architecture

[Data Pipelines (Spark, dbt, Airflow)]
         ↓
   [Nginx / Load Balancer]
         ↓
     [LakeFS API]
      ↙        ↘
[PostgreSQL]  [S3/GCS/Azure Blob]

Vigilmon monitors:
  ✓ LakeFS API (external HTTP to load balancer)
  ✓ PostgreSQL (TCP to DB port)
  ✓ LakeFS UI (HTTP with keyword check)
  ✓ Repositories endpoint (API + storage chain)

Putting it all together

| Monitor | Type | What it catches | |---------|------|----------------| | https://lakefs.yourdomain.com/api/v1/healthcheck | HTTP (expect 204) | Process crashes, proxy failures, DNS issues | | https://lakefs.yourdomain.com/ | HTTP + keyword | UI serving failures | | https://lakefs.yourdomain.com/api/v1/repositories | HTTP + keyword | Database + object store connectivity | | postgres.yourdomain.com:5432 | TCP | Database-level failures |

What's next

  • Heartbeat monitoring — if you run nightly data pipeline jobs that commit to LakeFS branches, use Vigilmon heartbeats to alert when jobs stop completing
  • SSL certificate monitoring — Vigilmon tracks your LakeFS endpoint's TLS certificate and alerts before expiry
  • Multi-environment — separate monitors for development, staging, and production LakeFS instances, grouped on the status page

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles