tutorial

How to Monitor DataHub with Vigilmon (HTTP, TCP, and Alerts)

DataHub is the open-source metadata management platform built by LinkedIn, now widely adopted at companies like Netflix, Airbnb, and Expedia. It ingests, cat...

DataHub is the open-source metadata management platform built by LinkedIn, now widely adopted at companies like Netflix, Airbnb, and Expedia. It ingests, catalogs, and exposes lineage for every dataset across your data stack. When DataHub goes down, data engineers lose visibility into dataset ownership, lineage, and quality — and pipelines that depend on its APIs start throwing errors silently.

In this tutorial you'll set up end-to-end monitoring for a self-hosted DataHub deployment using Vigilmon — free tier, no credit card required.

Why DataHub needs external monitoring

DataHub is a multi-service platform. A typical deployment includes:

  • GMS (Generalized Metadata Service) — the core API backend on port 8080
  • Frontend — the React UI on port 9002
  • Kafka + ZooKeeper — event streaming layer for metadata ingestion
  • Elasticsearch — stores entity search indexes
  • MySQL or PostgreSQL — persistence for metadata aspects

Any single component going down can silently degrade the platform while Docker reports every container as "running." Specific failure modes to watch for:

  • GMS crashes under load — ingestion pipelines queue up and begin failing after the queue fills
  • Elasticsearch shard failure — search returns zero results but no error is shown to the user
  • Kafka lag — metadata events pile up; lineage graphs show stale data for hours
  • Frontend timeouts — GMS is healthy but the frontend proxy can't reach it due to a Docker network misconfiguration

An external monitor that probes each service endpoint from outside the Docker network catches failures that internal health checks cannot.

What you'll need

  • A running DataHub deployment (Docker Compose or Kubernetes)
  • A free Vigilmon account — takes 30 seconds to create

Step 1: Identify your DataHub health endpoints

DataHub ships with built-in health endpoints. Make sure these are accessible from outside your deployment:

GMS health check:

GET http://<your-host>:8080/health

Returns {"status":"UP"} when healthy.

GMS actuator endpoint (detailed):

GET http://<your-host>:8080/actuator/health

Returns individual component statuses including Kafka, Elasticsearch, and the database.

Frontend health:

GET http://<your-host>:9002/health

To verify these endpoints are reachable from your Vigilmon monitoring region, test them from a machine outside your private network:

curl -f https://datahub.yourcompany.com/health
# Expected: {"status":"UP"}

curl -f https://datahub.yourcompany.com/actuator/health
# Expected: {"status":"UP","components":{...}}

If you're behind a reverse proxy (nginx, Caddy, Traefik), make sure the /health and /actuator/health paths are proxied through to GMS port 8080.

Step 2: Expose DataHub ports via reverse proxy

A production DataHub deployment should not expose GMS port 8080 directly to the internet. Use nginx to proxy requests and enable TLS:

server {
    listen 443 ssl;
    server_name datahub.yourcompany.com;

    ssl_certificate     /etc/letsencrypt/live/datahub.yourcompany.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/datahub.yourcompany.com/privkey.pem;

    # Frontend UI
    location / {
        proxy_pass http://localhost:9002;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # GMS API (used by ingestion pipelines and the frontend)
    location /api/ {
        proxy_pass http://localhost:8080/;
        proxy_set_header Host $host;
    }

    # Health endpoint — no auth required, safe to expose
    location /health {
        proxy_pass http://localhost:8080/health;
    }

    location /actuator/health {
        proxy_pass http://localhost:8080/actuator/health;
    }
}

After applying this config, reload nginx and confirm the health endpoints return 200 from an external network.

Step 3: Set up HTTP monitoring in Vigilmon

With your health endpoints accessible, create Vigilmon monitors:

GMS health monitor:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to https://datahub.yourcompany.com/health
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"UP"
  6. Save the monitor

Frontend health monitor:

Repeat the same steps with URL https://datahub.yourcompany.com and expected status code 200. This catches the case where GMS is healthy but the frontend can't reach it.

Actuator deep health monitor:

Create an additional HTTP monitor pointing to https://datahub.yourcompany.com/actuator/health with body assertion "status":"UP". The actuator endpoint also returns statuses for each downstream component — if Elasticsearch or Kafka is degraded, this endpoint will return a non-200 or include "status":"DOWN" in the body, and Vigilmon will open an incident.

Step 4: Monitor Kafka and Elasticsearch via TCP

DataHub's event streaming layer is Kafka. If ZooKeeper or Kafka brokers lose their port binding, ingestion will silently fail:

  1. In Vigilmon, go to Monitors → New Monitor
  2. Choose TCP Port as the type
  3. Enter your Kafka host and port 9092
  4. Save the monitor

Add another TCP monitor for Elasticsearch on port 9200 if it's exposed (internal-only is fine if you have a proxy in front).

| Monitor | Type | Port | What it catches | |---------|------|------|-----------------| | /health | HTTP | 443 | GMS down, database failures | | /actuator/health | HTTP | 443 | Elasticsearch, Kafka degraded | | Frontend | HTTP | 443 | UI unreachable | | Kafka | TCP | 9092 | Ingestion pipeline failures | | Elasticsearch | TCP | 9200 | Search index unavailable |

Step 5: Monitor DataHub on Kubernetes

If you deploy DataHub via Helm, the GMS pod exposes its health endpoint on port 8080. Create an Ingress rule to allow Vigilmon to probe it from outside the cluster:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: datahub-health-ingress
  namespace: datahub
spec:
  rules:
    - host: datahub-health.yourcompany.com
      http:
        paths:
          - path: /health
            pathType: Exact
            backend:
              service:
                name: datahub-gms
                port:
                  number: 8080

Point a Vigilmon HTTP monitor at https://datahub-health.yourcompany.com/health with body assertion "status":"UP". Add a second monitor for the frontend pod via its Ingress at https://datahub.yourcompany.com. This gives you external visibility even if the Kubernetes internal health probes are passing — a misconfigured NetworkPolicy can cause GMS to pass its livenessProbe while being unreachable from outside the cluster.

Step 6: Set up DataHub ingestion pipeline alerting

DataHub is often the central piece of a nightly ingestion workflow — dbt runs, Airflow DAGs, and Spark jobs all push metadata through the DataHub ingestion client. If GMS is down when a pipeline tries to push, metadata goes stale and the failure is usually silent.

Add a Vigilmon heartbeat monitor so your ingestion pipeline checks in after every successful run:

# In your DataHub ingestion script (Python)
import requests
import subprocess

def run_ingestion():
    result = subprocess.run(
        ["datahub", "ingest", "-c", "my_pipeline.yml"],
        capture_output=True
    )
    if result.returncode == 0:
        # Ping Vigilmon heartbeat — only on success
        requests.get("https://heartbeat.vigilmon.online/YOUR_HEARTBEAT_ID", timeout=5)
    else:
        print("Ingestion failed:", result.stderr.decode())

run_ingestion()

In Vigilmon, create a Heartbeat monitor with an expected interval of 24 hours (or whatever your ingestion cadence is). If the heartbeat isn't received within that window, Vigilmon opens an incident and alerts your team.

Step 7: Configure alerts for on-call rotation

When DataHub goes down, data engineers need to know immediately:

  1. Go to Alert Channels → Add Channel → Webhook in Vigilmon
  2. Enter your Slack or PagerDuty webhook URL
  3. Assign the channel to all DataHub monitors

Example incident payload Vigilmon sends when GMS goes down:

{
  "monitor_name": "DataHub GMS /health",
  "status": "down",
  "url": "https://datahub.yourcompany.com/health",
  "started_at": "2026-01-15T03:14:00Z",
  "duration_seconds": 120
}

For a data platform, 3 AM incidents are common — Kafka lag usually surfaces overnight after a heavy ingestion batch. Make sure your alert channel reaches the right on-call engineer, not just a Slack channel that nobody checks at night.

Step 8: Create a status page for your data platform

If your data team is spread across time zones, a status page lets engineers self-diagnose before filing a ticket:

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name it "Data Platform Status" or similar
  3. Add your DataHub monitors: GMS, Frontend, Kafka TCP, Elasticsearch TCP
  4. Publish the page and share the URL in your data team's Slack channel and internal wiki

The status page shows real-time uptime percentages and full incident history — useful for post-mortems, audit logs, and monthly SLA reporting.

Putting it all together

Here's a summary of the complete monitoring setup for a DataHub deployment:

DataHub GMS /health       → HTTP monitor, 1-min interval, body: "status":"UP"
DataHub actuator/health   → HTTP monitor, 1-min interval, body: "status":"UP"
DataHub Frontend          → HTTP monitor, 1-min interval, status: 200
Kafka broker              → TCP monitor on port 9092
Elasticsearch             → TCP monitor on port 9200
Nightly ingestion         → Heartbeat monitor, 24h interval

Alert channel: Slack webhook (or PagerDuty for on-call escalation) Status page: shared with the data team for self-service status checks

What's next

  • SSL certificate monitoring — Vigilmon monitors your TLS certificate expiry and alerts you before it lapses, preventing the GMS API from becoming unreachable for ingestion clients
  • Response time tracking — if DataHub API latency increases as your metadata catalog grows, Vigilmon response time charts help you catch degradation before it becomes an outage
  • Multi-region checks — if your data teams are in different geographies, Vigilmon can check DataHub availability from multiple regions simultaneously

Get started free at vigilmon.online — no credit card required, and your first monitors are running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles