tutorial

How to Monitor OpenMetadata with Vigilmon (HTTP, TCP, and Alerts)

OpenMetadata is the open-source metadata platform that unifies data discovery, data quality, observability, and lineage in a single UI. Teams at organization...

OpenMetadata is the open-source metadata platform that unifies data discovery, data quality, observability, and lineage in a single UI. Teams at organizations like Collibra, Astronomer, and various data-driven startups use it to govern their data ecosystems. When OpenMetadata is unavailable, data stewards can't access lineage graphs, data quality pipelines stop posting results, and discovery search goes dark.

In this tutorial you'll set up comprehensive monitoring for a self-hosted OpenMetadata deployment using Vigilmon — free tier, no credit card required.

Why OpenMetadata needs external monitoring

OpenMetadata is a multi-service application with several components that can fail independently:

  • OpenMetadata Server — the main API and UI backend, typically on port 8585
  • Elasticsearch / OpenSearch — powers the search index; loss means discovery returns no results
  • MySQL or PostgreSQL — persistence layer; loss means all API calls fail with 500 errors
  • Airflow — the bundled pipeline scheduler that runs ingestion workflows
  • Ingestion pipelines — metadata crawlers that keep your catalog current

Failure scenarios that external monitoring catches and internal health checks often miss:

  • JVM out-of-memory crash — the OpenMetadata server JVM OOMs, the process exits, Docker marks it as stopped but doesn't alert anyone
  • Airflow scheduler stall — the Airflow web UI is reachable but the scheduler is dead; pipelines queue forever and metadata goes stale
  • Elasticsearch yellow status — unassigned shards cause intermittent search failures that manifest as empty discovery results rather than visible errors
  • Database connection pool exhaustion — API calls start timing out; the server logs show errors but the container health check (which hits a lightweight endpoint) still returns 200

What you'll need

  • A running OpenMetadata deployment (Docker Compose or Kubernetes via Helm)
  • A free Vigilmon account — takes 30 seconds to create

Step 1: Identify OpenMetadata health endpoints

OpenMetadata exposes health endpoints on its main application port (default 8585):

Application health:

GET http://<your-host>:8585/healthcheck

Returns HTTP 200 with {"status":"healthy"} when the server and its database connection are operational.

Liveness probe:

GET http://<your-host>:8585/api/v1/system/config/jwks

Returns 200 if the API is serving. Used by Kubernetes liveness probes.

Test these from an external network to confirm they're accessible:

curl -f https://openmetadata.yourcompany.com/healthcheck
# Expected: {"status":"healthy"}

curl -o /dev/null -s -w "%{http_code}" https://openmetadata.yourcompany.com/api/v1/system/config/jwks
# Expected: 200

Step 2: Set up a reverse proxy with TLS

For production deployments, proxy OpenMetadata through nginx and terminate TLS before exposing it externally. OpenMetadata's default port 8585 should not be internet-facing:

server {
    listen 443 ssl http2;
    server_name openmetadata.yourcompany.com;

    ssl_certificate     /etc/letsencrypt/live/openmetadata.yourcompany.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/openmetadata.yourcompany.com/privkey.pem;

    # Proxy all traffic to the OpenMetadata server
    location / {
        proxy_pass http://localhost:8585;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 120s;
    }

    # Health endpoint for external monitors — no authentication
    location /healthcheck {
        proxy_pass http://localhost:8585/healthcheck;
        access_log off;
    }
}

After reloading nginx, verify the health endpoint is accessible from outside:

curl https://openmetadata.yourcompany.com/healthcheck

Step 3: Create HTTP monitors in Vigilmon

Primary health monitor:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to https://openmetadata.yourcompany.com/healthcheck
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"healthy"
  6. Save the monitor

API availability monitor:

Create a second HTTP monitor pointing to https://openmetadata.yourcompany.com/api/v1/system/config/jwks with expected status 200. This confirms the API is actively serving, not just passing the lightweight healthcheck.

Frontend availability monitor:

Create a third HTTP monitor pointing to https://openmetadata.yourcompany.com (root URL). A 200 here means the React frontend is loading, which also validates that the static asset serving layer is working.

Step 4: Monitor Elasticsearch and MySQL via TCP

OpenMetadata's search and persistence layers are critical. Add TCP monitors for each:

Elasticsearch / OpenSearch:

  1. In Vigilmon, go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your Elasticsearch host and port 9200
  4. Save

MySQL / PostgreSQL:

Add a TCP monitor for your database on port 3306 (MySQL) or 5432 (PostgreSQL).

Complete TCP monitoring table:

| Service | Type | Port | Alert trigger | |---------|------|------|---------------| | Elasticsearch | TCP | 9200 | Search goes dark; discovery broken | | MySQL | TCP | 3306 | All API calls fail with 500 | | Airflow webserver | TCP | 8080 | Pipeline scheduling unavailable |

Step 5: Add heartbeat monitoring for ingestion pipelines

OpenMetadata ingestion pipelines run on Airflow and push metadata to the server on a schedule. When pipelines stop running (Airflow scheduler crash, missing DAG, connection timeout), the metadata catalog goes stale without any visible alert.

Wrap your OpenMetadata ingestion command with a Vigilmon heartbeat ping:

# openmetadata_ingestion_wrapper.py
import subprocess
import requests
import sys

VIGILMON_HEARTBEAT_URL = "https://heartbeat.vigilmon.online/YOUR_HEARTBEAT_ID"

def main():
    result = subprocess.run(
        ["metadata", "ingest", "-c", "my_connector.yaml"],
        capture_output=True,
        text=True
    )

    if result.returncode == 0:
        # Signal success to Vigilmon heartbeat monitor
        try:
            requests.get(VIGILMON_HEARTBEAT_URL, timeout=5)
        except Exception:
            pass  # Don't fail the pipeline on heartbeat errors
        print(result.stdout)
    else:
        print("Ingestion failed:", result.stderr, file=sys.stderr)
        sys.exit(result.returncode)

if __name__ == "__main__":
    main()

In Vigilmon, create a Heartbeat monitor with an expected check-in interval matching your pipeline schedule (e.g., 6 hours for a pipeline that runs every 6 hours). If the pipeline doesn't check in within that window, Vigilmon opens an incident.

Step 6: Monitor OpenMetadata on Kubernetes

OpenMetadata's Helm chart (open-metadata/openmetadata) deploys the server as a Deployment on port 8585. To give Vigilmon external access to the health endpoint, add an Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: openmetadata-health
  namespace: openmetadata
spec:
  rules:
    - host: openmetadata.yourcompany.com
      http:
        paths:
          - path: /healthcheck
            pathType: Exact
            backend:
              service:
                name: openmetadata
                port:
                  number: 8585

Once the Ingress is active, create a Vigilmon HTTP monitor for https://openmetadata.yourcompany.com/healthcheck with keyword assertion "status":"healthy". Check that your Ingress allows unauthenticated access to /healthcheck — some teams protect every route with OAuth, which blocks external monitoring probes. A dedicated allow-list annotation (nginx.ingress.kubernetes.io/whitelist-source-range) for Vigilmon's IP ranges solves this without opening the health route publicly.

Step 7: Configure alert channels

When OpenMetadata goes down, data stewards and data engineers need immediate notification:

  1. In Vigilmon, go to Alert Channels → Add Channel → Webhook
  2. Enter your Slack webhook URL or PagerDuty integration key
  3. Assign the channel to all your OpenMetadata monitors

For a data governance tool like OpenMetadata, consider separate alert routing:

  • P1 (GMS API down) → PagerDuty on-call rotation
  • P2 (Elasticsearch down, search broken) → Slack #data-platform channel
  • P3 (ingestion heartbeat missed) → Slack #data-engineering channel

In Vigilmon you can create multiple alert channels and assign them to specific monitors independently.

Step 8: Set up a status page for your data team

A status page gives your data consumers a place to self-diagnose before raising a ticket:

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name it "Data Platform Status" or "OpenMetadata Status"
  3. Add all your OpenMetadata monitors to the page
  4. Publish the status page and link to it from your team wiki

The page shows real-time uptime, active incidents, and 90-day history — useful for monthly data platform SLA reviews.

Putting it all together

Complete monitoring setup for OpenMetadata:

OpenMetadata /healthcheck     → HTTP monitor, 1-min interval, body: "status":"healthy"
OpenMetadata API /jwks        → HTTP monitor, 1-min interval, status: 200
OpenMetadata Frontend         → HTTP monitor, 1-min interval, status: 200
Elasticsearch                 → TCP monitor on port 9200
MySQL                         → TCP monitor on port 3306
Airflow webserver             → TCP monitor on port 8080
Nightly ingestion pipeline    → Heartbeat monitor, 6h interval

Alert channel: Slack for P2/P3, PagerDuty for P1 Status page: shared with data team and stakeholders

Handling OpenMetadata upgrades safely

OpenMetadata releases frequent updates, and the upgrade process involves database migrations that can take several minutes. During this window, the server is intentionally unavailable.

Use Vigilmon's maintenance windows feature to suppress alerts during planned upgrades:

  1. In Vigilmon, go to Maintenance → New Window
  2. Set the start and end time for your upgrade window
  3. Select the OpenMetadata monitors to suppress
  4. Save the maintenance window

Vigilmon will skip alerting during the window and resume normal monitoring automatically when it ends. Your incident history won't show false positives from planned maintenance.

What's next

  • SSL certificate monitoring — OpenMetadata uses JWTs internally; if your TLS certificate expires, external clients can't authenticate at all. Vigilmon tracks cert expiry and alerts you before it lapses.
  • Response time monitoring — as your metadata catalog grows to millions of assets, API response times can degrade. Vigilmon's response time charts help you catch latency regressions before users complain.
  • Multi-region probing — for globally distributed data teams, Vigilmon can probe your OpenMetadata instance from multiple regions and alert region-specifically.

Get started free at vigilmon.online — no credit card required, and your first monitors are running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles