tutorial

How to Monitor Apache Atlas (Data Governance) with Vigilmon

Apache Atlas is the metadata backbone of your data governance platform — but REST API failures, HBase backend degradation, and lineage indexing stalls are invisible without external monitoring. Learn how to monitor Atlas health, API availability, and ingestion pipelines with Vigilmon.

Apache Atlas is the metadata and data governance layer for your Hadoop ecosystem — tracking lineage, classification, and data ownership across thousands of datasets. When Atlas goes down, your data stewards lose visibility, compliance audits stall, and downstream tools like Ranger and Hive that depend on Atlas classification policies begin operating on stale data. Yet Atlas is often monitored with nothing more than a process check on the JVM, if that.

Vigilmon gives you external visibility into Apache Atlas availability through HTTP probe monitoring of its REST API and Admin endpoints, plus heartbeat monitoring for your metadata ingestion pipelines. This tutorial covers both.

Why Apache Atlas Needs External Monitoring

Apache Atlas runs as a multi-component system where each layer can fail independently:

  • Atlas REST Server — the main HTTP API used by data stewards, Ranger, and ingestion clients. If it's down, no metadata can be read or written.
  • HBase backend — Atlas stores its entity graph in HBase. If HBase degrades, Atlas REST requests start timing out or returning errors.
  • Solr indexing — Atlas uses Solr (or embedded Elasticsearch) for full-text metadata search. A Solr outage means search queries return empty results without obvious error messages.
  • Kafka ingestion hooks — Atlas receives metadata events from Hive, Spark, and HDFS via Kafka. If the Kafka bridge process stops, metadata for new datasets goes unrecorded silently.

Standard infrastructure monitoring (JVM heap checks, process counts) tells you whether Atlas is running, not whether it's correctly serving metadata. External monitoring with Vigilmon adds:

  • API availability probes via the Atlas REST Admin endpoint
  • Search functionality checks via the Atlas type search API
  • Heartbeat monitoring for Kafka bridge and metadata ingestion processes
  • Multi-region probe consensus to avoid false alerts from transient network partitions between Vigilmon's probes and your Atlas cluster

Step 1: Identify Atlas Health Endpoints

Apache Atlas ships with built-in REST endpoints you can probe directly.

Admin Status Endpoint

The Atlas Admin API exposes a server status endpoint that confirms Atlas is alive and connected to its backend:

# Check Atlas admin status — returns JSON with server state
curl -u admin:admin -i http://atlas-host:21000/api/atlas/admin/status

# Expected healthy response:
# {"Status":"ACTIVE"}

# During startup or backend issues:
# {"Status":"STARTING"} or {"Status":"PASSIVE"}

Monitor this endpoint: a non-200 response or a status other than "ACTIVE" indicates Atlas is unavailable or degraded.

Version Endpoint (Unauthenticated)

For a quick liveness check without credentials:

curl -i http://atlas-host:21000/api/atlas/admin/version
# Returns Atlas version info — 200 means the HTTP server is up

Search API Health Check

Verify that Solr search is working by running a lightweight type search:

curl -u admin:admin \
  "http://atlas-host:21000/api/atlas/v2/search/basic?typeName=hive_table&limit=1" \
  -H "Accept: application/json"

A 200 response with a JSON search result confirms Atlas REST and Solr are both functional. A timeout or error indicates Solr indexing issues.

Custom Health Wrapper

For cleaner Vigilmon integration, wrap these checks behind a single health endpoint in a sidecar:

# health/atlas.py
from fastapi import FastAPI
from fastapi.responses import JSONResponse
import httpx
import os

app = FastAPI()

ATLAS_BASE = os.environ.get("ATLAS_URL", "http://atlas-host:21000")
ATLAS_USER = os.environ.get("ATLAS_USER", "admin")
ATLAS_PASS = os.environ.get("ATLAS_PASSWORD", "admin")

@app.get("/health/atlas")
async def atlas_health():
    auth = (ATLAS_USER, ATLAS_PASS)
    async with httpx.AsyncClient(timeout=8.0) as client:
        try:
            status_resp = await client.get(
                f"{ATLAS_BASE}/api/atlas/admin/status", auth=auth
            )
            if status_resp.status_code != 200:
                return JSONResponse(
                    {"status": "down", "reason": "admin_api_unavailable", "code": status_resp.status_code},
                    503
                )

            body = status_resp.json()
            if body.get("Status") != "ACTIVE":
                return JSONResponse(
                    {"status": "degraded", "reason": "atlas_not_active", "atlas_status": body.get("Status")},
                    503
                )

            # Verify search is working
            search_resp = await client.get(
                f"{ATLAS_BASE}/api/atlas/v2/search/basic?typeName=hive_table&limit=1",
                auth=auth,
                headers={"Accept": "application/json"},
            )
            if search_resp.status_code != 200:
                return JSONResponse(
                    {"status": "degraded", "reason": "search_unavailable", "code": search_resp.status_code},
                    503
                )

            return {"status": "ok", "atlas_status": "ACTIVE", "search": "ok"}

        except httpx.TimeoutException:
            return JSONResponse({"status": "down", "reason": "timeout"}, 503)
        except Exception as e:
            return JSONResponse({"status": "down", "error": str(e)}, 503)

# Deploy the health sidecar
pip install fastapi httpx uvicorn
ATLAS_URL=http://atlas-host:21000 ATLAS_USER=admin ATLAS_PASSWORD=secret \
  uvicorn health.atlas:app --host 0.0.0.0 --port 8090

Step 2: Configure Vigilmon HTTP Monitors for Apache Atlas

Primary Atlas Status Monitor

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Atlas health endpoint:
    • Sidecar: https://your-server.example.com/health/atlas
    • Direct (if publicly accessible): http://atlas-host:21000/api/atlas/admin/status
  4. Set the check interval to 2 minutes
  5. Under Expected response:
    • Status code: 200
    • Response body contains: "status":"ok" (sidecar) or "ACTIVE" (direct)
    • Response time threshold: 8000ms (Atlas admin API can be slow under HBase pressure)
  6. Under Alert channels, assign your data platform Slack channel + PagerDuty
  7. Save the monitor

Atlas Search Availability Monitor

Add a second monitor for Solr search functionality:

  • URL: https://your-server.example.com/health/atlas (or a dedicated /health/atlas/search)
  • Expected: 200, body contains "search":"ok"
  • Interval: 5 minutes
  • Alert channel: Slack (P2 — search degradation before full API outage)
  • Response time threshold: 10000ms (search can be slower than admin API)

Direct Admin Endpoint Monitor (Backup)

For belt-and-suspenders coverage if the sidecar is not deployed:

  • URL: http://atlas-host:21000/api/atlas/admin/version
  • Expected: 200
  • Interval: 1 minute
  • Alert: PagerDuty (P1 for process-down)
  • Authentication: Use Vigilmon's HTTP Basic Auth header option: Authorization: Basic base64(admin:password)

Step 3: Heartbeat Monitoring for Atlas Ingestion Pipelines

The Atlas REST API being healthy doesn't mean metadata is being ingested. Atlas hooks for Hive, Spark, and HDFS run as separate bridge processes that send metadata events to Kafka for Atlas to consume. If these bridges stop running, new tables, columns, and lineage events go unrecorded — silently.

Vigilmon heartbeat monitors detect bridge stalls: each bridge sends a ping to Vigilmon after processing a batch of events. If pings stop, Vigilmon alerts before metadata drift becomes a compliance problem.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name it: atlas-hive-bridge
  3. Set expected interval: 15 minutes
  4. Set grace period: 30 minutes
  5. Save — copy the heartbeat URL

Wire Into Your Hive Atlas Bridge Script

# bridge/hive_metadata_sync.py
"""
Syncs Hive table metadata to Apache Atlas via REST API.
Run this on a schedule (cron / Airflow) after Hive DDL operations.
"""
import subprocess
import requests
import os
import sys

ATLAS_URL = os.environ["ATLAS_URL"]
ATLAS_USER = os.environ["ATLAS_USER"]
ATLAS_PASSWORD = os.environ["ATLAS_PASSWORD"]
VIGILMON_HEARTBEAT = os.environ["VIGILMON_HEARTBEAT_URL"]

def sync_hive_metadata():
    # Invoke Atlas Hive hook import
    result = subprocess.run(
        ["hive", "--service", "hiveimport", "--database", "default"],
        capture_output=True, text=True, timeout=600
    )

    if result.returncode != 0:
        print(f"Hive import failed:\n{result.stderr}", file=sys.stderr)
        return False

    # Verify a sample entity was imported
    resp = requests.get(
        f"{ATLAS_URL}/api/atlas/v2/search/basic?typeName=hive_table&limit=1",
        auth=(ATLAS_USER, ATLAS_PASSWORD),
        timeout=10,
    )
    if resp.status_code != 200:
        print(f"Atlas search check failed: {resp.status_code}", file=sys.stderr)
        return False

    # Ping Vigilmon to confirm successful sync
    requests.get(VIGILMON_HEARTBEAT, timeout=5)
    print("Hive metadata sync complete, heartbeat sent")
    return True

if __name__ == "__main__":
    success = sync_hive_metadata()
    sys.exit(0 if success else 1)

Wire Into an Airflow DAG

# dags/atlas_metadata_sync.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
import requests
import os

VIGILMON_HEARTBEAT = os.environ.get("VIGILMON_HEARTBEAT_URL_ATLAS_SYNC")

def sync_atlas(**context):
    import subprocess
    result = subprocess.run(
        ["python", "/opt/bridge/hive_metadata_sync.py"],
        capture_output=True, text=True
    )
    if result.returncode != 0:
        raise RuntimeError(f"Atlas sync failed: {result.stderr}")

    # Ping Vigilmon on successful DAG run
    if VIGILMON_HEARTBEAT:
        requests.get(VIGILMON_HEARTBEAT, timeout=5)

with DAG(
    "atlas_metadata_sync",
    default_args={"owner": "data-platform", "retries": 1, "retry_delay": timedelta(minutes=5)},
    schedule_interval="@hourly",
    start_date=datetime(2026, 1, 1),
    catchup=False,
) as dag:
    sync_task = PythonOperator(
        task_id="sync_hive_to_atlas",
        python_callable=sync_atlas,
    )

Step 4: Alert Routing for Atlas Failures

Atlas failures have layered impact: an HBase degradation causes Atlas API slowness before full outage, and a Solr failure causes silent search gaps before users notice. Configure alert routing to catch failures early:

| Monitor | Alert Channel | Priority | |---|---|---| | Atlas /health/atlas (full health) | Slack + PagerDuty | P1 | | Atlas admin /api/atlas/admin/version (liveness) | PagerDuty | P1 | | Atlas search endpoint | Slack | P2 | | Heartbeat: Hive bridge | Slack + email | P2 | | Heartbeat: Spark lineage bridge | Email | P3 |

Response time thresholds for early warning:

  • Alert at 5000ms for admin API (slow admin response signals HBase pressure before full outage)
  • Alert at 15000ms for search endpoint (Solr latency climbing)
  • Alert at 12000ms for sidecar health check (end-to-end check including search)

For governance-critical deployments, configure a status page in Vigilmon so data stewards and compliance teams can see Atlas status without needing to contact the data platform team.

Summary

Apache Atlas is the metadata backbone of your data governance platform — when it's down, compliance visibility, lineage tracking, and policy enforcement all degrade. External monitoring with Vigilmon ensures you find out before your data stewards do:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on Atlas admin status | Atlas REST server availability, ACTIVE state | | HTTP monitor on Atlas search API | Solr search functionality | | HTTP monitor on admin version endpoint | Raw process liveness | | Heartbeat monitor | Hive/Spark metadata ingestion bridges |

Get started free at vigilmon.online — your first Atlas monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles