tutorial

Monitoring Atlan Data Catalog Infrastructure with Vigilmon

Atlan is the active data catalog for modern data teams — but when Atlan goes down, metadata syncs fail silently, lineage graphs go stale, and data teams lose the context they depend on to ship safely. Here's how to monitor Atlan availability, connector sync health, metadata freshness, and API performance with Vigilmon.

Atlan is an active data catalog and metadata management platform that connects to your data sources, BI tools, and transformation frameworks to provide unified governance, lineage, and discovery for data teams. It syncs metadata from Snowflake, BigQuery, dbt, Looker, Airflow, and dozens of other tools through managed connectors, building a living catalog of assets, lineage, and ownership. When Atlan's infrastructure degrades — the web application goes down, a connector stops syncing metadata, or the lineage graph becomes stale after a source update — data teams lose the ability to find trusted datasets, trace data lineage for compliance, and understand the downstream impact of schema changes. Vigilmon gives you external monitoring for Atlan's application availability, API performance, connector sync health, and metadata freshness so your data platform team catches catalog outages before they become data governance incidents.

What You'll Set Up

  • Atlan application URL availability monitoring
  • Atlan API health and response time tracking
  • Connector metadata sync freshness checks
  • Lineage completeness monitoring for critical assets
  • Webhook and notification delivery health

Prerequisites

  • Atlan workspace URL (e.g., https://your-org.atlan.com)
  • Atlan API token with assets:read and connections:read scope
  • Access to connector sync logs or Atlan's audit API
  • A free Vigilmon account

Step 1: Monitor Atlan Application Availability

Atlan is a SaaS product, but enterprise deployments with private networking (PrivateLink, VPN tunnels, or SSO-enforced access) can experience availability degradation that the Atlan status page doesn't reflect — a routing issue between your corporate network and Atlan's endpoint, an SSO provider outage that blocks authentication, or a CDN misconfiguration that returns errors only from specific regions. Monitor the URL your users actually access:

  1. In Vigilmon, click Add Monitor → HTTP(S).
  2. Set the URL to your Atlan workspace (e.g., https://your-org.atlan.com).
  3. Set Check interval to 2 minutes.
  4. Set Expected status to 200.
  5. Enable Response time threshold and set it to 5000ms — Atlan's catalog UI should load in under 5 seconds.

For API-level availability, add a second monitor targeting the Atlan metadata API:

  1. Add another HTTP(S) monitor.
  2. URL: https://your-org.atlan.com/api/meta/entity/bulk (or equivalent ping endpoint).
  3. Add header: Authorization: Bearer YOUR_API_TOKEN.
  4. Expected status: 200.

This two-layer check distinguishes between the web application being down (user-facing outage) and the API being degraded (connector and integration failures) while the UI still loads.

Step 2: Monitor Connector Metadata Sync Freshness

Atlan connectors sync metadata from your data sources — Snowflake table schemas, dbt model documentation, Looker dashboard owners, Airflow DAG lineage. These syncs run on a schedule (typically every 24 hours, configurable per connector). When a connector fails silently — an API credential rotated in Snowflake but not updated in Atlan, a dbt Cloud job URL change, or a network timeout during a large metadata crawl — the catalog shows stale asset metadata without any visible error to catalog users. Monitor sync freshness via the Atlan API:

#!/bin/bash
# /usr/local/bin/atlan-sync-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
ATLAN_URL="https://your-org.atlan.com"
ATLAN_TOKEN="your-api-token"
MAX_SYNC_AGE_HOURS=26   # Allow up to 26 hours for daily syncs
TIMEOUT=20

# Query Atlan connections and check their last sync time
CONNECTIONS=$(curl -s \
  --max-time "$TIMEOUT" \
  -H "Authorization: Bearer $ATLAN_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  "${ATLAN_URL}/api/meta/search/indexsearch" \
  -d '{
    "dsl": {
      "query": {"match_all": {}},
      "post_filter": {"term": {"__typeName.keyword": "Connection"}},
      "size": 50
    },
    "attributes": ["name", "lastSyncRunAt", "connectorName"]
  }')

STALE_CONNECTORS=$(echo "$CONNECTIONS" | python3 -c "
import sys, json
from datetime import datetime, timezone, timedelta

data = json.load(sys.stdin)
entities = data.get('entities', [])
now = datetime.now(timezone.utc)
max_age = timedelta(hours=$MAX_SYNC_AGE_HOURS)
stale = []

for entity in entities:
    attrs = entity.get('attributes', {})
    name = attrs.get('name', 'unknown')
    connector = attrs.get('connectorName', 'unknown')
    last_sync = attrs.get('lastSyncRunAt')

    if last_sync:
        try:
            # lastSyncRunAt is epoch milliseconds
            last_sync_dt = datetime.fromtimestamp(int(last_sync) / 1000, tz=timezone.utc)
            age = now - last_sync_dt
            if age > max_age:
                stale.append(f'{name} ({connector}): {int(age.total_seconds() / 3600)}h ago')
        except (ValueError, TypeError):
            stale.append(f'{name} ({connector}): invalid timestamp')
    else:
        stale.append(f'{name} ({connector}): never synced')

if stale:
    print('\n'.join(stale))
else:
    print('')
" 2>/dev/null)

if [ -z "$STALE_CONNECTORS" ]; then
  echo "All connectors synced within ${MAX_SYNC_AGE_HOURS} hours"
  curl -s "$HEARTBEAT_URL"
else
  echo "Stale connectors detected:"
  echo "$STALE_CONNECTORS"
  exit 1
fi

Set the Vigilmon heartbeat to 70 minutes (just over your sync schedule). A connector that hasn't synced in 26+ hours on a daily schedule indicates the last sync silently failed.

Step 3: Track Atlan API Response Time

Atlan's performance matters as much as its availability. When the metadata API responds slowly — due to a large graph traversal, an overloaded search index, or a heavy catalog crawl running in the background — data engineers querying asset lineage, running impact analysis, and checking data trust scores all experience degraded productivity. Monitor API response time as a first-class signal:

#!/bin/bash
# /usr/local/bin/atlan-latency-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
ATLAN_URL="https://your-org.atlan.com"
ATLAN_TOKEN="your-api-token"
MAX_LATENCY_MS=3000   # Alert if API response exceeds 3 seconds
TIMEOUT=15

# Measure time for a lightweight metadata search query
START_MS=$(date +%s%3N)

HTTP_CODE=$(curl -s -o /dev/null \
  --max-time "$TIMEOUT" \
  -w "%{http_code}" \
  -H "Authorization: Bearer $ATLAN_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  "${ATLAN_URL}/api/meta/search/indexsearch" \
  -d '{
    "dsl": {
      "query": {"match_all": {}},
      "post_filter": {"term": {"__typeName.keyword": "Database"}},
      "size": 1
    }
  }')

END_MS=$(date +%s%3N)
LATENCY_MS=$(( END_MS - START_MS ))

if [ "$HTTP_CODE" != "200" ]; then
  echo "Atlan API error: HTTP $HTTP_CODE"
  exit 1
fi

if [ "$LATENCY_MS" -le "$MAX_LATENCY_MS" ]; then
  echo "API latency OK: ${LATENCY_MS}ms (max: ${MAX_LATENCY_MS}ms)"
  curl -s "$HEARTBEAT_URL"
else
  echo "API latency high: ${LATENCY_MS}ms (max: ${MAX_LATENCY_MS}ms)"
  exit 1
fi

Run this check every 5 minutes with a Vigilmon heartbeat at the same interval. Latency spikes that correlate with connector sync schedules (large crawls) versus random times indicate background processing contention rather than a systemic API performance problem.

Step 4: Monitor Critical Asset Lineage Completeness

Atlan builds data lineage graphs by parsing SQL from query logs, ingesting lineage from dbt and Airflow, and aggregating cross-tool relationships. When a lineage source is unavailable — the query log connector pauses, a dbt run stops sending lineage events, or a Spark job's OpenLineage integration breaks — the lineage graph becomes incomplete without any alert. Monitor that your most critical production assets have active upstream lineage:

#!/bin/bash
# /usr/local/bin/atlan-lineage-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
ATLAN_URL="https://your-org.atlan.com"
ATLAN_TOKEN="your-api-token"
MAX_LINEAGE_AGE_HOURS=48  # Alert if lineage hasn't been refreshed in 48 hours
TIMEOUT=20

# List of critical Atlan asset GUIDs to verify lineage for
# Get these from Atlan UI: Asset → Details → Copy GUID
CRITICAL_ASSETS=(
  "a1b2c3d4-e5f6-7890-abcd-ef1234567890"   # production.analytics.revenue_daily
  "b2c3d4e5-f6a7-8901-bcde-f12345678901"   # production.ml.model_features
)

FAILED=0

for ASSET_GUID in "${CRITICAL_ASSETS[@]}"; do
  # Query lineage graph for this asset
  LINEAGE_RESP=$(curl -s \
    --max-time "$TIMEOUT" \
    -H "Authorization: Bearer $ATLAN_TOKEN" \
    "${ATLAN_URL}/api/meta/lineage/uniqueAttribute/typeName/Table?attr:qualifiedName=${ASSET_GUID}&depth=1&direction=INPUT")

  HAS_UPSTREAM=$(echo "$LINEAGE_RESP" | python3 -c "
import sys, json
data = json.load(sys.stdin)
relations = data.get('relations', [])
print('yes' if relations else 'no')
" 2>/dev/null || echo "error")

  if [ "$HAS_UPSTREAM" = "yes" ]; then
    echo "Lineage OK for asset: $ASSET_GUID"
  elif [ "$HAS_UPSTREAM" = "no" ]; then
    echo "MISSING upstream lineage for critical asset: $ASSET_GUID"
    FAILED=1
  else
    echo "Failed to check lineage for asset: $ASSET_GUID"
    FAILED=1
  fi
done

if [ "$FAILED" -eq 0 ]; then
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Set the heartbeat to 60 minutes. Missing lineage on critical production tables is a data governance incident — compliance teams, impact analysis workflows, and data trust scoring all depend on complete lineage graphs. Running this check twice a day catches lineage gaps before a compliance audit surfaces them.

Step 5: Monitor Atlan Webhook and Notification Delivery

Atlan can push event notifications — new request approvals, policy violations, data quality alerts, ownership change requests — to Slack, email, and custom webhooks. When Atlan's notification delivery breaks (an expired Slack webhook, a firewall rule blocking outbound HTTPS from Atlan, or a misconfigured integration), governance workflows stall silently: data owners never see that their approval is needed, quality alerts never reach the on-call team, and policy violations accumulate unresolved. Monitor notification delivery with a test webhook:

#!/bin/bash
# /usr/local/bin/atlan-webhook-check.sh
# Run this as a cron job that verifies Atlan can reach your webhook receiver

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
ATLAN_URL="https://your-org.atlan.com"
ATLAN_TOKEN="your-api-token"
WEBHOOK_RECEIVER_LOG="/var/log/atlan-webhook-test.log"
MAX_DELIVERY_AGE_MINUTES=65  # Alert if webhook hasn't been received in 65 minutes
TIMEOUT=15

# Trigger a test event via Atlan API (or check a regularly-occurring real event)
# Here we check when our monitoring webhook receiver last received an Atlan event

LAST_DELIVERY=$(cat "$WEBHOOK_RECEIVER_LOG" 2>/dev/null | tail -1 | \
  python3 -c "import sys, json; d = json.load(sys.stdin); print(d.get('timestamp', ''))" 2>/dev/null)

if [ -z "$LAST_DELIVERY" ]; then
  echo "No webhook deliveries recorded yet — check receiver config"
  # Don't fail on first run
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

AGE_MINUTES=$(python3 -c "
from datetime import datetime, timezone
last = datetime.fromisoformat('$LAST_DELIVERY').replace(tzinfo=timezone.utc)
age = (datetime.now(timezone.utc) - last).total_seconds() / 60
print(int(age))
" 2>/dev/null || echo 9999)

if [ "$AGE_MINUTES" -le "$MAX_DELIVERY_AGE_MINUTES" ]; then
  echo "Webhook delivery OK: last received ${AGE_MINUTES} min ago"
  curl -s "$HEARTBEAT_URL"
else
  echo "Webhook delivery stale: last received ${AGE_MINUTES} min ago (max: ${MAX_DELIVERY_AGE_MINUTES} min)"
  exit 1
fi

For the webhook receiver, deploy a small HTTP server that logs incoming Atlan events:

# webhook_receiver.py (deploy alongside your infrastructure)
from fastapi import FastAPI, Request
import json
from datetime import datetime, timezone

app = FastAPI()
LOG_FILE = "/var/log/atlan-webhook-test.log"

@app.post("/atlan-events")
async def receive_event(request: Request):
    body = await request.json()
    entry = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "event_type": body.get("type"),
        "entity": body.get("entity", {}).get("displayText"),
    }
    with open(LOG_FILE, "a") as f:
        f.write(json.dumps(entry) + "\n")
    return {"status": "received"}

Configure this endpoint as an Atlan webhook in Settings → Integrations → Webhooks with a broad event subscription.

Step 6: Set Up Alert Channels

Configure Vigilmon to notify the data platform team:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #data-platform-alerts — catalog outages affect all data teams immediately.
  3. Add PagerDuty for the application availability and API latency monitors — these are user-facing outages requiring immediate response.
  4. Add email notifications for sync freshness, lineage completeness, and webhook monitors — these degrade over hours and need investigation, not a midnight page.
  5. Set Consecutive failures before alert to 2 for the API latency monitor — Atlan's search index occasionally takes longer during connector syncs.

Add maintenance windows during connector reconfiguration or Atlan workspace migrations:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "ATLAN_SYNC_MONITOR_ID",
    "duration_minutes": 120,
    "reason": "Snowflake connector reconfiguration after credential rotation"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (URL) | https://your-org.atlan.com | Application down, SSO outage, CDN failure | | HTTP monitor (API) | Atlan search API /indexsearch | API degraded while UI loads, auth token expiry | | Cron heartbeat (sync freshness) | Connector lastSyncRunAt | Failed crawl, rotated credentials, network timeout | | Cron heartbeat (API latency) | Search API response time | Overloaded index, background crawl contention | | Cron heartbeat (lineage) | Critical asset upstream relations | Broken lineage source, OpenLineage integration failure | | Cron heartbeat (webhooks) | Webhook receiver last delivery | Slack token expiry, firewall blocks, integration breaks |

Atlan is the metadata layer your entire data platform depends on — broken sync means stale governance, missing lineage means risky schema changes, and a down catalog means slower data work across every team. Vigilmon's external monitoring gives your data platform team early warning before catalog failures become data governance incidents or compliance gaps.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles