tutorial

How to Monitor Amundsen (Data Catalog) with Vigilmon

Amundsen is your team's data discovery layer — but a metadata service crash, a Neo4j backend failure, or a stalled ingestion pipeline makes the catalog invisible. Learn how to monitor Amundsen service health and ingestion pipelines with Vigilmon.

Amundsen is Lyft's open-source data discovery and metadata platform — it's where data engineers, analysts, and scientists find the tables, dashboards, and features they need to do their work. When Amundsen goes down, your entire data team loses the ability to find data. When the ingestion pipeline stalls, the catalog silently goes stale and people waste hours finding datasets through other means.

Vigilmon gives you external visibility into Amundsen's health through HTTP probe monitoring of its metadata and search services, plus heartbeat monitoring for your Databuilder ingestion pipelines. This tutorial covers all the layers.

Why Amundsen Needs External Monitoring

Amundsen is a microservices-based data catalog with several independently deployable components:

  • Frontend service — the React-based web UI served by a Flask backend. If it's down, no user can access the catalog.
  • Metadata service — the REST API layer that reads from and writes to the graph database (Neo4j or Apache Atlas). This is the core API used by both the frontend and the Databuilder ingestion pipeline.
  • Search service — the Elasticsearch-backed search API that powers full-text dataset discovery. A search service outage means users can find nothing via the search bar.
  • Neo4j (or other graph backend) — the persistent graph store for entity relationships. Degradation here causes the metadata service to time out or return partial results.
  • Databuilder — the ingestion framework that populates the catalog from your data sources. A stalled Databuilder job means the catalog goes stale.

Standard infrastructure monitoring (container uptime, CPU) doesn't tell you whether search is returning results or whether the metadata API is correctly reading from Neo4j. External monitoring with Vigilmon adds:

  • Service-level probes via Amundsen's built-in /healthcheck endpoints
  • Search functionality verification via the Amundsen search API
  • Heartbeat monitoring for Databuilder ingestion jobs
  • Multi-region probe consensus so transient network issues don't page your on-call team

Step 1: Identify Amundsen Health Endpoints

Amundsen's Python services (Flask-based) expose health check endpoints out of the box.

Metadata Service Health

# Default port 5002 for the metadata service
curl -i http://amundsen-metadata:5002/healthcheck

# Healthy response:
# HTTP/1.1 200 OK
# {"status": "ok"}

# If Neo4j is unreachable:
# HTTP/1.1 503 Service Unavailable
# {"status": "error", "msg": "..."}

Search Service Health

# Default port 5001 for the search service
curl -i http://amundsen-search:5001/healthcheck

Frontend Service Health

# Default port 5000 for the frontend
curl -i http://amundsen-frontend:5000/healthcheck

Search API Smoke Test

Beyond the healthcheck endpoint, verify that search is actually working by running a lightweight search query:

# Search for any table with a wildcard query
curl -i "http://amundsen-search:5001/search?query=*&page_index=0&index=tables"

# A 200 response with a JSON result confirms Elasticsearch is reachable and indexed

Custom Health Aggregator

For a single Vigilmon monitor that covers all Amundsen services:

# health/amundsen.py
from fastapi import FastAPI
from fastapi.responses import JSONResponse
import httpx
import os

app = FastAPI()

METADATA_URL = os.environ.get("AMUNDSEN_METADATA_URL", "http://amundsen-metadata:5002")
SEARCH_URL = os.environ.get("AMUNDSEN_SEARCH_URL", "http://amundsen-search:5001")
FRONTEND_URL = os.environ.get("AMUNDSEN_FRONTEND_URL", "http://amundsen-frontend:5000")

@app.get("/health/amundsen")
async def amundsen_health():
    results = {}
    degraded = []

    async with httpx.AsyncClient(timeout=6.0) as client:
        for name, url in [
            ("metadata", f"{METADATA_URL}/healthcheck"),
            ("search", f"{SEARCH_URL}/healthcheck"),
            ("frontend", f"{FRONTEND_URL}/healthcheck"),
        ]:
            try:
                resp = await client.get(url)
                if resp.status_code == 200:
                    results[name] = "ok"
                else:
                    results[name] = f"error_{resp.status_code}"
                    degraded.append(name)
            except httpx.TimeoutException:
                results[name] = "timeout"
                degraded.append(name)
            except Exception as e:
                results[name] = f"error: {str(e)}"
                degraded.append(name)

        # Verify search actually returns results
        try:
            search_resp = await client.get(
                f"{SEARCH_URL}/search",
                params={"query": "*", "page_index": 0, "index": "tables"},
            )
            results["search_query"] = "ok" if search_resp.status_code == 200 else f"error_{search_resp.status_code}"
            if search_resp.status_code != 200:
                degraded.append("search_query")
        except Exception as e:
            results["search_query"] = f"error: {str(e)}"
            degraded.append("search_query")

    if degraded:
        return JSONResponse({"status": "degraded", "degraded": degraded, "services": results}, 503)

    return {"status": "ok", "services": results}

Deploy on port 8092 alongside your Amundsen services:

AMUNDSEN_METADATA_URL=http://amundsen-metadata:5002 \
AMUNDSEN_SEARCH_URL=http://amundsen-search:5001 \
AMUNDSEN_FRONTEND_URL=http://amundsen-frontend:5000 \
uvicorn health.amundsen:app --host 0.0.0.0 --port 8092

Or as a Docker service in your existing docker-compose.yml:

amundsen-health:
  image: python:3.11-slim
  command: >
    sh -c "pip install fastapi httpx uvicorn &&
           uvicorn health.amundsen:app --host 0.0.0.0 --port 8092"
  environment:
    AMUNDSEN_METADATA_URL: http://amundsen-metadataservice:5002
    AMUNDSEN_SEARCH_URL: http://amundsen-search:5001
    AMUNDSEN_FRONTEND_URL: http://amundsen-frontend:5000
  ports:
    - "8092:8092"
  volumes:
    - ./health:/health

Step 2: Configure Vigilmon HTTP Monitors for Amundsen

Aggregated Health Monitor

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL: https://your-server.example.com/health/amundsen
  4. Set the check interval to 2 minutes
  5. Under Expected response:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms (multiple downstream checks take time)
  6. Under Alert channels, assign your data platform Slack channel + PagerDuty
  7. Save

Individual Service Monitors (Per-Service Granularity)

For faster triage, add one monitor per Amundsen service:

Metadata service:

  • URL: http://amundsen-metadata:5002/healthcheck (or via ingress)
  • Expected: 200
  • Interval: 1 minute
  • Alert: PagerDuty (P1 — metadata down = catalog broken)

Search service:

  • URL: http://amundsen-search:5001/healthcheck
  • Expected: 200
  • Interval: 1 minute
  • Alert: Slack (P2 — search down, metadata still accessible)

Frontend service:

  • URL: http://amundsen-frontend:5000/healthcheck
  • Expected: 200
  • Interval: 1 minute
  • Alert: Slack (P1 — users cannot access UI)

Per-service monitors tell your on-call exactly which component failed, rather than "something in Amundsen is broken."

Step 3: Heartbeat Monitoring for Databuilder Ingestion Jobs

Databuilder is the ingestion framework that populates Amundsen with metadata from your data warehouse, BI tools, and data sources. It typically runs as a scheduled job (Airflow DAG, Kubernetes CronJob, or cron task). If a Databuilder job silently fails — perhaps due to a schema change in the source, an expired credential, or a transient Neo4j write failure — your catalog goes stale without any visible error.

Vigilmon heartbeat monitors catch this pattern: each successful Databuilder run sends a ping to Vigilmon. If pings stop, you know before analysts start filing "why is this table missing from the catalog?" tickets.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name it: amundsen-databuilder-bigquery
  3. Set expected interval: 6 hours (adjust to your Databuilder schedule)
  4. Set grace period: 2 hours
  5. Save — copy the heartbeat URL

Wire Into Your Databuilder Pipeline (Python)

# pipelines/bigquery_metadata.py
"""
Databuilder pipeline to ingest BigQuery table metadata into Amundsen.
Run via Airflow, Kubernetes CronJob, or cron.
"""
from pyhocon import ConfigFactory
from databuilder.extractor.bigquery_metadata_extractor import BigQueryMetadataExtractor
from databuilder.loader.fs_neo4j_csv_loader import FsNeo4jCSVLoader
from databuilder.publisher.neo4j_csv_publisher import Neo4jCsvPublisher
from databuilder.task import DefaultTask
import requests
import os
import logging

logger = logging.getLogger(__name__)

VIGILMON_HEARTBEAT = os.environ["VIGILMON_HEARTBEAT_URL_DATABUILDER_BQ"]

def run_bigquery_pipeline():
    extractor = BigQueryMetadataExtractor()
    loader = FsNeo4jCSVLoader()
    publisher = Neo4jCsvPublisher()

    conf = ConfigFactory.from_dict({
        "extractor.bigquery_metadata_extractor.project_id_list": [
            os.environ["GCP_PROJECT_ID"]
        ],
        "loader.filesystem.neo4j.node_dir_path": "/tmp/amundsen/nodes",
        "loader.filesystem.neo4j.relation_dir_path": "/tmp/amundsen/relations",
        "publisher.neo4j.neo4j_endpoint": os.environ["NEO4J_URL"],
        "publisher.neo4j.neo4j_user": os.environ["NEO4J_USER"],
        "publisher.neo4j.neo4j_password": os.environ["NEO4J_PASSWORD"],
        "publisher.neo4j.node_files_pattern": "/tmp/amundsen/nodes/part-*",
        "publisher.neo4j.relation_files_pattern": "/tmp/amundsen/relations/part-*",
    })

    task = DefaultTask(extractor=extractor, loader=loader, transformer=None)
    task.init(conf)
    task.run()

    publisher_task = DefaultTask(extractor=None, loader=None, publisher=publisher)
    publisher_task.init(conf)
    publisher_task.run()

    # Ping Vigilmon on successful ingestion
    try:
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
        logger.info("Databuilder BigQuery pipeline complete, heartbeat sent")
    except Exception as e:
        logger.warning(f"Failed to send Vigilmon heartbeat: {e}")

if __name__ == "__main__":
    run_bigquery_pipeline()

Wire Into an Airflow DAG

# dags/amundsen_ingestion.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
import requests
import os

VIGILMON_HEARTBEAT = os.environ.get("VIGILMON_HEARTBEAT_URL_AMUNDSEN_INGESTION")

def run_ingestion(**context):
    from pipelines.bigquery_metadata import run_bigquery_pipeline
    run_bigquery_pipeline()
    # Heartbeat is sent inside run_bigquery_pipeline on success
    # This outer task will raise if the pipeline raises, which is correct

with DAG(
    "amundsen_metadata_ingestion",
    default_args={
        "owner": "data-platform",
        "retries": 2,
        "retry_delay": timedelta(minutes=10),
        "email_on_failure": True,
        "email": ["data-platform@example.com"],
    },
    schedule_interval="0 */6 * * *",  # every 6 hours
    start_date=datetime(2026, 1, 1),
    catchup=False,
) as dag:
    ingest = PythonOperator(
        task_id="ingest_bigquery_to_amundsen",
        python_callable=run_ingestion,
    )

Kubernetes CronJob with Heartbeat

# k8s/amundsen-databuilder-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: amundsen-databuilder
  namespace: data-platform
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: databuilder
            image: your-registry/amundsen-databuilder:latest
            command:
            - sh
            - -c
            - |
              python /app/pipelines/bigquery_metadata.py && \
              curl -sf "$VIGILMON_HEARTBEAT_URL" || echo "heartbeat failed"
            env:
            - name: GCP_PROJECT_ID
              valueFrom:
                secretKeyRef:
                  name: amundsen-secrets
                  key: gcp-project-id
            - name: NEO4J_URL
              value: "bolt://neo4j:7687"
            - name: NEO4J_USER
              value: "neo4j"
            - name: NEO4J_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: amundsen-secrets
                  key: neo4j-password
            - name: VIGILMON_HEARTBEAT_URL
              valueFrom:
                secretKeyRef:
                  name: amundsen-secrets
                  key: vigilmon-heartbeat-url
          restartPolicy: OnFailure

Step 4: Alert Routing for Amundsen Failures

Configure alert routing based on user impact:

| Monitor | Alert Channel | Priority | |---|---|---| | Aggregated /health/amundsen | Slack + PagerDuty | P1 | | Metadata service /healthcheck | PagerDuty | P1 | | Frontend service /healthcheck | Slack | P1 | | Search service /healthcheck | Slack | P2 | | Heartbeat: Databuilder BigQuery | Slack + email | P2 | | Heartbeat: Databuilder Snowflake | Email | P3 |

Response time thresholds:

  • Alert at 5000ms for the aggregated health endpoint (cumulative latency from all three services)
  • Alert at 3000ms for the metadata service (Neo4j query time climbing)
  • Alert at 5000ms for the search service (Elasticsearch latency)

For large catalogs with millions of entities, consider a scheduled search smoke test — a separate monitor that runs a known-good search query and validates that it returns at least one result. This catches Elasticsearch index corruption before users report empty search results.

Summary

Amundsen is your team's data discovery front door — when it's down or stale, data engineers and analysts lose hours searching for datasets through other means, and confidence in the catalog erodes. External monitoring with Vigilmon catches failures before your users do:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/amundsen | All three Amundsen services + search query | | HTTP monitor on metadata /healthcheck | Metadata service + Neo4j connectivity | | HTTP monitor on search /healthcheck | Search service + Elasticsearch availability | | HTTP monitor on frontend /healthcheck | Web UI availability | | Heartbeat monitor | Databuilder ingestion pipeline liveness |

Get started free at vigilmon.online — your first Amundsen monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles