tutorial

How to Monitor LanceDB with Vigilmon

LanceDB powers vector search and multimodal AI retrieval at scale — but silent index corruption, vector search latency spikes, and storage compaction failures can degrade your AI applications without obvious errors. Learn how to monitor index health, vector search latency, storage compaction, and embedding ingestion rates with Vigilmon.

LanceDB is an open-source multimodal database built for AI applications — combining vector search, tabular data storage, and full-text search in a single embedded or serverless database. It stores data in the Lance columnar format (backed by Apache Arrow), supports ANN (approximate nearest neighbor) indexes for fast similarity search, and integrates natively with LangChain, LlamaIndex, and popular embedding providers. Teams use it to power RAG pipelines, semantic search, image retrieval, and recommendation systems.

But LanceDB failures can be subtle. An ANN index that was never built on a large table (falling back to slow brute-force scan), a compaction job that fell behind and left thousands of small fragment files degrading read performance, or an embedding ingestion pipeline that silently stopped writing new rows — none of these cause the database process to crash or return errors on basic health checks. Vigilmon adds the layer of external visibility your vector database infrastructure needs: search latency probes, index health checks, storage compaction monitoring, and heartbeat monitoring for ingestion pipelines.

Why LanceDB Needs External Monitoring

LanceDB can be embedded directly in your application or accessed through LanceDB Cloud. Either way, standard process monitoring is insufficient:

  • Index health checks verify that ANN indexes exist and are current — an unindexed table returns correct results but at O(n) scan cost instead of O(log n)
  • Vector search latency probes detect when search times have regressed due to index fragmentation, memory pressure, or large dataset growth
  • Storage compaction monitoring catches fragment accumulation before it degrades read performance
  • Embedding ingestion heartbeats prove that your pipeline is actively writing new vectors — a stalled ingestion pipeline means stale search results
  • Row count drift alerts detect when new data has stopped arriving, signaling an upstream pipeline failure

Step 1: Set Up LanceDB and Verify Basic Connectivity

LanceDB can run embedded (local Lance files) or against LanceDB Cloud. Check that your database is accessible:

import lancedb

# Local file-backed database
db = lancedb.connect("/data/lancedb")

# Or LanceDB Cloud
# db = lancedb.connect(
#     "db://your-project",
#     api_key=os.environ["LANCEDB_API_KEY"],
#     region="us-east-1"
# )

# List tables
tables = db.table_names()
print(f"Tables: {tables}")

# Open a table and check row count
table = db.open_table("documents")
print(f"Row count: {table.count_rows()}")

For local deployments, verify the Lance data directory is readable and contains the expected schema files:

ls /data/lancedb/
# Expected: documents.lance/  embeddings.lance/  metadata.lance/

ls /data/lancedb/documents.lance/
# Expected: _latest.manifest  data/  _indices/

Step 2: Build a LanceDB Health Endpoint

Wrap LanceDB in a Flask health endpoint that probes search latency, index existence, and fragment count:

# lancedb_health.py
import lancedb, os, time, numpy as np
from flask import Flask, jsonify

app = Flask(__name__)

DB_URI          = os.environ.get('LANCEDB_URI', '/data/lancedb')
API_KEY         = os.environ.get('LANCEDB_API_KEY')
TABLES_TO_CHECK = os.environ.get('LANCEDB_TABLES', 'documents').split(',')
VECTOR_DIM      = int(os.environ.get('LANCEDB_VECTOR_DIM', '1536'))
LATENCY_WARN_MS = int(os.environ.get('LATENCY_WARN_MS', '500'))
LATENCY_CRIT_MS = int(os.environ.get('LATENCY_CRIT_MS', '3000'))
FRAG_WARN       = int(os.environ.get('LANCEDB_FRAG_WARN', '100'))

def get_db():
    if API_KEY:
        return lancedb.connect(DB_URI, api_key=API_KEY)
    return lancedb.connect(DB_URI)

@app.route('/health/lancedb')
def health():
    issues = []
    table_stats = []

    try:
        db = get_db()
    except Exception as e:
        return jsonify(status='down', reason=f'connection_failed: {e}'), 503

    for table_name in TABLES_TO_CHECK:
        table_name = table_name.strip()
        try:
            table = db.open_table(table_name)
        except Exception as e:
            issues.append(f'table_{table_name}_not_found')
            continue

        # Row count
        try:
            row_count = table.count_rows()
        except Exception:
            row_count = -1
            issues.append(f'table_{table_name}_count_failed')

        # Vector search latency probe
        latency_ms = None
        try:
            query_vec = np.random.rand(VECTOR_DIM).tolist()
            t0 = time.time()
            results = table.search(query_vec).limit(1).to_list()
            latency_ms = int((time.time() - t0) * 1000)

            if latency_ms > LATENCY_CRIT_MS:
                issues.append(f'table_{table_name}_search_critical_{latency_ms}ms')
            elif latency_ms > LATENCY_WARN_MS:
                issues.append(f'table_{table_name}_search_slow_{latency_ms}ms')
        except Exception as e:
            issues.append(f'table_{table_name}_search_failed: {e}')

        # Index existence check
        has_index = False
        try:
            indices = table.list_indices()
            has_index = len(indices) > 0
            if not has_index and row_count > 10000:
                issues.append(f'table_{table_name}_no_index_large_table')
        except Exception:
            pass

        table_stats.append({
            'table': table_name,
            'row_count': row_count,
            'latency_ms': latency_ms,
            'has_index': has_index,
        })

    if issues:
        return jsonify(status='degraded', issues=issues, tables=table_stats), 503

    return jsonify(status='ok', tables=table_stats), 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3030)

Run the sidecar:

LANCEDB_URI=/data/lancedb \
LANCEDB_TABLES=documents,images,code_embeddings \
LANCEDB_VECTOR_DIM=1536 \
LATENCY_WARN_MS=300 \
LATENCY_CRIT_MS=2000 \
python lancedb_health.py

Step 3: Storage Compaction and Fragment Health Monitoring

LanceDB writes data in fragments (small Parquet-like files). Without periodic compaction, fragment count accumulates and degrades read performance as the query engine must scan many small files. Monitor fragment health:

@app.route('/health/lancedb/compaction')
def compaction():
    issues = []
    db = get_db()
    results = []

    for table_name in TABLES_TO_CHECK:
        table_name = table_name.strip()
        try:
            table = db.open_table(table_name)

            # Lance stores fragment metadata in the dataset stats
            stats = table.stats() if hasattr(table, 'stats') else {}

            # Check fragment count via dataset internals
            dataset = table.to_lance()
            fragments = list(dataset.get_fragments())
            frag_count = len(fragments)

            status = 'ok'
            if frag_count > FRAG_WARN:
                status = 'warn'
                issues.append(f'table_{table_name}_high_fragment_count_{frag_count}')

            results.append({
                'table': table_name,
                'fragment_count': frag_count,
                'status': status,
            })

        except Exception as e:
            issues.append(f'table_{table_name}_compaction_check_failed: {e}')

    http_status = 503 if issues else 200
    return jsonify(
        status='degraded' if issues else 'ok',
        issues=issues,
        tables=results,
    ), http_status

@app.route('/health/lancedb/compact', methods=['POST'])
def trigger_compact():
    """Trigger compaction on a table. Call from a maintenance script, not Vigilmon."""
    db = get_db()
    table_name = os.environ.get('COMPACT_TABLE', TABLES_TO_CHECK[0].strip())
    try:
        table = db.open_table(table_name)
        table.compact_files()
        table.cleanup_old_versions()
        return jsonify(status='ok', table=table_name, action='compacted'), 200
    except Exception as e:
        return jsonify(status='error', reason=str(e)), 500

Wire compaction into a scheduled job (cron or Paperclip routine) to run after high-ingestion periods:

# Run compaction after nightly ingestion job
python -c "
import lancedb
db = lancedb.connect('/data/lancedb')
for name in db.table_names():
    t = db.open_table(name)
    t.compact_files()
    t.cleanup_old_versions()
    print(f'Compacted {name}')
"

Step 4: Configure Vigilmon HTTP Monitors

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your LanceDB health endpoint: https://your-app.example.com/health/lancedb
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your Slack or PagerDuty integration
  7. Save the monitor

Add monitors for each health dimension:

| Monitor URL | Purpose | Interval | |---|---|---| | /health/lancedb | Search latency + table availability | 1 min | | /health/lancedb/compaction | Fragment count per table | 10 min |

For LanceDB Cloud deployments, also add an HTTP monitor directly against the LanceDB Cloud REST API endpoint to catch cloud-side outages independently of your application layer.

Step 5: Heartbeat Monitoring for Embedding Ingestion Pipelines

Most LanceDB deployments include a continuous or scheduled embedding ingestion pipeline — a worker that takes new documents, calls an embedding model, and writes vectors to LanceDB. A stalled ingestion pipeline means your vector database has stale content: searches return outdated or missing results.

Vigilmon heartbeat monitors detect pipeline staleness: your ingestion worker pings Vigilmon after each batch of vectors is written. Silence triggers an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: lancedb-embedding-ingestion
  3. Set the expected interval: 15 minutes
  4. Set the grace period: 30 minutes
  5. Save — copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123xyz

Wire Into Your Embedding Ingestion Pipeline

# embedding_ingestion.py
import lancedb, openai, requests, os, time

VIGILMON_URL   = os.environ['VIGILMON_HEARTBEAT_URL']
LANCEDB_URI    = os.environ.get('LANCEDB_URI', '/data/lancedb')
TABLE_NAME     = os.environ.get('LANCEDB_TABLE', 'documents')
BATCH_SIZE     = int(os.environ.get('BATCH_SIZE', '100'))
EMBEDDING_MODEL = os.environ.get('EMBEDDING_MODEL', 'text-embedding-3-small')

client = openai.OpenAI(api_key=os.environ['OPENAI_API_KEY'])

def embed_texts(texts):
    response = client.embeddings.create(input=texts, model=EMBEDDING_MODEL)
    return [e.embedding for e in response.data]

def ping_vigilmon():
    try:
        requests.get(VIGILMON_URL, timeout=5)
    except Exception:
        pass

def ingest_batch(documents):
    db = lancedb.connect(LANCEDB_URI)

    try:
        table = db.open_table(TABLE_NAME)
    except Exception:
        table = None

    texts = [doc['content'] for doc in documents]
    embeddings = embed_texts(texts)

    rows = [
        {
            'id': doc['id'],
            'content': doc['content'],
            'vector': embedding,
            'metadata': doc.get('metadata', {}),
            'ingested_at': int(time.time()),
        }
        for doc, embedding in zip(documents, embeddings)
    ]

    if table is None:
        import pyarrow as pa
        schema = pa.schema([
            pa.field('id', pa.string()),
            pa.field('content', pa.string()),
            pa.field('vector', pa.list_(pa.float32(), len(embeddings[0]))),
            pa.field('ingested_at', pa.int64()),
        ])
        db.create_table(TABLE_NAME, data=rows, schema=schema)
    else:
        table.add(rows)

    ping_vigilmon()
    print(f"Ingested {len(rows)} documents, heartbeat sent")

def main():
    batch = []
    for doc in get_documents_from_source():   # your document source
        batch.append(doc)
        if len(batch) >= BATCH_SIZE:
            ingest_batch(batch)
            batch = []
    if batch:
        ingest_batch(batch)

if __name__ == '__main__':
    main()

For incremental ingestion pipelines running continuously, track the last ingested document ID or timestamp and ping Vigilmon after each successful write window to prove the pipeline is current.

Step 6: Row Count Drift Monitoring

For tables where new embeddings are expected regularly, monitor row count growth. A table that stopped growing signals a broken upstream pipeline:

@app.route('/health/lancedb/growth')
def growth():
    """Check that tables have grown since last check (indicates active ingestion)."""
    import json
    from pathlib import Path

    SNAPSHOT_PATH = Path('/tmp/lancedb_row_counts.json')
    db = get_db()
    issues = []
    current = {}

    for table_name in TABLES_TO_CHECK:
        table_name = table_name.strip()
        try:
            table = db.open_table(table_name)
            current[table_name] = table.count_rows()
        except Exception:
            current[table_name] = -1

    # Compare with previous snapshot if it exists
    if SNAPSHOT_PATH.exists():
        previous = json.loads(SNAPSHOT_PATH.read_text())
        for name, count in current.items():
            prev_count = previous.get(name, 0)
            if count <= prev_count and count > 0:
                issues.append(f'table_{name}_no_growth_since_last_check')

    SNAPSHOT_PATH.write_text(json.dumps(current))

    http_status = 503 if issues else 200
    return jsonify(
        status='degraded' if issues else 'ok',
        issues=issues,
        row_counts=current,
    ), http_status

Alert Routing

| Monitor | Alert Channel | Priority | |---|---|---| | Search latency /health/lancedb | Slack + PagerDuty | P1 | | Compaction health /health/lancedb/compaction | Slack | P2 | | Row count growth /health/lancedb/growth | Slack | P2 | | Heartbeat: embedding ingestion | Slack + email | P2 |

Latency threshold guidelines:

  • 300ms vector search latency: warn — table may lack an ANN index or index is stale
  • 2000ms vector search latency: critical — falling back to brute-force scan on a large table
  • 100+ fragments per table: warn — compaction overdue
  • 500+ fragments per table: critical — read performance significantly degraded

For RAG applications where search latency directly affects user response time, set tighter thresholds: 200ms warn, 800ms critical.

Summary

LanceDB performance and freshness failures are invisible to standard uptime monitoring — the application stays alive while searches slow down or return stale results. External monitoring with Vigilmon catches these conditions before users notice:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/lancedb | Search latency and table availability | | HTTP monitor on /health/lancedb/compaction | Fragment accumulation and storage health | | HTTP monitor on /health/lancedb/growth | Row count drift and ingestion pipeline activity | | Heartbeat monitor | Embedding ingestion pipeline liveness |

Get started free at vigilmon.online — your first LanceDB monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles