tutorial

Monitoring Tantivy Full-Text Search with Vigilmon: HTTP Health Checks, Index Status, and Alerting

How to monitor Tantivy full-text search with Vigilmon — health endpoint integration for Tantivy-backed services, index availability checks, heartbeat monitoring, and alerting.

Tantivy is a high-performance full-text search engine library written in Rust, used as the search core behind projects like Quickwit, Meilisearch's early versions, and many custom Rust applications. Like Bleve in Go, Tantivy is embedded as a library rather than running as a standalone server daemon. Monitoring a Tantivy deployment means monitoring the application or service that wraps it. Vigilmon integrates through a health endpoint your Rust service exposes, giving you external visibility into index health, reader freshness, and indexing pipeline liveness.

What You'll Build

  • An HTTP health endpoint in your Rust service (Axum or Actix-Web) reporting Tantivy index state
  • A Vigilmon HTTP monitor against the health endpoint
  • A segment count and reader freshness check to detect stale readers
  • A heartbeat monitor for background indexing threads or tasks

Prerequisites

  • A Rust application embedding Tantivy (0.21+) with an HTTP server already running
  • Or a Quickwit or other Tantivy-backed service exposing a health API
  • A free account at vigilmon.online

Step 1: Expose a Tantivy Health Endpoint in Your Rust Service

Tantivy's IndexReader and Index types expose useful state: segment count, number of documents, and whether a reader reload has succeeded. Add a /health/tantivy route to your HTTP server:

Axum Example

// health.rs — Tantivy health handler for Axum
use axum::{extract::State, http::StatusCode, response::Json};
use serde::Serialize;
use std::sync::Arc;
use tantivy::{Index, IndexReader};

#[derive(Serialize)]
pub struct TantivyHealth {
    pub status: &'static str,
    pub num_docs: u64,
    pub num_segments: usize,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub error: Option<String>,
}

pub struct AppState {
    pub index: Arc<Index>,
    pub reader: Arc<IndexReader>,
}

pub async fn tantivy_health(
    State(state): State<Arc<AppState>>,
) -> Result<Json<TantivyHealth>, (StatusCode, Json<TantivyHealth>)> {
    // Reload the reader to get fresh segment view
    if let Err(e) = state.reader.reload() {
        return Err((
            StatusCode::SERVICE_UNAVAILABLE,
            Json(TantivyHealth {
                status: "degraded",
                num_docs: 0,
                num_segments: 0,
                error: Some(e.to_string()),
            }),
        ));
    }

    let searcher = state.reader.searcher();
    let num_docs = searcher.num_docs();
    let num_segments = searcher.segment_readers().len();

    Ok(Json(TantivyHealth {
        status: "ok",
        num_docs,
        num_segments,
        error: None,
    }))
}

Register the route in your Axum router:

use axum::{routing::get, Router};
use std::sync::Arc;

#[tokio::main]
async fn main() {
    let index = Index::open_in_dir("./search_index").unwrap();
    let reader = index
        .reader_builder()
        .reload_policy(tantivy::ReloadPolicy::OnCommitWithDelay)
        .try_into()
        .unwrap();

    let state = Arc::new(AppState {
        index: Arc::new(index),
        reader: Arc::new(reader),
    });

    let app = Router::new()
        .route("/health/tantivy", get(tantivy_health))
        // ... your other routes ...
        .with_state(state);

    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await.unwrap();
    axum::serve(listener, app).await.unwrap();
}

Verify the endpoint:

curl -i http://your-app.example.com:8080/health/tantivy
# HTTP/1.1 200 OK
# {"status":"ok","num_docs":58432,"num_segments":12}

Actix-Web Example

// health.rs — Tantivy health handler for Actix-Web
use actix_web::{web, HttpResponse};
use serde::Serialize;
use std::sync::{Arc, RwLock};
use tantivy::IndexReader;

#[derive(Serialize)]
struct TantivyHealth {
    status: &'static str,
    num_docs: u64,
    num_segments: usize,
}

pub async fn tantivy_health(
    reader: web::Data<Arc<RwLock<IndexReader>>>,
) -> HttpResponse {
    let reader = reader.read().unwrap();
    if reader.reload().is_err() {
        return HttpResponse::ServiceUnavailable().json(serde_json::json!({
            "status": "degraded"
        }));
    }
    let searcher = reader.searcher();
    HttpResponse::Ok().json(TantivyHealth {
        status: "ok",
        num_docs: searcher.num_docs(),
        num_segments: searcher.segment_readers().len(),
    })
}

Step 2: Monitor Quickwit or Other Tantivy Services

If you're running Quickwit (which embeds Tantivy), it exposes a built-in health endpoint:

# Quickwit liveness check
curl -i http://quickwit.example.com:7280/health/livez

# Quickwit readiness check (checks index availability)
curl -i http://quickwit.example.com:7280/health/readyz

A ready Quickwit node returns 200. Use these directly in Vigilmon without building a custom wrapper:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://quickwit.example.com:7280/health/readyz
  3. Expected status: 200.
  4. Check interval: 60 seconds.
  5. Click Save.

Step 3: Create a Vigilmon HTTP Monitor for Tantivy

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://your-app.example.com:8080/health/tantivy
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "status":"ok".
  7. Click Save.

What This Catches

| Failure Mode | Application Logs | Vigilmon | |---|---|---| | Rust service panic or crash | ✓ | ✓ | | Tantivy index directory missing | ✓ | ✓ | | Segment merge failure corrupting reader | ✓ | ✓ | | Disk full — commits failing silently | ✗ | ✓ | | Network partition from reverse proxy | ✗ | ✓ | | Reader reload error after schema change | ✓ | ✓ |

Step 4: Check Index Segment Health and Reader Freshness

A large number of segments relative to committed documents can indicate a merge policy issue, which degrades query performance without surfacing as a failure. Extend the health endpoint to expose segment density:

#[derive(Serialize)]
pub struct TantivyDetailedHealth {
    pub status: &'static str,
    pub num_docs: u64,
    pub num_segments: usize,
    pub segment_density: f64,  // docs per segment, lower = more fragmented
    pub last_commit_opstamp: u64,
}

pub async fn tantivy_health_detailed(
    State(state): State<Arc<AppState>>,
) -> Json<TantivyDetailedHealth> {
    let _ = state.reader.reload();
    let searcher = state.reader.searcher();
    let num_docs = searcher.num_docs();
    let num_segments = searcher.segment_readers().len();
    let segment_density = if num_segments > 0 {
        num_docs as f64 / num_segments as f64
    } else {
        0.0
    };

    let meta = state.index.load_metas().unwrap_or_default();
    Json(TantivyDetailedHealth {
        status: "ok",
        num_docs,
        num_segments,
        segment_density,
        last_commit_opstamp: meta.opstamp,
    })
}

Monitor last_commit_opstamp in conjunction with your expected commit frequency — if the opstamp stops advancing, your writer has stalled.

Step 5: Heartbeat Monitoring for Indexing Tasks

Tantivy indexing typically runs in a background thread or async task. If the writer panics, deadlocks on a lock, or encounters an unhandled error, the index silently stops updating.

Vigilmon heartbeat monitors alert you when an expected ping stops arriving from your indexing task.

Set Up the Heartbeat Monitor

  1. In Vigilmon → Add Monitor → Heartbeat.
  2. Name: tantivy-writer.
  3. Expected interval: Match your commit cycle (e.g., 1 minute for near-real-time indexing).
  4. Grace period: 5 minutes.
  5. Copy the unique heartbeat URL: https://vigilmon.online/heartbeat/your-unique-id.

Wire Into Your Indexing Task

use std::time::Duration;
use tantivy::{Index, IndexWriter, Document};
use tokio::time;

async fn run_indexer(index: Index) {
    let heartbeat_url = std::env::var("VIGILMON_HEARTBEAT_URL").ok();
    let mut writer: IndexWriter = index.writer(50_000_000).unwrap();

    let mut interval = time::interval(Duration::from_secs(60));
    loop {
        interval.tick().await;

        match index_batch(&mut writer).await {
            Ok(count) => {
                tracing::info!("indexed {} documents", count);
                // Ping Vigilmon heartbeat on success
                if let Some(url) = &heartbeat_url {
                    let _ = reqwest::get(url).await;
                }
            }
            Err(e) => {
                tracing::error!("indexing failed: {}", e);
                // Do NOT ping heartbeat on error — this is what triggers the alert
            }
        }
    }
}

async fn index_batch(writer: &mut IndexWriter) -> Result<usize, tantivy::TantivyError> {
    let docs = fetch_new_documents().await; // your fetch logic
    let count = docs.len();
    for doc in docs {
        writer.add_document(doc)?;
    }
    writer.commit()?;
    Ok(count)
}

Shell script for external indexing processes:

#!/bin/bash
# Run Tantivy indexer and ping heartbeat on success
./target/release/indexer --config /etc/indexer.toml && \
  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null 2>&1

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, set up your alert routing:

| Monitor | Trigger | Recommended Action | |---|---|---| | Tantivy HTTP health | 503 or keyword absent | Check if Rust service is running; inspect panic logs | | Detailed segment health | High segment count with low density | Force a merge; check merge policy configuration | | Writer heartbeat | Ping not received | Check writer thread/task; look for lock contention |

Recommended thresholds:

  • Confirmation period: 2 consecutive failures before alerting (avoids noise from rolling restarts or GC pressure in the host OS)
  • Response time alert: 1500ms (Tantivy health checks are fast; slowness indicates I/O pressure or lock contention on the reader)
  • Recovery notification: Enable "alert on recovery" so your team knows when indexing resumes

Common Tantivy Failure Modes

| Scenario | What Vigilmon Catches | |---|---| | Rust service panic on startup | HTTP health check never responds | | Index directory missing or corrupted | Health endpoint returns 503 | | IndexWriter lock held by crashed process | New writer fails; application returns errors; health catches | | Disk full — commits fail silently | Heartbeat stops arriving; freshness endpoint stales | | Merge thread panic | Segment count grows unbounded; detailed health catches | | Network partition from reverse proxy | External probe catches what local checks miss |

Tantivy fails quietly — a panicked writer thread or a corrupt index directory produces no visible HTTP error to downstream callers until queries start returning wrong results. Vigilmon's health endpoint monitors and heartbeat checks give you the outside-in visibility that Rust's unwrap() chains and background thread panics otherwise hide.

Get started free at vigilmon.online — your Tantivy monitor is running in under 5 minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles