tutorial

Monitoring Bleve Full-Text Search with Vigilmon: HTTP Health Checks, Index Status, and Alerting

How to monitor Bleve full-text search with Vigilmon — HTTP health endpoint integration, index availability checks, heartbeat monitoring for indexing pipelines, and alerting.

Bleve is a full-text search and indexing library for Go, embedded directly into applications rather than running as a separate server process. It powers search in tools like Couchbase, CockroachDB's docs, and countless custom Go services. Because Bleve is a library — not a standalone daemon — monitoring it means monitoring the application that embeds it. Vigilmon integrates through a health endpoint you expose from your Go application, giving you external visibility into index availability and indexing pipeline health.

What You'll Build

  • An HTTP health endpoint inside your Go application that reports Bleve index status
  • A Vigilmon HTTP monitor checking the health endpoint
  • An index document count and size check to detect stale or corrupt indices
  • A heartbeat monitor for background indexing goroutines

Prerequisites

  • A Go application embedding Bleve (v2 or v1) with an HTTP server already running
  • Bleve HTTP server mode, if you're using the standalone bleve CLI
  • A free account at vigilmon.online

Step 1: Expose a Bleve Health Endpoint in Your Go Application

Bleve indices expose a DocCount() method and an Stats() method that report index health. Add a /health/bleve route to your existing HTTP server:

// health.go — Bleve health handler
package main

import (
	"encoding/json"
	"net/http"

	"github.com/blevesearch/bleve/v2"
)

type BleveHealthResponse struct {
	Status     string `json:"status"`
	DocCount   uint64 `json:"doc_count"`
	IndexName  string `json:"index_name"`
	Error      string `json:"error,omitempty"`
}

// RegisterBleveHealth wires up /health/bleve on the given mux.
func RegisterBleveHealth(mux *http.ServeMux, idx bleve.Index) {
	mux.HandleFunc("/health/bleve", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "application/json")

		count, err := idx.DocCount()
		if err != nil {
			w.WriteHeader(http.StatusServiceUnavailable)
			json.NewEncoder(w).Encode(BleveHealthResponse{
				Status: "degraded",
				Error:  err.Error(),
			})
			return
		}

		resp := BleveHealthResponse{
			Status:    "ok",
			DocCount:  count,
			IndexName: idx.Name(),
		}
		json.NewEncoder(w).Encode(resp)
	})
}

Wire it into your application's server initialization:

func main() {
	index, err := bleve.Open("./search.bleve")
	if err != nil {
		// Attempt to create if missing
		mapping := bleve.NewIndexMapping()
		index, err = bleve.New("./search.bleve", mapping)
		if err != nil {
			log.Fatalf("failed to open or create bleve index: %v", err)
		}
	}
	defer index.Close()

	mux := http.NewServeMux()
	RegisterBleveHealth(mux, index)
	// ... register your other routes ...

	log.Println("listening on :8080")
	http.ListenAndServe(":8080", mux)
}

Verify the endpoint:

curl -i http://your-app.example.com:8080/health/bleve
# HTTP/1.1 200 OK
# {"status":"ok","doc_count":14823,"index_name":"search.bleve"}

If your application uses multiple indices, expose one health route per index:

RegisterBleveHealth(mux, productIndex)      // /health/bleve
RegisterBleveHealthNamed(mux, articleIndex, "/health/bleve/articles")

Step 2: Add an External Wrapper for Library-Only Applications

If you cannot modify the application embedding Bleve, run a lightweight sidecar that opens the same Bleve index path in read-only mode and exposes a health endpoint. Note that Bleve indexes support only one writer at a time; a read-only sidecar is safe alongside a running writer:

// sidecar/main.go — read-only Bleve health sidecar
package main

import (
	"encoding/json"
	"log"
	"net/http"
	"os"

	"github.com/blevesearch/bleve/v2"
)

func main() {
	indexPath := os.Getenv("BLEVE_INDEX_PATH")
	if indexPath == "" {
		indexPath = "./search.bleve"
	}

	index, err := bleve.OpenUsing(indexPath, map[string]interface{}{
		"read_only": true,
	})
	if err != nil {
		log.Fatalf("failed to open bleve index: %v", err)
	}
	defer index.Close()

	http.HandleFunc("/health/bleve", func(w http.ResponseWriter, r *http.Request) {
		count, err := index.DocCount()
		w.Header().Set("Content-Type", "application/json")
		if err != nil {
			w.WriteHeader(503)
			json.NewEncoder(w).Encode(map[string]string{"status": "degraded", "error": err.Error()})
			return
		}
		json.NewEncoder(w).Encode(map[string]interface{}{"status": "ok", "doc_count": count})
	})

	log.Println("bleve health sidecar on :3003")
	http.ListenAndServe(":3003", nil)
}

Step 3: Create a Vigilmon HTTP Monitor for Bleve

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://your-app.example.com:8080/health/bleve
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "status":"ok".
  7. Click Save.

What This Catches

| Failure Mode | Application Logs | Vigilmon | |---|---|---| | Application crash | ✓ | ✓ | | Bleve index open failure on startup | ✓ | ✓ | | Index corruption (DocCount error) | ✓ | ✓ | | Disk full causing index writes to fail | ✗ | ✓ | | Network partition from load balancer | ✗ | ✓ | | Health endpoint returning 503 | ✓ | ✓ |

Step 4: Monitor Index Document Count for Staleness

An index can be healthy in terms of availability but stale if the background indexer has stopped updating it. Extend the health endpoint to expose a last_indexed_at timestamp from your application state:

import (
	"sync/atomic"
	"time"
)

var lastIndexedAt atomic.Value // stores time.Time

func onDocumentIndexed() {
	lastIndexedAt.Store(time.Now())
}

mux.HandleFunc("/health/bleve/freshness", func(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")

	last, ok := lastIndexedAt.Load().(time.Time)
	if !ok || last.IsZero() {
		w.WriteHeader(503)
		json.NewEncoder(w).Encode(map[string]string{"status": "no_data"})
		return
	}

	age := time.Since(last)
	if age > 15*time.Minute {
		w.WriteHeader(503)
		json.NewEncoder(w).Encode(map[string]interface{}{
			"status":         "stale",
			"age_seconds":    int(age.Seconds()),
			"last_indexed_at": last.Format(time.RFC3339),
		})
		return
	}

	json.NewEncoder(w).Encode(map[string]interface{}{
		"status":         "ok",
		"age_seconds":    int(age.Seconds()),
		"last_indexed_at": last.Format(time.RFC3339),
	})
})

Add a Vigilmon monitor for /health/bleve/freshness with keyword "status":"ok" and a 2-minute check interval.

Step 5: Heartbeat Monitoring for Background Indexing

Bleve indexing often runs in a background goroutine or a separate process that batches documents. If that goroutine panics or deadlocks, the index silently stops updating.

Vigilmon heartbeat monitors alert you when an expected ping stops arriving from your indexing job.

Set Up the Heartbeat Monitor

  1. In Vigilmon → Add Monitor → Heartbeat.
  2. Name: bleve-indexer.
  3. Expected interval: Match your indexing goroutine's sleep cycle (e.g., 5 minutes for batch indexing).
  4. Grace period: 10 minutes.
  5. Copy the unique heartbeat URL: https://vigilmon.online/heartbeat/your-unique-id.

Wire Into Your Indexing Goroutine

package main

import (
	"log"
	"net/http"
	"os"
	"time"

	"github.com/blevesearch/bleve/v2"
)

func runIndexer(index bleve.Index) {
	heartbeatURL := os.Getenv("VIGILMON_HEARTBEAT_URL")
	ticker := time.NewTicker(5 * time.Minute)
	defer ticker.Stop()

	for range ticker.C {
		if err := indexBatch(index); err != nil {
			log.Printf("indexing error: %v", err)
			continue // Don't ping heartbeat on error
		}

		// Signal successful cycle to Vigilmon
		if heartbeatURL != "" {
			go func() {
				resp, err := http.Get(heartbeatURL)
				if err == nil {
					resp.Body.Close()
				}
			}()
		}
	}
}

func indexBatch(index bleve.Index) error {
	docs := fetchNewDocuments() // your fetch logic
	batch := index.NewBatch()
	for _, doc := range docs {
		if err := batch.Index(doc.ID, doc); err != nil {
			return err
		}
	}
	return index.Batch(batch)
}

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, set up your alert routing:

| Monitor | Trigger | Recommended Action | |---|---|---| | Bleve HTTP health | 503 or keyword absent | Check application logs; verify index path is accessible | | Index freshness | Stale or no data | Check indexing goroutine; look for panics in logs | | Indexer heartbeat | Ping not received | Restart the indexing process; inspect for deadlocks or OOM |

Recommended thresholds:

  • Confirmation period: 2 consecutive failures before alerting (avoids false positives from brief application restarts or GC pauses)
  • Response time alert: 2000ms (Bleve's DocCount is fast; slow response may indicate index lock contention or disk I/O pressure)
  • Recovery notification: Enable "alert on recovery" so your team knows when the monitor returns to green

Common Bleve Failure Modes

| Scenario | What Vigilmon Catches | |---|---| | Application process crash | HTTP health check fails immediately | | Bleve index open failure at startup | Health endpoint returns 503 | | Index corruption (bolt/scorch backend) | DocCount returns an error; health returns 503 | | Disk full — writes silently discarded | Freshness endpoint ages out; heartbeat stops | | Background indexer goroutine panic | Heartbeat monitor fires | | Network issue between load balancer and app | External probe catches what internal metrics miss |

Bleve's library nature means there's no separate process to monitor — failures surface through your application, often silently. Vigilmon's external HTTP health checks and heartbeat monitors give you the outside-in view that application-internal metrics and log scrapers can miss.

Get started free at vigilmon.online — your Bleve monitor is running in under 5 minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles