Monitoring Manticore Search with Vigilmon: HTTP Health Checks, Index Status, and Alerting

Manticore Search is a high-performance open-source full-text search engine forked from Sphinx Search, widely used for e-commerce product search, log analysis, and real-time data exploration. When Manticore goes down or an index becomes unavailable, search queries return errors or empty results — often silently from the application's perspective. Vigilmon gives you external visibility into your Manticore Search nodes through HTTP API health checks and heartbeat monitoring for indexing pipelines.

What You'll Build

• An HTTP monitor on Manticore's /sql HTTP API for node availability
• An index-level health check using the Manticore HTTP API
• A heartbeat monitor for real-time indexing jobs
• Alerting configured for search engine outages

Prerequisites

• A running Manticore Search instance (single node or cluster) with HTTP API enabled
• Manticore HTTP API accessible at http://manticore.example.com:9308 (default port)
• A free account at vigilmon.online

Step 1: Verify the Manticore HTTP API

Manticore exposes an HTTP API on port 9308 by default. Confirm it responds before configuring monitors:

curl -s http://manticore.example.com:9308/sql \
  -d "mode=raw&query=SHOW STATUS"

A healthy node returns a JSON array with status variables:

[
  {
    "columns": [{"Variable_name": {"type": "string"}}, {"Value": {"type": "string"}}],
    "data": [
      {"Variable_name": "uptime", "Value": "12345"},
      {"Variable_name": "connections", "Value": "5"},
      {"Variable_name": "maxed_out", "Value": "0"}
    ],
    "total": 20,
    "error": "",
    "warning": ""
  }
]

The "error": "" field in the response indicates no errors. If the HTTP API is not enabled, add the following to manticore.conf:

searchd {
    # ...existing config...
    listen = 9308:http
}

Restart Manticore after the config change:

sudo systemctl restart manticore

Step 2: Build a Dedicated Health Endpoint

Rather than probing /sql directly with a POST body, expose a thin wrapper that Vigilmon can GET:

Node.js Example

// manticore-health.js — thin HTTP wrapper for Manticore health
const express = require('express');
const axios = require('axios');

const app = express();
const MANTICORE_URL = process.env.MANTICORE_URL || 'http://localhost:9308';

app.get('/health/manticore', async (req, res) => {
  try {
    const response = await axios.post(
      `${MANTICORE_URL}/sql`,
      'mode=raw&query=SHOW STATUS',
      { headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, timeout: 5000 }
    );

    const data = response.data;
    const hasError = data[0]?.error && data[0].error !== '';

    if (hasError) {
      return res.status(503).json({ status: 'degraded', error: data[0].error });
    }

    return res.status(200).json({ status: 'ok', uptime: data[0]?.data?.[0]?.Value });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Python Example

# manticore_health.py — Manticore Search health wrapper
import os
import requests
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
MANTICORE_URL = os.environ.get("MANTICORE_URL", "http://localhost:9308")

@app.get("/health/manticore")
def manticore_health():
    try:
        resp = requests.post(
            f"{MANTICORE_URL}/sql",
            data="mode=raw&query=SHOW STATUS",
            headers={"Content-Type": "application/x-www-form-urlencoded"},
            timeout=5,
        )
        resp.raise_for_status()
        data = resp.json()
        error = data[0].get("error", "") if data else "empty response"

        if error:
            return JSONResponse(status_code=503, content={"status": "degraded", "error": error})

        return {"status": "ok"}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Deploy this wrapper and verify it:

curl -i http://your-app.example.com:3001/health/manticore
# HTTP/1.1 200 OK
# {"status":"ok","uptime":"12345"}

Step 3: Create a Vigilmon HTTP Monitor for Manticore

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: http://your-app.example.com:3001/health/manticore
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: "status":"ok".
7. Click Save.

If you want to probe Manticore's HTTP API directly without a wrapper, use a POST-capable monitor:

1. Add Monitor → HTTP.
2. URL: http://manticore.example.com:9308/sql.
3. Method: POST.
4. Body: mode=raw&query=SHOW STATUS.
5. Headers: Content-Type: application/x-www-form-urlencoded.
6. Keyword: "error":"".
7. Click Save.

What This Catches

| Failure Mode | Process Monitor | Vigilmon | |---|---|---| | Manticore process crash | ✓ | ✓ | | HTTP API not responding on port 9308 | ✗ | ✓ | | Configuration error preventing startup | ✗ | ✓ | | Network partition from application servers | ✗ | ✓ | | Manticore returning SQL errors | ✗ | ✓ |

Step 4: Monitor Individual Index Availability

Check that specific search indices are present and queryable:

curl -s http://manticore.example.com:9308/sql \
  -d "mode=raw&query=SHOW TABLES LIKE 'products'"

Add an index-level monitor:

1. Add Monitor → HTTP.
2. URL: http://your-app.example.com:3001/health/manticore/index/products (adjust to your wrapper).
3. Check interval: 2 minutes.
4. Keyword: "status":"ok".

Or extend your wrapper to support index checks:

app.get('/health/manticore/index/:name', async (req, res) => {
  try {
    const { name } = req.params;
    const response = await axios.post(
      `${MANTICORE_URL}/sql`,
      `mode=raw&query=SHOW TABLES LIKE '${name}'`,
      { headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, timeout: 5000 }
    );
    const rows = response.data[0]?.data || [];
    if (rows.length === 0) {
      return res.status(503).json({ status: 'missing', index: name });
    }
    return res.status(200).json({ status: 'ok', index: name });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

Step 5: Heartbeat Monitoring for Indexing Pipelines

Manticore is often fed by real-time indexing jobs — MySQL replication, ETL scripts, or custom producers. If the indexing pipeline stalls, your search index goes stale silently.

Vigilmon heartbeat monitors alert you when an expected ping stops arriving from your indexing job.

Set Up the Heartbeat Monitor

1. In Vigilmon → Add Monitor → Heartbeat.
2. Name: manticore-indexer.
3. Expected interval: Match your indexing cadence (e.g., 5 minutes for periodic batch jobs).
4. Grace period: 10 minutes.
5. Copy the unique heartbeat URL: https://vigilmon.online/heartbeat/your-unique-id.

Wire Into Your Indexing Script

Python batch indexer:

import requests
import os

VIGILMON_HEARTBEAT = os.environ["VIGILMON_HEARTBEAT_URL"]

def run_index_cycle():
    # ... your indexing logic here ...
    docs = fetch_new_documents()
    push_to_manticore(docs)

    # Signal success to Vigilmon
    try:
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
    except Exception:
        pass  # Don't let heartbeat failure break indexing

if __name__ == "__main__":
    while True:
        run_index_cycle()
        time.sleep(300)  # 5-minute interval

Shell cron job:

#!/bin/bash
# /etc/cron.d/manticore-indexer — runs every 5 minutes

python3 /opt/indexer/run.py && \
  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null 2>&1

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, set up your alert routing:

| Monitor | Trigger | Recommended Action | |---|---|---| | Manticore node health | 503 or keyword absent | Check Manticore process; review searchd logs | | Index availability | Index missing or 503 | Check if index was accidentally dropped; verify ETL | | Indexer heartbeat | Ping not received | Check indexing job logs; restart if stalled |

Recommended thresholds:

• Confirmation period: 2 consecutive failures before alerting (avoids false positives from brief restarts)
• Response time alert: 2000ms (Manticore health checks should be very fast; slow response indicates load or memory pressure)
• Recovery notification: Enable "alert on recovery" so your team knows when the monitor goes green

Common Manticore Search Failure Modes

| Scenario | What Vigilmon Catches | |---|---| | searchd process crash | HTTP health check fails immediately | | Port 9308 not bound (config error) | Connection refused on monitor | | Index corruption on restart | Index availability check returns 503 | | Indexing pipeline stalled | Heartbeat monitor fires | | OOM kill under heavy search load | Node health check fails | | Network issue between app and Manticore | External probe catches what internal checks miss |

Manticore Search moves fast and fails quietly when something goes wrong at the process or network level. Vigilmon's external HTTP probes and heartbeat monitors give you the outside-in view that internal process managers can't provide.

Get started free at vigilmon.online — your Manticore monitor is running in under 5 minutes.