tutorial

Monitoring ManticoreSearch with Vigilmon: HTTP REST API, SQL Engine Health, MySQL Protocol TCP Probe & Index Integrity Heartbeat

How to monitor ManticoreSearch with Vigilmon — HTTP REST API availability on port 9308, SQL engine SHOW STATUS via the /sql endpoint, TCP probe on port 9306 for the MySQL protocol interface, SSL certificate alerts, and a scheduled index document count heartbeat to detect corrupted or cleared real-time indexes.

ManticoreSearch is a drop-in replacement for Sphinx Search and a fast, memory-efficient alternative to Elasticsearch for full-text search workloads. When ManticoreSearch goes down, every application depending on it for search — product catalogs, forum search, document retrieval — falls back to silent errors or broken search results. Unlike database failures, search engine downtime often produces degraded user experience rather than hard errors: empty results, stale data, or slow pagination that teams dismiss as a bug rather than infrastructure failure. Vigilmon gives you continuous external visibility into ManticoreSearch's HTTP REST API, its MySQL-compatible SQL engine, the MySQL protocol port that application libraries connect on, and the health of your real-time indexes before data loss or corruption affects end users.

What You'll Build

  • An HTTP monitor on ManticoreSearch's REST API root (/) on port 9308 for version confirmation
  • An HTTP monitor on the /sql endpoint with a SHOW STATUS probe to confirm the SQL engine is operational
  • A TCP monitor on port 9306 to verify the MySQL protocol interface is listening
  • An SSL certificate monitor for HTTPS-proxied ManticoreSearch deployments
  • A heartbeat monitor with an index document count check to detect real-time index corruption or clearing

Prerequisites

  • ManticoreSearch running on your server (HTTP API on port 9308, MySQL protocol on port 9306)
  • A domain or IP:port accessible from the internet (or an internal network if monitoring internally)
  • A free account at vigilmon.online

Step 1: Check the ManticoreSearch HTTP REST API

ManticoreSearch's HTTP REST API listens on port 9308. A GET request to the root returns version information confirming the HTTP endpoint is serving:

curl http://manticore.yourdomain.com:9308/
# {"version":{"build_date":"2024-01-15T10:00:00Z","commit":"abc123","version":"6.2.12"}}

A 200 response confirms the C++ HTTP server is running and the REST API is accepting connections. A connection refused on port 9308 means ManticoreSearch has crashed or failed to start.

Step 2: Create the REST API Monitor in Vigilmon

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://manticore.yourdomain.com:9308/
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: version (matches the version object in the JSON response).
  7. Label: ManticoreSearch REST API
  8. Click Save.

This monitor catches:

  • ManticoreSearch process crashes or segmentation faults (common in C++ search engines under heavy indexing load)
  • Configuration errors on restart that prevent the HTTP listener from binding to port 9308
  • Out-of-memory kills when indexing large datasets
  • File descriptor exhaustion that causes the HTTP server to stop accepting new connections

Step 3: Monitor the SQL Engine via the /sql Endpoint

The HTTP REST API on port 9308 also accepts SQL queries via POST to /sql. Sending a SHOW STATUS query returns key operational metrics including uptime, active connections, and the number of loaded indexes — confirming the ManticoreSearch SQL engine is fully operational:

curl -X POST http://manticore.yourdomain.com:9308/sql \
  -d 'query=SHOW STATUS'
# [{"columns":[...],"data":[{"Counter":"uptime","Value":"3600"},{"Counter":"connections","Value":"12"},{"Counter":"index_count","Value":"3"},...]}]

A 200 response with a Counter array confirms the SQL query engine and index metadata are healthy. A connection error or 500 response while the REST API root returns 200 indicates the SQL engine subsystem has degraded.

  1. Add Monitor → HTTP.
  2. URL: http://manticore.yourdomain.com:9308/sql
  3. HTTP method: POST.
  4. Request body: query=SHOW STATUS
  5. Expected status: 200.
  6. Keyword: uptime (present in the SHOW STATUS output).
  7. Check interval: 60 seconds.
  8. Label: ManticoreSearch SQL Engine
  9. Click Save.

If the REST API root returns 200 but this check fails, the HTTP listener is up but the SQL engine has encountered an internal error — this distinction helps narrow down whether the failure is in ManticoreSearch's HTTP layer or its core indexing subsystem.

Step 4: TCP Probe on the MySQL Protocol Port

ManticoreSearch provides a MySQL-compatible wire protocol on port 9306. Applications using MySQL client libraries (PHP PDO, Python mysql-connector, Ruby mysql2, Java JDBC) connect on this port — not port 9308. A failure on port 9306 while port 9308 is healthy means the MySQL listener has crashed or the port is unreachable, and all applications using MySQL clients to connect to ManticoreSearch will fail with connection errors:

# Test MySQL protocol connectivity with netcat
nc -zv manticore.yourdomain.com 9306
# Connection to manticore.yourdomain.com 9306 port [tcp] succeeded!
  1. Add Monitor → TCP.
  2. Host: manticore.yourdomain.com
  3. Port: 9306
  4. Check interval: 60 seconds.
  5. Response timeout: 10 seconds.
  6. Label: ManticoreSearch MySQL Protocol
  7. Click Save.

This monitor catches the split state where ManticoreSearch's HTTP API is operational but the MySQL protocol interface has failed — a scenario that can occur after certain configuration changes or after a partial restart.

Step 5: Monitor Your SSL Certificate

ManticoreSearch is often reverse-proxied behind nginx for HTTPS termination when exposed to the internet. A certificate expiry on the proxy domain causes TLS errors for all clients:

  1. Add Monitor → SSL Certificate.
  2. Domain: manticore.yourdomain.com
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days.
  5. Label: ManticoreSearch SSL Certificate

For internal ManticoreSearch deployments accessed over plain HTTP within a private network, skip this step and focus on the TCP and HTTP monitors above.

Step 6: Index Health Heartbeat

Real-time indexes in ManticoreSearch can be silently cleared, corrupted by a crash during indexing, or left in an error state after a replication failure in distributed setups. The document count dropping to zero — or dramatically lower than expected — on a live index is not surfaced as a service error: ManticoreSearch continues to return HTTP 200 on status checks while returning empty search results to applications.

Set up a server-side heartbeat that pings Vigilmon only when your primary index contains an expected minimum document count:

#!/bin/bash
# /etc/cron.d/manticore-index-heartbeat — runs every 5 minutes

MANTICORE_API="http://localhost:9308/sql"
# Replace with your primary index name and expected minimum document count
INDEX_NAME="products"
MIN_DOCS=1000

COUNT=$(curl -fsS -X POST "$MANTICORE_API" \
  -d "query=SELECT count(*) as cnt FROM $INDEX_NAME" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d[0]['data'][0]['cnt'])" 2>/dev/null)

if [ -n "$COUNT" ] && [ "$COUNT" -ge "$MIN_DOCS" ]; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-HEARTBEAT-ID
fi

In Vigilmon:

  1. Add Monitor → Heartbeat.
  2. Expected interval: 5 minutes.
  3. Grace period: 15 minutes.
  4. Label: ManticoreSearch Index Document Count

To also check for replication lag in distributed setups, extend the script to run SHOW INDEXES STATUS and verify no index reports an error state:

INDEXES_STATUS=$(curl -fsS -X POST "$MANTICORE_API" \
  -d "query=SHOW INDEXES STATUS")
ERRORS=$(echo "$INDEXES_STATUS" | python3 -c "
import sys, json
data = json.load(sys.stdin)
rows = data[0].get('data', []) if data else []
errors = [r for r in rows if 'error' in str(r.get('Value', '')).lower()]
print(len(errors))
" 2>/dev/null)

if [ "${ERRORS:-1}" -eq 0 ] && [ -n "$COUNT" ] && [ "$COUNT" -ge "$MIN_DOCS" ]; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-HEARTBEAT-ID
fi

Common ManticoreSearch Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | ManticoreSearch process crash or segfault | REST API returns connection refused; alert fires within 60 s | | MySQL protocol listener crash (port 9306 down) | TCP probe on 9306 fails; HTTP API may still return 200 | | SQL engine internal error (HTTP layer up, queries failing) | /sql SHOW STATUS probe returns error; REST API root returns 200 | | Real-time index cleared or corrupted | Index document count heartbeat stops; alert fires after grace period | | Distributed index replication lag or error | SHOW INDEXES STATUS shows error; heartbeat stops | | nginx reverse proxy drops port 9308 routing | REST API and SQL monitors fail; server may still be running | | SSL certificate expires on proxied domain | SSL monitor alerts at 30 days before expiry | | File descriptor exhaustion under heavy indexing load | REST API stops accepting connections; monitor detects connection refused | | OOM kill during large batch indexing | Process crash detected by REST API connection refused |

ManticoreSearch's speed advantage comes with the same operational challenges as any self-hosted search engine: crashes under load, index corruption, and silent degradation where the server stays running but the data becomes unreliable. Vigilmon monitors every layer — the REST API, the SQL engine, the MySQL protocol port, SSL, and index document integrity — so you catch search failures before your users file bug reports about empty results.

Start monitoring ManticoreSearch in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles