Monitoring Sphinx Search with Vigilmon: SphinxQL Health Checks, Index Status, and Alerting

Sphinx Search is a battle-tested full-text search engine used widely in e-commerce, content platforms, and classified ad sites. The searchd daemon serves queries over a MySQL-compatible protocol (SphinxQL) on port 9306 and, in Sphinx 3.x, over an HTTP API on port 9312. When searchd goes down or an index becomes unavailable, search queries fail or return empty results — often with generic MySQL errors that are hard to distinguish from a normal empty result set. Vigilmon gives you external visibility into Sphinx Search through HTTP API health checks and heartbeat monitoring for your indexing jobs.

What You'll Build

• An HTTP health check on Sphinx's HTTP API endpoint (Sphinx 3.x)
• A thin health wrapper that queries SHOW STATUS via SphinxQL for older versions
• Index-level availability monitoring
• A heartbeat monitor for indexer batch jobs and real-time indexing pipelines

Prerequisites

• A running Sphinx Search daemon (searchd) reachable at sphinx.example.com
• Sphinx 3.x recommended for HTTP API; Sphinx 2.x uses SphinxQL-only approach
• A free account at vigilmon.online

Step 1: Verify the Sphinx Search Daemon

Confirm searchd is running and accepting connections before configuring monitors.

Sphinx 3.x — HTTP API check:

curl -s http://sphinx.example.com:9312/

Sphinx 3.x returns a JSON response with version info when the HTTP listener is active.

SphinxQL check (all versions):

# Uses MySQL client — SphinxQL is MySQL protocol-compatible
mysql -h sphinx.example.com -P 9306 -e "SHOW STATUS"

A healthy Sphinx node returns a table with uptime, connections, maxed_out, and other metrics:

+---------------+---------+
| Counter       | Value   |
+---------------+---------+
| uptime        | 86400   |
| connections   | 42      |
| maxed_out     | 0       |
| command_search| 18234   |
+---------------+---------+

If searchd is not running, the mysql command returns ERROR 2003 (HY000): Can't connect to MySQL server.

Ensure the HTTP listener is enabled in sphinx.conf (Sphinx 3.x):

searchd {
    listen = 9306:mysql41
    listen = 9312:http
    # ... other config ...
}

Restart after config changes:

sudo systemctl restart searchd

Step 2: Build a Dedicated Health Endpoint

Option A: Use the Sphinx HTTP API Directly (Sphinx 3.x)

Sphinx 3.x exposes / and /sql over HTTP. Use Vigilmon's HTTP monitor to probe it directly:

1. Add Monitor → HTTP.
2. URL: http://sphinx.example.com:9312/.
3. Expected status: 200.
4. Check interval: 60 seconds.
5. Click Save.

For a more specific check, probe the SphinxQL endpoint over HTTP:

curl -s -X POST http://sphinx.example.com:9312/sql \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "query=SHOW STATUS&mode=raw"

Option B: Build a SphinxQL Health Wrapper (All Versions)

For Sphinx 2.x or when you want richer health data, run a thin HTTP wrapper that queries SphinxQL:

// sphinx-health.js — HTTP health wrapper for Sphinx Search
const express = require('express');
const mysql = require('mysql2/promise');

const app = express();
const SPHINX_HOST = process.env.SPHINX_HOST || 'localhost';
const SPHINX_PORT = parseInt(process.env.SPHINX_PORT || '9306', 10);

app.get('/health/sphinx', async (req, res) => {
  let connection;
  try {
    connection = await mysql.createConnection({
      host: SPHINX_HOST,
      port: SPHINX_PORT,
      connectTimeout: 5000,
    });

    const [rows] = await connection.query('SHOW STATUS');
    const status = {};
    for (const row of rows) {
      status[row.Counter] = row.Value;
    }

    const maxedOut = parseInt(status['maxed_out'] || '0', 10);
    if (maxedOut > 0) {
      return res.status(503).json({ status: 'degraded', maxed_out: maxedOut });
    }

    return res.status(200).json({
      status: 'ok',
      uptime: status['uptime'],
      connections: status['connections'],
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  } finally {
    if (connection) await connection.end();
  }
});

app.listen(3004, () => console.log('Sphinx health wrapper on :3004'));

Python Example

# sphinx_health.py — Sphinx Search health wrapper using PyMySQL
import os
import pymysql
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

SPHINX_HOST = os.environ.get("SPHINX_HOST", "localhost")
SPHINX_PORT = int(os.environ.get("SPHINX_PORT", "9306"))


@app.get("/health/sphinx")
def sphinx_health():
    try:
        conn = pymysql.connect(
            host=SPHINX_HOST,
            port=SPHINX_PORT,
            connect_timeout=5,
        )
        with conn.cursor() as cur:
            cur.execute("SHOW STATUS")
            rows = {row[0]: row[1] for row in cur.fetchall()}
        conn.close()

        maxed_out = int(rows.get("maxed_out", 0))
        if maxed_out > 0:
            return JSONResponse(status_code=503, content={"status": "degraded", "maxed_out": maxed_out})

        return {
            "status": "ok",
            "uptime": rows.get("uptime"),
            "connections": rows.get("connections"),
        }
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Deploy and verify:

curl -i http://your-app.example.com:3004/health/sphinx
# HTTP/1.1 200 OK
# {"status":"ok","uptime":"86400","connections":"42"}

Step 3: Create a Vigilmon HTTP Monitor for Sphinx

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: http://your-app.example.com:3004/health/sphinx
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: "status":"ok".
7. Click Save.

What This Catches

| Failure Mode | Process Monitor | Vigilmon | |---|---|---| | searchd process crash | ✓ | ✓ | | SphinxQL port not responding | ✗ | ✓ | | Connection limit exhausted (maxed_out > 0) | ✗ | ✓ | | Network partition from application servers | ✗ | ✓ | | searchd hanging on a query | ✗ | ✓ |

Step 4: Monitor Individual Index Availability

Check that specific indices (Sphinx calls them "indexes" or "tables") are present and queryable:

# Check if an index is accessible over SphinxQL
mysql -h sphinx.example.com -P 9306 -e "SHOW TABLES LIKE 'products'"

Extend your health wrapper to support index-level checks:

app.get('/health/sphinx/index/:name', async (req, res) => {
  let connection;
  try {
    connection = await mysql.createConnection({
      host: SPHINX_HOST,
      port: SPHINX_PORT,
      connectTimeout: 5000,
    });

    const [rows] = await connection.execute('SHOW TABLES LIKE ?', [req.params.name]);
    if (rows.length === 0) {
      return res.status(503).json({ status: 'missing', index: req.params.name });
    }

    // Run a test query to confirm the index is searchable
    const [results] = await connection.execute(
      `SELECT COUNT(*) AS total FROM \`${req.params.name}\` LIMIT 1`
    );

    return res.status(200).json({
      status: 'ok',
      index: req.params.name,
      doc_count: results[0]?.total ?? 0,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  } finally {
    if (connection) await connection.end();
  }
});

Add a Vigilmon monitor per critical index:

1. URL: http://your-app.example.com:3004/health/sphinx/index/products
2. Keyword: "status":"ok".
3. Check interval: 2 minutes.

Step 5: Heartbeat Monitoring for indexer Jobs

Sphinx Search uses a separate indexer binary that rebuilds or updates disk-based indices from a data source (MySQL, PostgreSQL, XML pipe). If the indexer job fails silently, your index goes stale while the searchd daemon keeps serving old results.

Vigilmon heartbeat monitors alert you when the expected ping from your indexer cron job stops arriving.

Set Up the Heartbeat Monitor

1. In Vigilmon → Add Monitor → Heartbeat.
2. Name: sphinx-indexer.
3. Expected interval: Match your indexer cron schedule (e.g., 30 minutes for incremental reindex).
4. Grace period: 15 minutes.
5. Copy the unique heartbeat URL: https://vigilmon.online/heartbeat/your-unique-id.

Wire Into Your Indexer Cron Job

#!/bin/bash
# /etc/cron.d/sphinx-reindex — incremental reindex every 30 minutes
# * * * * * root /opt/sphinx/scripts/reindex.sh

set -e

VIGILMON_HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL:-}"

# Run incremental reindex
/usr/bin/indexer --config /etc/sphinxsearch/sphinx.conf --rotate products_delta

# Merge delta into main index (optional pattern)
# /usr/bin/indexer --config /etc/sphinxsearch/sphinx.conf --merge products products_delta --rotate

# Ping heartbeat only on success (set -e ensures failures skip this line)
if [ -n "$VIGILMON_HEARTBEAT_URL" ]; then
  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null 2>&1 || true
fi

For real-time indexes (rt index type in Sphinx 2.x+), which are updated via SphinxQL INSERT/REPLACE:

import os
import pymysql
import requests

SPHINX_HOST = os.environ.get("SPHINX_HOST", "localhost")
SPHINX_PORT = int(os.environ.get("SPHINX_PORT", "9306"))
VIGILMON_HEARTBEAT = os.environ["VIGILMON_HEARTBEAT_URL"]


def push_to_rt_index(documents):
    conn = pymysql.connect(host=SPHINX_HOST, port=SPHINX_PORT)
    with conn.cursor() as cur:
        for doc in documents:
            cur.execute(
                "REPLACE INTO products_rt (id, title, body) VALUES (%s, %s, %s)",
                (doc["id"], doc["title"], doc["body"]),
            )
    conn.commit()
    conn.close()


def run_rt_ingest():
    documents = fetch_new_documents()  # your fetch logic
    push_to_rt_index(documents)

    # Ping Vigilmon only on success
    try:
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
    except Exception:
        pass

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, set up your alert routing:

| Monitor | Trigger | Recommended Action | |---|---|---| | Sphinx daemon health | 503 or keyword absent | Check searchd process; review Sphinx error log at /var/log/sphinxsearch/searchd.log | | Index availability | Index missing or doc count zero | Check if indexer dropped the table; verify data source connectivity | | indexer heartbeat | Ping not received | Check cron job logs; verify data source (MySQL/PGSQL) is reachable | | maxed_out alert | Connections saturated | Increase max_connections in sphinx.conf; scale out |

Recommended thresholds:

• Confirmation period: 2 consecutive failures before alerting (avoids false positives from brief searchd rotations during --rotate)
• Response time alert: 2000ms (SphinxQL SHOW STATUS is near-instant; slow response indicates connection saturation or I/O bottleneck)
• Recovery notification: Enable "alert on recovery" so your team knows when searchd comes back after a restart or --rotate

Common Sphinx Search Failure Modes

| Scenario | What Vigilmon Catches | |---|---| | searchd process crash | SphinxQL health check fails immediately | | Port 9306 not bound (config error) | Connection refused on monitor | | Connection limit hit (maxed_out > 0) | Health wrapper returns 503 with maxed_out count | | indexer job fails or isn't scheduled | Heartbeat monitor fires | | Index accidentally dropped or corrupted | Index availability check returns missing | | Real-time index stopped receiving inserts | Heartbeat monitor fires; doc count stagnates | | OOM kill under heavy query load | Process check fails; Vigilmon alert fires immediately | | Network issue between app and searchd | External probe catches what internal connection pools miss |

Sphinx Search is mature and stable but fails quietly when searchd crashes, a cron-driven indexer job silently errors, or the connection pool saturates. Vigilmon's external HTTP probes and heartbeat monitors give you the outside-in view that searchd's own query log and process monitors can't provide.

Get started free at vigilmon.online — your Sphinx monitor is running in under 5 minutes.