tutorial

Monitoring Sonic Search with Vigilmon: TCP Health Checks, Channel Status, and Alerting

How to monitor Sonic fast search backend with Vigilmon — TCP port monitoring, HTTP health wrapper, heartbeat monitoring for indexing pipelines, and alerting.

Sonic is a fast, lightweight, and schema-less search backend written in Rust. It exposes a custom TCP protocol on port 1491 and is widely used as a drop-in backend for applications that need low-latency full-text search without the operational overhead of Elasticsearch or Solr. Because Sonic uses a custom TCP wire protocol rather than HTTP, standard uptime monitors can't probe it directly — Vigilmon fills this gap with TCP port monitors and a lightweight HTTP wrapper that turns Sonic's protocol into a monitorable health endpoint.

What You'll Build

  • A TCP port monitor to detect when Sonic's listener goes down
  • An HTTP health wrapper that connects to Sonic's protocol and exposes a /health endpoint
  • A Vigilmon HTTP monitor against the health wrapper
  • A heartbeat monitor for indexing pipelines that push data into Sonic

Prerequisites

  • A running Sonic instance reachable at sonic.example.com:1491 (default port)
  • A free account at vigilmon.online

Step 1: Verify the Sonic TCP Listener

Sonic speaks its own line-based protocol over TCP. Confirm the daemon is reachable:

# Quick TCP connectivity check
nc -zv sonic.example.com 1491

For a protocol-level check, connect and send the START handshake:

# Manually probe the Sonic protocol
(echo "START search SecretPassword"; sleep 1; echo "QUIT") | nc sonic.example.com 1491

A healthy Sonic node replies with a banner and then ENDED quit:

CONNECTED <sonic-server v1.4.9>
STARTED search
ENDED quit

If nc can't connect, confirm the Sonic process is running and the listen address in config.cfg matches the expected interface:

[server]
listen = "0.0.0.0:1491"

Step 2: Build a Dedicated Health Endpoint

Sonic's TCP protocol is not directly probeable by HTTP-based monitors. Build a thin HTTP wrapper that connects to Sonic on each request and translates the result into an HTTP status.

Node.js Example

// sonic-health.js — HTTP wrapper for Sonic Search health
const express = require('express');
const net = require('net');

const app = express();
const SONIC_HOST = process.env.SONIC_HOST || 'localhost';
const SONIC_PORT = parseInt(process.env.SONIC_PORT || '1491', 10);
const SONIC_PASSWORD = process.env.SONIC_PASSWORD || 'SecretPassword';

app.get('/health/sonic', (req, res) => {
  const client = new net.Socket();
  let buffer = '';
  let responded = false;

  const finish = (ok, detail) => {
    if (responded) return;
    responded = true;
    client.destroy();
    if (ok) {
      res.status(200).json({ status: 'ok', detail });
    } else {
      res.status(503).json({ status: 'down', detail });
    }
  };

  client.setTimeout(5000);
  client.connect(SONIC_PORT, SONIC_HOST, () => {
    // Wait for CONNECTED banner, then send START
  });

  client.on('data', (data) => {
    buffer += data.toString();
    if (buffer.includes('CONNECTED') && !buffer.includes('STARTED')) {
      client.write(`START search ${SONIC_PASSWORD}\r\n`);
    }
    if (buffer.includes('STARTED search')) {
      finish(true, buffer.split('\n')[0].trim());
    }
    if (buffer.includes('ENDED') || buffer.includes('ERR')) {
      finish(false, buffer.trim());
    }
  });

  client.on('timeout', () => finish(false, 'connection timed out'));
  client.on('error', (err) => finish(false, err.message));
});

app.listen(3002, () => console.log('Sonic health wrapper on :3002'));

Python Example

# sonic_health.py — Sonic Search health wrapper
import os
import socket
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

SONIC_HOST = os.environ.get("SONIC_HOST", "localhost")
SONIC_PORT = int(os.environ.get("SONIC_PORT", "1491"))
SONIC_PASSWORD = os.environ.get("SONIC_PASSWORD", "SecretPassword")


@app.get("/health/sonic")
def sonic_health():
    try:
        with socket.create_connection((SONIC_HOST, SONIC_PORT), timeout=5) as s:
            banner = s.recv(512).decode()
            if "CONNECTED" not in banner:
                return JSONResponse(status_code=503, content={"status": "down", "detail": "unexpected banner"})

            s.sendall(f"START search {SONIC_PASSWORD}\r\n".encode())
            reply = s.recv(512).decode()

            if "STARTED" not in reply:
                return JSONResponse(status_code=503, content={"status": "degraded", "detail": reply.strip()})

            s.sendall(b"QUIT\r\n")
            return {"status": "ok", "banner": banner.strip()}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Deploy the wrapper and verify:

curl -i http://your-app.example.com:3002/health/sonic
# HTTP/1.1 200 OK
# {"status":"ok","banner":"CONNECTED <sonic-server v1.4.9>"}

Step 3: Create a Vigilmon HTTP Monitor for Sonic

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://your-app.example.com:3002/health/sonic
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "status":"ok".
  7. Click Save.

For bare TCP port availability (no protocol check), add a second monitor:

  1. Add Monitor → TCP Port.
  2. Host: sonic.example.com.
  3. Port: 1491.
  4. Check interval: 60 seconds.
  5. Click Save.

What This Catches

| Failure Mode | Process Monitor | Vigilmon | |---|---|---| | Sonic process crash | ✓ | ✓ | | TCP listener not bound (config error) | ✗ | ✓ | | Password/auth misconfiguration | ✗ | ✓ | | Network partition from application servers | ✗ | ✓ | | Sonic responding but refusing search channel | ✗ | ✓ |

Step 4: Monitor Search Channel Availability

Sonic exposes three channels: search, ingest, and control. If the search channel specifically is degraded, queries return no results even though the process is running. Extend your health wrapper to check all three channels:

// Check all three Sonic channels
app.get('/health/sonic/channels', async (req, res) => {
  const channels = ['search', 'ingest', 'control'];
  const results = {};

  for (const channel of channels) {
    results[channel] = await checkSonicChannel(channel);
  }

  const allOk = Object.values(results).every((r) => r === 'ok');
  res.status(allOk ? 200 : 503).json({ status: allOk ? 'ok' : 'degraded', channels: results });
});

async function checkSonicChannel(channel) {
  return new Promise((resolve) => {
    const client = new net.Socket();
    let buffer = '';
    client.setTimeout(5000);
    client.connect(SONIC_PORT, SONIC_HOST, () => {});
    client.on('data', (data) => {
      buffer += data.toString();
      if (buffer.includes('CONNECTED') && !buffer.includes('STARTED')) {
        client.write(`START ${channel} ${SONIC_PASSWORD}\r\n`);
      }
      if (buffer.includes(`STARTED ${channel}`)) {
        client.destroy();
        resolve('ok');
      }
      if (buffer.includes('ERR')) {
        client.destroy();
        resolve('error');
      }
    });
    client.on('timeout', () => { client.destroy(); resolve('timeout'); });
    client.on('error', () => { resolve('unreachable'); });
  });
}

Add a Vigilmon monitor for the channel check endpoint with keyword "status":"ok".

Step 5: Heartbeat Monitoring for Indexing Pipelines

Sonic is fed by ingest pipelines that push documents using the PUSH command over the ingest channel. If the pipeline stalls, the search index goes stale — no error is visible to search users.

Vigilmon heartbeat monitors alert you when an expected ping stops arriving from your ingest job.

Set Up the Heartbeat Monitor

  1. In Vigilmon → Add Monitor → Heartbeat.
  2. Name: sonic-ingest-pipeline.
  3. Expected interval: Match your ingest job cadence (e.g., 5 minutes for batch ingest).
  4. Grace period: 10 minutes.
  5. Copy the unique heartbeat URL: https://vigilmon.online/heartbeat/your-unique-id.

Wire Into Your Ingest Script

Python ingest pipeline:

import os
import socket
import requests

SONIC_HOST = os.environ.get("SONIC_HOST", "localhost")
SONIC_PORT = int(os.environ.get("SONIC_PORT", "1491"))
SONIC_PASSWORD = os.environ.get("SONIC_PASSWORD", "SecretPassword")
VIGILMON_HEARTBEAT = os.environ["VIGILMON_HEARTBEAT_URL"]


def push_to_sonic(collection, bucket, object_id, text):
    with socket.create_connection((SONIC_HOST, SONIC_PORT), timeout=10) as s:
        s.recv(512)  # CONNECTED banner
        s.sendall(f"START ingest {SONIC_PASSWORD}\r\n".encode())
        s.recv(512)  # STARTED
        s.sendall(f'PUSH {collection} {bucket} {object_id} "{text}"\r\n'.encode())
        s.recv(512)  # OK
        s.sendall(b"QUIT\r\n")


def run_ingest():
    documents = fetch_new_documents()  # your fetch logic
    for doc in documents:
        push_to_sonic("articles", "default", doc["id"], doc["text"])

    # Signal success to Vigilmon only after all documents indexed
    try:
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
    except Exception:
        pass  # Don't let heartbeat failure break ingest


if __name__ == "__main__":
    import time
    while True:
        run_ingest()
        time.sleep(300)

Shell cron job:

#!/bin/bash
# /etc/cron.d/sonic-ingest — runs every 5 minutes

python3 /opt/ingest/sonic_push.py && \
  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null 2>&1

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, set up your alert routing:

| Monitor | Trigger | Recommended Action | |---|---|---| | Sonic TCP port | Port unreachable | Check if sonic process is running; inspect system logs | | Sonic HTTP health | 503 or keyword absent | Verify TCP connectivity; check Sonic config and password | | Channel availability | Any channel degraded | Restart Sonic; check for corrupted data files | | Ingest heartbeat | Ping not received | Check ingest pipeline logs; verify queue is draining |

Recommended thresholds:

  • Confirmation period: 2 consecutive failures before alerting (avoids false positives from brief Sonic restarts or GC pauses)
  • Response time alert: 1000ms (Sonic health checks are very fast in normal operation; slow response indicates resource pressure)
  • Recovery notification: Enable "alert on recovery" so your team knows when the monitor goes green

Common Sonic Search Failure Modes

| Scenario | What Vigilmon Catches | |---|---| | Sonic process crash | TCP port monitor and HTTP health check both fail | | Port 1491 not bound (config error) | TCP monitor connection refused | | Wrong password in config | HTTP health wrapper returns 503 with auth error | | Ingest pipeline stalled | Heartbeat monitor fires | | Disk full (Sonic data dir) | Process may keep running but ingest fails; heartbeat catches | | Network issue between app and Sonic | External probe catches what internal health checks miss |

Sonic fails quietly when the TCP listener drops or the ingest pipeline stalls — there are no obvious HTTP error pages or easy-to-parse status endpoints. Vigilmon's TCP port monitors, protocol-aware HTTP wrapper, and heartbeat monitors give you the external visibility Sonic doesn't provide out of the box.

Get started free at vigilmon.online — your Sonic monitor is running in under 5 minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles