Monitoring ZincSearch with Vigilmon

ZincSearch (formerly Zinc Search) is a full-text search engine built in Go that provides an Elasticsearch-compatible API at a fraction of the resource cost. It runs as a single binary on port 4080 and stores index data locally. That simplicity makes it fast to deploy — but it also means there are no redundant nodes to absorb a crash. Vigilmon monitors ZincSearch's API endpoints, indexing pipeline, and storage health so you know the moment your search layer degrades.

What You'll Set Up

• Web server availability on port 4080
• Indexing API health check
• Search query endpoint response time monitoring
• Admin dashboard availability
• Storage backend connectivity via a heartbeat
• Data ingestion pipeline health

Prerequisites

• ZincSearch 0.4+ running on port 4080
• Admin credentials (default: admin / set during first run)
• A free Vigilmon account

Step 1: Monitor Web Server Availability

ZincSearch's built-in Go HTTP server listens on port 4080. The /healthz endpoint returns a 200 OK when the server is running and accepting connections:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter: http://your-zincsearch-host:4080/healthz
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Set Expected body contains: "OK".
7. Click Save.

The /healthz endpoint is lightweight — it returns immediately without hitting storage and is safe to poll at 1-minute intervals.

Step 2: Monitor the Indexing API

ZincSearch accepts document indexing requests at the Elasticsearch-compatible bulk index endpoint. A failing indexer silently drops data that applications think was indexed.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-zincsearch-host:4080/api/vigilmon_probe/_doc
3. Method: POST
4. Request headers:
   • Content-Type: application/json
   • Authorization: Basic YOUR_BASE64_CREDENTIALS
5. Request body:

{"probe": "vigilmon", "ts": "2026-01-01T00:00:00Z"}

6. Expected HTTP status: 200
7. Check interval: 2 minutes
8. Click Save.

This creates a probe document in a vigilmon_probe index (create it first via the admin UI or API). A 500 or timeout means the indexing path is broken. You can clean up probe documents periodically by deleting the index.

Step 3: Monitor Search Query Response Times

ZincSearch's search endpoint must return results within an acceptable latency. Slow queries often indicate that the index has grown large without compaction, or that the storage backend is under I/O pressure.

1. Add an HTTP / HTTPS monitor.
2. URL: http://your-zincsearch-host:4080/api/vigilmon_probe/_search
3. Method: POST
4. Request headers:
   • Content-Type: application/json
   • Authorization: Basic YOUR_BASE64_CREDENTIALS
5. Request body:

{
  "search_type": "matchall",
  "query": {"term": "probe"},
  "from": 0,
  "max_results": 1
}

6. Expected HTTP status: 200
7. Under Advanced, enable Response time alert and set the threshold to 2000 ms.
8. Check interval: 1 minute
9. Click Save.

A response time consistently above 2 seconds on a small probe index signals storage I/O problems or compaction falling behind.

Step 4: Monitor the Admin Dashboard

ZincSearch ships with a built-in admin UI at the root path (/). This is the primary operational interface for managing indexes and users.

1. Add an HTTP / HTTPS monitor.
2. URL: http://your-zincsearch-host:4080/
3. Expected HTTP status: 200
4. Expected body contains: "ZincSearch"
5. Check interval: 5 minutes
6. Click Save.

The admin dashboard failing independently of the API is unlikely (they share the same process), but this check provides a user-facing availability signal distinct from the API-level probes.

Step 5: Monitor the Index List API

The index list endpoint confirms ZincSearch can enumerate its stored indexes. A failure here can indicate that the data directory is corrupt or the index manifest file is unreadable:

1. Add an HTTP / HTTPS monitor.
2. URL: http://your-zincsearch-host:4080/api/index
3. Method: GET
4. Request header: Authorization: Basic YOUR_BASE64_CREDENTIALS
5. Expected HTTP status: 200
6. Expected body contains: "list"
7. Check interval: 2 minutes
8. Click Save.

A 500 from this endpoint while /healthz returns 200 often means a corrupted index manifest, which requires restoring from backup.

Step 6: Storage Backend Health via Heartbeat

ZincSearch stores all index data on the local filesystem (or S3 in cloud deployments). If the data directory fills up or an S3 bucket becomes unreachable, indexing silently fails.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Expected interval: 5 minutes.
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).
4. Create a script on the ZincSearch host:

#!/bin/bash
# Checks ZincSearch data directory health
ZINC_DATA_PATH="${ZINC_DATA_PATH:-/var/lib/zincsearch}"

# Check disk usage
USAGE=$(df "$ZINC_DATA_PATH" | awk 'NR==2 {print $5}' | tr -d '%')

# Check directory is writable
if [ "$USAGE" -lt 85 ] && touch "$ZINC_DATA_PATH/.probe" 2>/dev/null; then
  rm -f "$ZINC_DATA_PATH/.probe"
  curl -s https://vigilmon.online/heartbeat/abc123
fi

5. Add to cron: */5 * * * * /opt/scripts/zinc_storage_check.sh

The heartbeat stops when disk usage exceeds 85% or the data directory becomes read-only, triggering a Vigilmon alert.

Step 7: Data Ingestion Pipeline Monitoring

If your application pushes documents to ZincSearch via a queue or an ETL pipeline, the ingestion pipeline can stall independently of ZincSearch itself. Use a heartbeat from your ingestion process:

1. Add another Cron Heartbeat monitor with your pipeline's expected run interval.
2. In your ingestion script, ping the heartbeat after each successful batch:

import requests

def index_batch(documents):
    # Index documents to ZincSearch
    response = requests.post(
        'http://your-zincsearch-host:4080/api/_bulk',
        json=build_bulk_payload(documents),
        auth=('admin', 'YOUR_PASSWORD')
    )
    response.raise_for_status()

    # Signal successful ingestion to Vigilmon
    requests.get('https://vigilmon.online/heartbeat/def456', timeout=5)

If the ingestion script crashes or hangs, the heartbeat stops and Vigilmon alerts independently of whether ZincSearch itself is healthy.

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. For the /healthz monitor, set Consecutive failures before alert to 1 — ZincSearch is a single process with no automatic restart.
3. For the indexing API monitor, set Consecutive failures before alert to 2 — a single timeout can occur during compaction.
4. For the storage heartbeat, set Consecutive failures before alert to 1 — a missed heartbeat means storage is unhealthy or the monitoring script itself crashed.
5. For the search latency alert, set the threshold at 2000 ms but only alert after 3 consecutive slow responses to avoid noise during brief I/O spikes.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web server | :4080/healthz | Process crash, port not listening | | Indexing API | :4080/api/INDEX/_doc | Write path failure | | Search query | :4080/api/INDEX/_search | Read path failure, slow queries | | Admin dashboard | :4080/ | UI unavailability | | Index list API | :4080/api/index | Index manifest corruption | | Storage heartbeat | Heartbeat URL | Disk full, data dir read-only | | Ingestion pipeline | Heartbeat URL | ETL/pipeline stall |

ZincSearch's single-binary design means zero cluster overhead — but also zero redundancy. With Vigilmon watching the health endpoint, indexing path, search latency, and storage health, you get full visibility into every failure mode so your application's search feature never goes dark without warning.