tutorial

Monitoring Marqo with Vigilmon: Health API Checks, Index Monitoring, and Alerting for Neural Search

How to monitor Marqo neural search engine with Vigilmon — health endpoint checks, index availability, vector search query probes, and alerting for this AI-powered search platform.

Marqo is an open-source neural search engine that combines vector embeddings and keyword search — automatically generating embeddings from text and images at index and query time. It is built on top of OpenSearch under the hood and requires both the Marqo application layer and the underlying search engine to be healthy. When Marqo degrades, neural search queries fail, image search returns empty results, or indexing stalls silently. Vigilmon gives you external visibility into Marqo availability through its health and status APIs.

What You'll Build

  • A monitor on Marqo's /health endpoint for application health
  • An index-level availability check via the Marqo REST API
  • A neural search query probe that verifies end-to-end search functionality
  • A heartbeat monitor for document indexing pipelines
  • Alerting tuned for the latency characteristics of a neural search engine

Prerequisites

  • A running Marqo instance (Docker or Kubernetes) with HTTP API access
  • Marqo API accessible at http://marqo.example.com:8882 (default port)
  • A free account at vigilmon.online

Step 1: Verify the Marqo Health Endpoint

Marqo exposes a health check endpoint at /health:

curl -s http://marqo.example.com:8882/health

A healthy Marqo instance returns:

{
  "status": "ok",
  "backend": {
    "status": "ok",
    "memory_is_available": true,
    "number_of_models_loaded": 1
  }
}

Key fields to watch:

  • status: Top-level Marqo application status (ok or error)
  • backend.status: Underlying OpenSearch/vector store status
  • backend.memory_is_available: Whether sufficient memory exists to load models

If the health endpoint returns a non-200 status or "status":"error", Marqo is in a degraded or failed state.

Verify the endpoint before configuring monitors:

curl -i http://marqo.example.com:8882/health
# HTTP/1.1 200 OK
# Content-Type: application/json
# {"status":"ok","backend":{"status":"ok","memory_is_available":true,...}}

Docker deployment: Marqo's default Docker command includes a built-in health check. Check the container health with docker inspect marqo --format='{{.State.Health.Status}}'. Vigilmon complements this by testing connectivity from outside the host.

Step 2: Create a Vigilmon HTTP Monitor for Marqo

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://marqo.example.com:8882/health.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: "status":"ok".
  7. Click Save.

Two-Signal Monitoring

Add a second keyword monitor to detect when models fail to load — this is a common Marqo failure mode where the application appears up but cannot perform neural search:

  1. Add Monitor → HTTP.
  2. URL: http://marqo.example.com:8882/health.
  3. Check interval: 60 seconds.
  4. Expected status: 200.
  5. Keyword: "memory_is_available":true.
  6. Label: Marqo model memory available.

What This Catches

| Failure Mode | Docker Health Check | Vigilmon | |---|---|---| | Marqo process crash | ✓ | ✓ | | Port 8882 not accessible | ✗ | ✓ | | Backend OpenSearch down | ✗ | ✓ (backend status in health response) | | Insufficient memory for model loading | ✗ | ✓ (memory_is_available check) | | Network partition to Marqo | ✗ | ✓ |

Step 3: Monitor Index Availability

Marqo organizes documents into indices (also called index names). Use the /indexes endpoint to verify your critical indices exist:

curl -s http://marqo.example.com:8882/indexes

This returns a list of all indices:

{
  "results": [
    {"indexName": "products"},
    {"indexName": "product-images"},
    {"indexName": "knowledge-base"}
  ]
}

Build a lightweight wrapper to expose per-index health checks:

// marqo-health.js — index availability wrapper for Vigilmon
const express = require('express');
const axios = require('axios');

const app = express();
const MARQO_URL = process.env.MARQO_URL || 'http://localhost:8882';

// Node health check
app.get('/health/marqo', async (req, res) => {
  try {
    const { data } = await axios.get(`${MARQO_URL}/health`, { timeout: 10000 });
    if (data.status !== 'ok' || data.backend?.status !== 'ok') {
      return res.status(503).json({ status: 'degraded', detail: data });
    }
    return res.status(200).json({ status: 'ok' });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Per-index availability check
app.get('/health/marqo/index/:name', async (req, res) => {
  try {
    const { data } = await axios.get(`${MARQO_URL}/indexes`, { timeout: 10000 });
    const indices = data.results || [];
    const found = indices.some(idx => idx.indexName === req.params.name);

    if (!found) {
      return res.status(503).json({ status: 'missing', index: req.params.name });
    }

    return res.status(200).json({ status: 'ok', index: req.params.name });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Add index monitors in Vigilmon:

  1. Add Monitor → HTTP.
  2. URL: http://your-app.example.com:3001/health/marqo/index/products.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: "status":"ok".

Step 4: Neural Search Query Probe

For Marqo specifically, a process-level check is not enough — model loading, embedding generation, and vector retrieval can all fail independently. Add an end-to-end query probe that actually performs a neural search:

# marqo_query_probe.py — end-to-end neural search health check
import os, time
import requests
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
MARQO_URL = os.environ.get("MARQO_URL", "http://localhost:8882")
PROBE_INDEX = os.environ.get("MARQO_PROBE_INDEX", "products")

@app.get("/health/marqo/search")
def marqo_search_health():
    start = time.time()
    try:
        # Lightweight text search — uses the model for embedding generation
        resp = requests.post(
            f"{MARQO_URL}/indexes/{PROBE_INDEX}/search",
            json={
                "q": "health check probe",
                "limit": 1,
                "searchMethod": "TENSOR",
            },
            timeout=15,  # Neural search can take a few seconds
        )

        elapsed_ms = int((time.time() - start) * 1000)

        if resp.status_code == 200:
            return {"status": "ok", "latency_ms": elapsed_ms, "hits": len(resp.json().get("hits", []))}

        return JSONResponse(
            status_code=503,
            content={"status": "error", "http_status": resp.status_code, "body": resp.text[:200]},
        )

    except requests.Timeout:
        return JSONResponse(status_code=503, content={"status": "timeout", "latency_ms": 15000})
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Add this query probe as a Vigilmon monitor:

  1. Add Monitor → HTTP.
  2. URL: http://your-app.example.com:3001/health/marqo/search.
  3. Check interval: 2 minutes.
  4. Response timeout: 20 seconds (neural search includes model inference).
  5. Expected status: 200.
  6. Keyword: "status":"ok".
  7. Response time alert: 10000ms (10 seconds — slow neural search indicates model or resource pressure).

Important: Neural search probes are slower than typical HTTP health checks due to embedding model inference. Set your response timeout to at least 20 seconds, and set response time alerts at 10+ seconds rather than the 1-2 second thresholds appropriate for simple HTTP checks.

Step 5: Heartbeat Monitoring for Indexing Pipelines

Marqo's indexing pipeline generates embeddings at write time — this is more compute-intensive than traditional search indexing. A stalled indexing pipeline means new documents are not searchable.

Set Up the Heartbeat Monitor

  1. In Vigilmon → Add Monitor → Heartbeat.
  2. Name: marqo-indexer.
  3. Expected interval: Match your indexing cadence (5 minutes for batch jobs, 1 minute for real-time producers).
  4. Grace period: 2× the expected interval.
  5. Copy the heartbeat URL: https://vigilmon.online/heartbeat/your-unique-id.

Wire Into Your Indexing Code

Python indexing pipeline:

import marqo
import requests
import os

mq = marqo.Client(url=os.environ.get("MARQO_URL", "http://localhost:8882"))
VIGILMON_HB = os.environ.get("VIGILMON_HEARTBEAT_URL")

def index_documents(index_name, docs):
    """Index a batch of documents into Marqo and ping Vigilmon on success."""
    result = mq.index(index_name).add_documents(
        docs,
        tensor_fields=["title", "description"],  # fields for vector embedding
    )

    # Check for indexing errors
    errors = [item for item in result.get("items", []) if item.get("status") >= 400]
    if errors:
        raise RuntimeError(f"{len(errors)} documents failed to index")

    # Heartbeat — signals a successful indexing cycle
    if VIGILMON_HB:
        try:
            requests.get(VIGILMON_HB, timeout=5)
        except Exception:
            pass  # Don't let heartbeat failure break indexing

def run_indexing_loop():
    while True:
        try:
            batch = fetch_pending_documents(limit=50)
            if batch:
                index_documents("products", batch)
                print(f"Indexed {len(batch)} documents")
        except Exception as e:
            print(f"Indexing error: {e}")
            # Don't ping heartbeat on failure — Vigilmon will alert after grace period
        
        time.sleep(60)

if __name__ == "__main__":
    run_indexing_loop()

Node.js indexing script:

const { MqClient } = require('marqo');
const axios = require('axios');

const mq = new MqClient({ url: process.env.MARQO_URL || 'http://localhost:8882' });
const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;

async function indexBatch(docs) {
  const result = await mq.index('products').addDocuments(docs, {
    tensorFields: ['title', 'description'],
  });

  const errors = result.items?.filter(item => item.status >= 400) || [];
  if (errors.length > 0) {
    throw new Error(`${errors.length} documents failed to index`);
  }

  // Heartbeat after successful batch
  if (HEARTBEAT_URL) {
    await axios.get(HEARTBEAT_URL).catch(() => {});
  }
}

Step 6: SSL Certificate Monitoring

If Marqo is exposed over HTTPS via a reverse proxy, add SSL certificate monitoring:

  1. Add Monitor → SSL Certificate.
  2. Domain: marqo.example.com.
  3. Alert when expiry is within: 30 days.
  4. Escalation: 14 days, 7 days, 3 days.
  5. Click Save.

Marqo Cloud deployments handle TLS automatically. For self-hosted deployments behind Nginx or Traefik, monitor the proxy's certificate.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure tiered alerts for Marqo's components:

| Monitor | Trigger | Severity | Recommended Response | |---|---|---|---| | /health status check | status not ok | High | Check Marqo process; inspect Docker logs; verify OpenSearch backend | | /health memory check | memory_is_available false | High | Increase container memory; check model cache size | | Index availability | Index missing | High | Check if index was deleted; verify creation scripts | | Neural search probe | 503 or timeout | High | Check model loading; verify query pipeline | | Search response time | > 10 seconds | Medium | Scale compute; reduce model size; optimize index settings | | Indexing heartbeat | Ping not received | Medium | Check indexing pipeline; verify embedding model availability | | SSL certificate | < 30 days | Low | Renew certificate |

Neural search-specific thresholds:

  • Response timeout: 20 seconds (embedding inference adds significant latency vs. keyword search)
  • Response time alert: 10000ms (set higher than for keyword search engines)
  • Confirmation period: 2 consecutive failures (Marqo may be briefly slow during model loading)

Common Marqo Failure Modes

| Scenario | What Vigilmon Catches | |---|---| | Marqo process OOM kill (models use significant RAM) | /health endpoint fails immediately | | OpenSearch backend down | backend.status in health check shows error | | Model fails to load (GPU/CPU memory exhausted) | memory_is_available: false in health response | | Index accidentally deleted | Index availability monitor fires | | Embedding model inference timeout under load | Neural search probe times out; alert fires | | Feed pipeline stalled (no new documents searchable) | Heartbeat monitor fires | | Container restart clears in-memory model cache | Health check briefly returns initializing/degraded | | TLS certificate expiry on proxy | SSL monitor alerts at 30-day threshold |

Marqo's neural search adds significant compute between input and results — embedding models, vector retrieval, and re-ranking all run in the serving path. Standard HTTP process monitors cannot detect when any of these layers degrade. Vigilmon's layered monitoring approach — health endpoint, index availability, and end-to-end query probes — gives you visibility into the entire neural search stack from outside the system.

Get started free at vigilmon.online — your Marqo monitor is running in under 5 minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles