tutorial

Monitoring Vespa with Vigilmon: Health API, Application Status, and Alerting for the Search & Recommendation Engine

How to monitor Vespa with Vigilmon — state API health checks, application deployment status, node availability monitoring, and alerting for this production-grade search and recommendation engine.

Vespa is Yahoo's open-source search, recommendation, and personalization engine — used in production by companies running large-scale retrieval and ranking at low latency. It operates as a distributed system with multiple node types: config servers, admin nodes, content nodes, and container nodes. When any tier degrades, queries silently receive degraded results or time out. Vigilmon gives you external visibility into your Vespa cluster through its state and status REST APIs.

What You'll Build

  • A monitor on Vespa's /state/v1/health endpoint for container node health
  • An application deployment status check via the config server API
  • Content node availability monitoring
  • A heartbeat monitor for feed (indexing) pipelines
  • Alerting for the different Vespa node tiers

Prerequisites

  • A running Vespa cluster (single-node or multi-node deployment) with HTTP access
  • Vespa container node accessible at http://vespa.example.com:8080 (default query port)
  • Vespa config server accessible at http://config.vespa.example.com:19071 (default admin port)
  • A free account at vigilmon.online

Step 1: Verify the Vespa State API

Every Vespa node exposes a /state/v1/health endpoint that reports its operational status:

# Container node health
curl -s http://vespa.example.com:8080/state/v1/health

A healthy container node returns:

{
  "status": {
    "code": "up",
    "message": ""
  },
  "metrics": {
    "snapshot": {
      "from": 1234567890,
      "to": 1234567950
    }
  }
}

The status.code field is the key signal:

  • up: Node is fully operational
  • initializing: Node is starting up or loading indices
  • down: Node is unavailable

Check the config server separately:

# Config server health
curl -s http://config.vespa.example.com:19071/state/v1/health

And content nodes on port 19107:

# Content node health
curl -s http://content.vespa.example.com:19107/state/v1/health

Vespa Cloud: If you use Vespa Cloud, the managed service handles node health internally. Focus on monitoring your application's container endpoints, not the underlying infrastructure.

Step 2: Create a Vigilmon HTTP Monitor for Vespa Container Nodes

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://vespa.example.com:8080/state/v1/health.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: "code":"up".
  7. Click Save.

For a multi-node deployment, add a separate monitor for each container node:

  • [vespa-container-1] /state/v1/health → http://container1.vespa.example.com:8080/state/v1/health
  • [vespa-container-2] /state/v1/health → http://container2.vespa.example.com:8080/state/v1/health

Use Vigilmon's label feature to distinguish nodes in alert notifications.

What This Catches

| Failure Mode | Kubernetes/systemd | Vigilmon | |---|---|---| | Container node process crash | ✓ | ✓ | | Node stuck in initializing state | ✗ | ✓ | | Port 8080 not reachable from clients | ✗ | ✓ | | Network partition to container tier | ✗ | ✓ | | Container node returning degraded status | ✗ | ✓ |

Step 3: Monitor Application Deployment Status

Vespa's config server exposes the application deployment status via its /application/v2/ REST API. An application that failed to deploy serves outdated models and schemas:

curl -s http://config.vespa.example.com:19071/application/v2/tenant/default/application/default/environment/prod/region/default/instance/default

A successfully deployed application returns HTTP 200 with application metadata. A failed or missing deployment returns 404.

Add a config server application monitor:

  1. Add Monitor → HTTP.
  2. URL: http://config.vespa.example.com:19071/application/v2/tenant/default/application/default.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Label: Vespa application deployment.

Note: The config server port (19071) is typically not exposed publicly. If your config server is internal, deploy a lightweight proxy or health wrapper that exposes a GET endpoint accessible to Vigilmon.

Step 4: Monitor Search Query Availability End-to-End

The highest-value monitor is one that verifies your application's query endpoint actually serves results. Vespa's YQL query API lets you run a lightweight test query:

curl "http://vespa.example.com:8080/search/?yql=select%20%2A%20from%20sources%20%2A%20where%20true%20limit%201"

This returns at minimum an empty result set if the index is empty, or a result if documents exist. The key is that the HTTP status is 200 and the response is valid JSON.

Build a health endpoint that wraps this probe:

// vespa-health.js — query availability probe
const express = require('express');
const axios = require('axios');

const app = express();
const VESPA_URL = process.env.VESPA_URL || 'http://localhost:8080';

app.get('/health/vespa', async (req, res) => {
  try {
    // Lightweight health check via state API
    const state = await axios.get(`${VESPA_URL}/state/v1/health`, { timeout: 5000 });
    const code = state.data?.status?.code;

    if (code !== 'up') {
      return res.status(503).json({ status: code || 'unknown', source: 'state_api' });
    }

    // Query probe — verify the search container accepts queries
    const query = await axios.get(
      `${VESPA_URL}/search/?yql=select%20*%20from%20sources%20*%20where%20true%20limit%201`,
      { timeout: 8000 }
    );

    if (query.status !== 200) {
      return res.status(503).json({ status: 'query_failed', code: query.status });
    }

    return res.status(200).json({ status: 'ok' });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Add this to Vigilmon:

  1. Add Monitor → HTTP.
  2. URL: http://your-app.example.com:3001/health/vespa.
  3. Check interval: 60 seconds.
  4. Expected status: 200.
  5. Keyword: "status":"ok".

Step 5: Monitor Content Node Availability

Content nodes store the actual document data and handle partial query execution. In Vespa clusters, content node failures reduce coverage — Vespa continues serving queries but with partial results (degraded coverage), which may not surface as an HTTP error.

Check content node state API endpoints:

curl -s http://content1.vespa.example.com:19107/state/v1/health

Add a monitor per content node:

  1. Add Monitor → HTTP.
  2. URL: http://content1.vespa.example.com:19107/state/v1/health.
  3. Check interval: 60 seconds.
  4. Expected status: 200.
  5. Keyword: "code":"up".
  6. Label: [vespa-content-1] node health.

For large clusters, monitor at least one content node per redundancy group to ensure full coverage monitoring.

Step 6: Heartbeat Monitoring for Document Feed Pipelines

Vespa's feeding (indexing) layer is separate from the query layer. A stalled feed pipeline means documents go unindexed, and search results become stale. Vigilmon heartbeat monitors detect stalled feed pipelines:

Set Up the Heartbeat Monitor

  1. In Vigilmon → Add Monitor → Heartbeat.
  2. Name: vespa-document-feed.
  3. Expected interval: Match your feed cadence (1 minute for continuous feeds, 15 minutes for batch).
  4. Grace period: 2× the expected interval.
  5. Copy the heartbeat URL: https://vigilmon.online/heartbeat/your-unique-id.

Wire Into Your Feed Pipeline

Java Vespa HTTP Client:

// VespaFeedPipeline.java — heartbeat after successful feed batch
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;

public class VespaFeedPipeline {
    private static final String HEARTBEAT_URL = System.getenv("VIGILMON_HEARTBEAT_URL");

    public void feedBatch(List<Document> docs) throws Exception {
        // ... feed documents to Vespa ...
        for (Document doc : docs) {
            vespaClient.feed(doc);
        }

        // Ping Vigilmon after successful batch
        if (HEARTBEAT_URL != null) {
            HttpClient.newHttpClient().send(
                HttpRequest.newBuilder().uri(URI.create(HEARTBEAT_URL)).GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        }
    }
}

Python feed script:

import requests, os, json, time

VESPA_URL = os.environ.get("VESPA_URL", "http://localhost:8080")
VIGILMON_HB = os.environ.get("VIGILMON_HEARTBEAT_URL")

def feed_document(doc_id, doc_fields):
    resp = requests.post(
        f"{VESPA_URL}/document/v1/my-namespace/my-doctype/docid/{doc_id}",
        json={"fields": doc_fields},
        timeout=10,
    )
    resp.raise_for_status()

def run_feed_cycle():
    docs = fetch_pending_documents()
    for doc_id, doc_fields in docs.items():
        feed_document(doc_id, doc_fields)

    # Heartbeat signals successful cycle to Vigilmon
    if VIGILMON_HB:
        requests.get(VIGILMON_HB, timeout=5)

while True:
    run_feed_cycle()
    time.sleep(60)

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alerts per node type:

| Monitor | Trigger | Severity | Recommended Response | |---|---|---|---| | Container node /state/v1/health | code not up | High | Check container node logs; verify JVM heap | | Config server application status | Non-200 | Medium | Check deployment; verify config server logs | | Search query probe | 503 or timeout | High | Check container availability; check content node coverage | | Content node health | code not up | Medium | Degraded coverage; check content node process | | Feed pipeline heartbeat | Ping not received | Medium | Check feed pipeline; verify network path |

Alert thresholds:

  • Container nodes: Alert after 2 consecutive failures (Vespa initializes briefly on restart)
  • Content nodes: Alert after 3 consecutive failures (re-indexing after restart can take time)
  • Feed pipeline: Alert after 1 missed heartbeat × grace period

Common Vespa Failure Modes

| Scenario | What Vigilmon Catches | |---|---| | Container JVM crash (OOM) | /state/v1/health fails immediately | | Container node stuck in initializing | Keyword "code":"up" absent; alert fires | | Config server down (no deploys possible) | Application status monitor fires | | Content node disk full | Content node health degrades; monitor fires | | Feed pipeline stalled (no new docs) | Heartbeat monitor fires | | Vespa cluster upgrade in progress | All health checks return initializing; fire after threshold | | Network partition between app and Vespa | External probe catches what internal JVM checks miss |

Vespa's distributed architecture means partial failures are common and subtle — a single content node failure reduces result coverage without producing HTTP errors. Vigilmon's per-node state API monitoring and query-level probes give you full-stack visibility from outside the cluster.

Get started free at vigilmon.online — your Vespa monitors are running in under 5 minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles