tutorial

How to Monitor Apache PredictionIO Health and Model Serving Availability with Vigilmon

PredictionIO engine server crashes and stale model deployments silently break your recommendation and prediction APIs. Learn how to monitor PredictionIO engine health, event server availability, and client application health with Vigilmon HTTP probes and heartbeat monitors.

Apache PredictionIO is the open-source machine learning server that lets developers build, deploy, and serve predictive models — powering recommendation engines, classification APIs, and real-time prediction services — without managing the underlying ML infrastructure. But when an engine server crashes after a model retrain, the Event Server becomes unreachable, or a deployed engine starts returning degraded predictions due to model drift, your production ML pipeline fails silently. Users receive stale recommendations, prediction APIs return errors, and the Event Server quietly drops training data without any visible alarm.

Vigilmon gives you external visibility into PredictionIO platform health through HTTP probe monitoring (via PredictionIO's built-in REST API) and heartbeat monitoring for your PredictionIO client applications. This tutorial covers both.

Why PredictionIO Needs External Monitoring

PredictionIO's built-in tooling (engine server status, Event Server API, HBase/Elasticsearch backends) provides basic operational diagnostics — but only if someone is watching. External monitoring with Vigilmon adds:

  • Proactive alerting when the engine server REST endpoint becomes unreachable (immediately visible to end users as prediction API failures)
  • Event Server liveness detection so dropped training events don't silently corrupt your next model retrain
  • Heartbeat monitoring so you know immediately when a client application stops sending events or querying predictions
  • Multi-region availability checking from outside your PredictionIO deployment network

These layers work together: the engine server can be reachable while returning degraded predictions due to a failed model retrain, and the Event Server can be accepting writes while silently failing to persist them to HBase.

Step 1: Build a PredictionIO Health Endpoint

PredictionIO exposes native HTTP endpoints on both the engine server (default port 8000) and Event Server (default port 7070). Use them directly or wrap in a custom health sidecar.

Using the PredictionIO REST API Directly

# Check engine server is running
curl -i http://pio-engine.example.com:8000/

# Query a prediction (replace with your engine's query schema)
curl -i -X POST http://pio-engine.example.com:8000/queries.json \
  -H 'Content-Type: application/json' \
  -d '{"user": "health-probe-user", "num": 1}'

# Check Event Server is reachable
curl -i http://pio-events.example.com:7070/

# Check events for a specific app
curl -i "http://pio-events.example.com:7070/events.json?accessKey=YOUR_ACCESS_KEY&limit=1"

Point a Vigilmon monitor directly at http://pio-engine.example.com:8000/ for a quick engine reachability check.

Custom Python Health Sidecar

For deeper health checks — engine server query responsiveness, Event Server connectivity, and backend storage availability:

# predictionio_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

ENGINE_URL = os.environ.get('PIO_ENGINE_URL', 'http://localhost:8000')
EVENT_SERVER_URL = os.environ.get('PIO_EVENT_SERVER_URL', 'http://localhost:7070')
ACCESS_KEY = os.environ.get('PIO_ACCESS_KEY', '')
PROBE_USER_ID = os.environ.get('PIO_PROBE_USER', 'health-probe-user')
PROBE_QUERY = os.environ.get('PIO_PROBE_QUERY', '{"user": "health-probe-user", "num": 1}')

@app.route('/health/predictionio')
def predictionio_health():
    issues = []

    # Check engine server responds to predictions
    try:
        resp = requests.post(
            f'{ENGINE_URL}/queries.json',
            json=eval(PROBE_QUERY),
            timeout=10
        )
        resp.raise_for_status()
        result = resp.json()
        if not result:
            issues.append('engine_returned_empty_prediction')
    except requests.RequestException as e:
        return jsonify({
            'status': 'down',
            'reason': 'engine_unreachable',
            'error': str(e),
        }), 503

    # Check Event Server is reachable
    try:
        event_resp = requests.get(
            f'{EVENT_SERVER_URL}/events.json',
            params={'accessKey': ACCESS_KEY, 'limit': 1},
            timeout=5
        )
        event_resp.raise_for_status()
    except requests.RequestException as e:
        issues.append(f'event_server_degraded: {str(e)}')

    if issues:
        return jsonify({'status': 'degraded', 'issues': issues}), 503

    return jsonify({
        'status': 'ok',
        'engine': 'reachable',
        'event_server': 'reachable',
    }), 200

if __name__ == '__main__':
    app.run(port=8094)

Java Health Sidecar (PredictionIO Client)

// PredictionIOHealthController.java
@RestController
public class PredictionIOHealthController {

    @Value("${pio.engine.url:http://localhost:8000}")
    private String engineUrl;

    @Value("${pio.event.server.url:http://localhost:7070}")
    private String eventServerUrl;

    @Value("${pio.access.key}")
    private String accessKey;

    private final RestTemplate restTemplate = new RestTemplate();

    @GetMapping("/health/predictionio")
    public ResponseEntity<Map<String, Object>> checkHealth() {
        Map<String, Object> status = new LinkedHashMap<>();
        List<String> issues = new ArrayList<>();

        // Check engine server with a probe query
        try {
            Map<String, Object> query = Map.of(
                "user", "health-probe-user",
                "num", 1
            );
            ResponseEntity<Map> engineResp = restTemplate.postForEntity(
                engineUrl + "/queries.json", query, Map.class);

            if (!engineResp.getStatusCode().is2xxSuccessful()
                    || engineResp.getBody() == null) {
                status.put("status", "down");
                status.put("reason", "engine_bad_response");
                return ResponseEntity.status(503).body(status);
            }
        } catch (Exception e) {
            status.put("status", "down");
            status.put("reason", "engine_unreachable");
            status.put("error", e.getMessage());
            return ResponseEntity.status(503).body(status);
        }

        // Check Event Server
        try {
            String eventUrl = eventServerUrl + "/events.json?accessKey=" + accessKey + "&limit=1";
            restTemplate.getForEntity(eventUrl, List.class);
        } catch (Exception e) {
            issues.add("event_server_degraded: " + e.getMessage());
        }

        if (!issues.isEmpty()) {
            status.put("status", "degraded");
            status.put("issues", issues);
            return ResponseEntity.status(503).body(status);
        }

        status.put("status", "ok");
        status.put("engine", "reachable");
        status.put("event_server", "reachable");
        return ResponseEntity.ok(status);
    }
}

Step 2: Configure Vigilmon HTTP Monitor for PredictionIO

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your PredictionIO health endpoint: https://your-app.example.com/health/predictionio
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms (prediction queries involve model inference and can be slow under cold-start conditions)
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the engine server directly:

  • URL: http://pio-engine.example.com:8000/
  • Expected: 200
  • Interval: 2 minutes
  • Alert channel: ml-ops channel

Vigilmon's multi-region probe consensus prevents false positives from transient network blips between probe nodes and your PredictionIO deployment.

Step 3: Heartbeat Monitoring for PredictionIO Client Applications

Engine and Event Server health checks are necessary — but not sufficient. A PredictionIO engine can serve predictions while the underlying model is stale because a scheduled retrain failed silently. An application can be calling the prediction API and receiving responses without actually sending the user interaction events needed for future model training. Vigilmon heartbeat monitors detect these silent pipeline breaks: your application pings Vigilmon after successfully completing each batch of prediction requests or event submissions.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: predictionio-event-pipeline
  3. Set the expected interval: 5 minutes (adjust to your event submission cadence)
  4. Set the grace period: 15 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your PredictionIO Python Client

# monitored_pio_client.py
import predictionio
import requests
import os
import time

HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']
ACCESS_KEY = os.environ['PIO_ACCESS_KEY']
ENGINE_URL = os.environ.get('PIO_ENGINE_URL', 'http://localhost:8000')

event_client = predictionio.EventClient(
    access_key=ACCESS_KEY,
    url=os.environ.get('PIO_EVENT_SERVER_URL', 'http://localhost:7070'),
    threads=1,
)
engine_client = predictionio.EngineClient(url=ENGINE_URL)

events_sent = 0
last_heartbeat = time.time()
HEARTBEAT_INTERVAL = 60  # seconds

def record_user_action(user_id: str, item_id: str, event: str = 'view'):
    global events_sent, last_heartbeat

    event_client.create_event(
        event=event,
        entity_type='user',
        entity_id=user_id,
        target_entity_type='item',
        target_entity_id=item_id,
    )
    events_sent += 1

    if time.time() - last_heartbeat > HEARTBEAT_INTERVAL:
        try:
            requests.get(HEARTBEAT_URL, timeout=5)
        except Exception:
            pass
        last_heartbeat = time.time()

def get_recommendations(user_id: str, num: int = 10) -> list:
    result = engine_client.send_query({'user': user_id, 'num': num})
    return result.get('itemScores', [])

Node.js PredictionIO Client with Heartbeat

// monitoredPioClient.js
const axios = require('axios');

const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;
const ACCESS_KEY = process.env.PIO_ACCESS_KEY;
const EVENT_SERVER_URL = process.env.PIO_EVENT_SERVER_URL || 'http://localhost:7070';
const ENGINE_URL = process.env.PIO_ENGINE_URL || 'http://localhost:8000';

let eventCount = 0;
let lastHeartbeat = Date.now();
const HEARTBEAT_INTERVAL_MS = 60 * 1000;

async function recordEvent(userId, itemId, event = 'view') {
  await axios.post(`${EVENT_SERVER_URL}/events.json?accessKey=${ACCESS_KEY}`, {
    event,
    entityType: 'user',
    entityId: userId,
    targetEntityType: 'item',
    targetEntityId: itemId,
    eventTime: new Date().toISOString(),
  });

  eventCount++;
  if (Date.now() - lastHeartbeat > HEARTBEAT_INTERVAL_MS) {
    try {
      await axios.get(HEARTBEAT_URL, { timeout: 5000 });
    } catch (_) {
      // Non-fatal
    }
    lastHeartbeat = Date.now();
  }
}

async function getRecommendations(userId, num = 10) {
  const resp = await axios.post(`${ENGINE_URL}/queries.json`, { user: userId, num });
  return resp.data.itemScores || [];
}

module.exports = { recordEvent, getRecommendations };

Step 4: Alert Routing for PredictionIO Failures

PredictionIO failures cascade: a failed model retrain leaves the engine serving stale predictions, event submission gaps degrade future model quality, and HBase or Elasticsearch unavailability causes the engine to fail entirely. Alert routing should reflect this cascade.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | PredictionIO health /health/predictionio | Slack + PagerDuty | P1 | | Engine server /:8000/ | Slack | P2 | | Heartbeat: event submission pipeline | Slack + email | P2 | | Heartbeat: recommendation API client | Email | P3 |

Set response time thresholds for early warning:

  • Alert at 5000ms for the health endpoint (slow prediction responses signal model inference latency or resource contention)
  • Alert at 15000ms for cold-start probe queries (engines cold-start on first request after deployment)

For recommendation-critical applications, enable three consecutive failures before alerting — PredictionIO engine deployments cause brief unavailability during model swap-in that resolves automatically.

Summary

PredictionIO failures are silent at the recommendation and prediction layer. External monitoring catches engine crashes, Event Server gaps, and training pipeline stalls before they degrade model quality or break user-facing APIs:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/predictionio | Engine server liveness, prediction API responsiveness, Event Server health | | HTTP monitor on engine server port | Engine availability for direct API callers | | Heartbeat monitor | Event submission pipeline liveness, client application throughput |

Get started free at vigilmon.online — your first PredictionIO monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles