tutorial

How to Monitor BentoML Model Serving with Vigilmon

BentoML is the leading open-source framework for packaging and serving machine learning models as production APIs. A BentoML service wraps your trained model...

BentoML is the leading open-source framework for packaging and serving machine learning models as production APIs. A BentoML service wraps your trained model — whether scikit-learn, PyTorch, TensorFlow, or an LLM — and exposes it as a REST or gRPC endpoint. When that endpoint goes down or starts returning errors, your entire downstream application breaks: recommendations vanish, fraud detection stops, chatbot responses fail.

In this tutorial you'll set up comprehensive monitoring for BentoML services using Vigilmon, covering health checks, inference endpoint monitoring, and alert routing.

Why BentoML services need dedicated monitoring

BentoML model servers fail in ways that are distinct from regular web services:

  • Model load failures — the server process starts successfully but the model fails to deserialize from storage; subsequent inference requests return 500 without any external health signal
  • GPU memory exhaustion — the HTTP server accepts requests but inference fails silently when the GPU runs out of memory, typically under batch load
  • Runner disconnects — BentoML 1.x separates model runners from the API server; if a runner process dies, the API server continues accepting requests but inference calls time out
  • Adaptive batching deadlocks — the batching layer can occasionally deadlock under high concurrency, accepting requests but never responding
  • Bentocloud runner pod evictions — on Kubernetes, runner pods can be evicted under memory pressure while the service pod stays healthy, causing partial service degradation that's invisible to pod-level health checks

External HTTP monitoring probes the actual inference path, not just whether the server process is alive.

What you'll need

Step 1: Add a health endpoint to your BentoML service

BentoML exposes a built-in /readyz and /livez endpoint automatically when you serve a Bento. You can also add a custom health route that exercises the inference path:

# service.py
import bentoml
from bentoml.io import JSON, NumpyNdarray
import numpy as np

# Load your model runner
runner = bentoml.sklearn.get("iris_clf:latest").to_runner()

svc = bentoml.Service("iris_classifier", runners=[runner])

@svc.api(input=NumpyNdarray(), output=JSON())
async def classify(input_data: np.ndarray) -> dict:
    result = await runner.predict.async_run(input_data)
    return {"predictions": result.tolist()}

@svc.api(input=JSON(), output=JSON(), route="/health/deep")
async def deep_health_check(input_data: dict) -> dict:
    """
    Deep health check that exercises the runner.
    Returns 200 only if the model can actually run inference.
    """
    try:
        test_input = np.array([[5.1, 3.5, 1.4, 0.2]])
        result = await runner.predict.async_run(test_input)
        return {
            "status": "ok",
            "runner": "connected",
            "test_prediction": result.tolist()[0]
        }
    except Exception as e:
        # BentoML returns 500 on unhandled exceptions
        raise bentoml.exceptions.BentoMLException(
            f"Runner health check failed: {e}"
        )

The built-in endpoints are available without any extra code:

# Liveness (is the server process alive?)
curl http://localhost:3000/livez
# Returns: {"status":"ok"}

# Readiness (is the model loaded and ready?)
curl http://localhost:3000/readyz
# Returns: {"status":"ok"} — only 200 when runners are ready

# Deep check (can the model actually do inference?)
curl -X POST http://localhost:3000/health/deep \
  -H "Content-Type: application/json" \
  -d '{}'

Step 2: Serve your BentoML service with health checks enabled

When running with Docker, expose both the main service port (3000) and configure proper restart behavior:

# Dockerfile - BentoML generates this automatically, but you can customize
FROM bentoml/bentoml:1.2-python3.11

COPY . /bento
WORKDIR /bento

# Install dependencies
RUN pip install -r requirements.txt

EXPOSE 3000

# BentoML's built-in health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
  CMD curl -f http://localhost:3000/readyz || exit 1

CMD ["bentoml", "serve", "service:svc", "--port", "3000", "--host", "0.0.0.0"]

With Docker Compose:

version: "3.9"

services:
  iris-classifier:
    build: .
    ports:
      - "3000:3000"
    environment:
      BENTOML_NUM_RUNNERS: "2"
      BENTOML_RUNNER_TIMEOUT: "60"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/readyz"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Note the start_period: 60s — BentoML takes time to load models into GPU memory, especially for large LLMs. Without a start period, Vigilmon may falsely report the service as down during initialization.

Step 3: Monitor the readiness endpoint in Vigilmon

The /readyz endpoint is the right target for Vigilmon because it returns 200 only when the model is loaded and runners are ready:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://your-bentoml-service.example.com/readyz
  4. Check interval: 1 minute
  5. Expected response:
    • Status code: 200
    • Body contains: "status":"ok"
  6. Save the monitor

Add a second HTTP monitor for the deep health check:

  • URL: https://your-bentoml-service.example.com/health/deep
  • Method: POST with body {}
  • Expected: 200, body contains "status":"ok"

The deep check will catch runner disconnects that the /readyz endpoint misses.

Step 4: Monitor BentoML on Kubernetes (BentoCloud / self-hosted)

When running BentoML on Kubernetes — either via BentoCloud or your own cluster — add TCP checks to catch pod evictions and service disruptions:

# Get your service external IP
kubectl get svc -n bentoml my-bento-service \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

In Vigilmon:

  1. Monitors → New Monitor → TCP Port
  2. Hostname: your LoadBalancer IP or hostname
  3. Port: 3000 (or whatever your service port is)
  4. Save

Add Kubernetes ingress monitoring if you use an ingress controller:

  • TCP monitor: :80 and :443 on your ingress IP

Step 5: Add inference monitoring with synthetic requests

For critical production services, monitor inference quality — not just reachability. Add a synthetic inference check that validates the response:

# vigilmon_inference_check.py
# Deploy as a separate lightweight service or cron job
from http.server import HTTPServer, BaseHTTPRequestHandler
import requests
import numpy as np
import json
import os

BENTOML_URL = os.getenv("BENTOML_URL", "http://localhost:3000")
VIGILMON_HEARTBEAT = os.getenv("VIGILMON_HEARTBEAT_URL", "")

class InferenceHealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return

        try:
            # Send a known test input with a known expected output
            test_input = [5.1, 3.5, 1.4, 0.2]  # Expected: Iris Setosa (class 0)
            
            response = requests.post(
                f"{BENTOML_URL}/classify",
                json=test_input,
                timeout=10
            )
            
            if response.status_code != 200:
                raise ValueError(f"Non-200 response: {response.status_code}")
            
            result = response.json()
            prediction = result.get("predictions", [None])[0]
            
            # Validate the prediction is correct
            if prediction != 0:
                raise ValueError(f"Wrong prediction: expected 0, got {prediction}")
            
            # Ping heartbeat if configured
            if VIGILMON_HEARTBEAT:
                requests.get(VIGILMON_HEARTBEAT, timeout=5)
            
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({
                "status": "ok",
                "prediction": prediction,
                "latency_ms": response.elapsed.total_seconds() * 1000
            }).encode())

        except Exception as e:
            self.send_response(503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({
                "status": "error",
                "detail": str(e)
            }).encode())

    def log_message(self, format, *args):
        pass

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8080), InferenceHealthHandler)
    print("Inference health check running on :8080")
    server.serve_forever()

Point a Vigilmon HTTP monitor at :8080/health to check both availability and prediction correctness.

Step 6: Monitor GPU-based services with heartbeat checks

For LLM or heavy model serving where inference takes seconds, standard HTTP health checks can time out and create false alerts. Use a heartbeat pattern instead:

# gpu_health_daemon.py - runs as a sidecar
import requests
import time
import os
import bentoml

VIGILMON_HEARTBEAT = os.getenv("VIGILMON_HEARTBEAT_URL")
BENTOML_URL = os.getenv("BENTOML_URL", "http://localhost:3000")
CHECK_INTERVAL = int(os.getenv("CHECK_INTERVAL_SECONDS", "60"))

def check_and_ping():
    """Run a test inference and ping heartbeat if successful."""
    try:
        response = requests.post(
            f"{BENTOML_URL}/generate",
            json={"prompt": "Hello", "max_tokens": 1},
            timeout=30  # Long timeout for GPU models
        )
        
        if response.status_code == 200:
            requests.get(VIGILMON_HEARTBEAT, timeout=5)
            print(f"Heartbeat sent at {time.strftime('%H:%M:%S')}")
        else:
            print(f"Inference failed: {response.status_code}")
    
    except Exception as e:
        print(f"Health check failed: {e}")
        # Not pinging — Vigilmon will alert on missed heartbeat

if __name__ == "__main__":
    print(f"GPU health daemon started, checking every {CHECK_INTERVAL}s")
    while True:
        check_and_ping()
        time.sleep(CHECK_INTERVAL)

In Vigilmon, create a Heartbeat monitor with a 3-minute grace period and point the daemon's VIGILMON_HEARTBEAT_URL to it.

Step 7: Configure alerts and escalation

  1. Go to Alert Channels → Add Channel
  2. Add Slack with your ML team webhook
  3. Add Email for the on-call engineer
  4. Add PagerDuty webhook for P0 services

Set up escalation rules:

  • Readiness check down → Slack immediately
  • Deep inference check failing → Slack + email
  • GPU heartbeat missed → Slack + PagerDuty

Step 8: Create a status page for your model APIs

If external teams or customers consume your BentoML inference APIs, a public status page reduces support noise:

  1. Status Pages → New Status Page
  2. Add monitors grouped by service:
    • "Image Classification API" → readyz + inference check
    • "Text Generation API" → readyz + GPU heartbeat
  3. Publish the page at https://status.vigilmon.online/your-page

Share the status page URL with API consumers so they can self-diagnose before opening support tickets.

Complete monitoring setup

| Monitor | Type | What it catches | |---------|------|-----------------| | /readyz | HTTP | Model load failure, runner crash | | /health/deep | HTTP | Runner disconnect, inference errors | | Inference synthetic check | HTTP | Wrong predictions, regression | | Service port :3000 | TCP | Port binding failures | | GPU model heartbeat | Heartbeat | Long-running inference stalls | | Ingress :443 | TCP | TLS/network issues |

What's next

  • Response time monitoring — Vigilmon tracks P50/P95 response time for your inference endpoint; catch latency regressions before they violate SLAs
  • SSL certificate monitoring — automatic alerts when your model serving HTTPS certificate approaches expiry
  • Multi-environment monitoring — set up separate monitors for staging and production BentoML services with different alert severity levels

Start monitoring your BentoML services for free at vigilmon.online — no credit card, monitors live in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles