tutorial

Monitoring ONNX Runtime with Vigilmon

ONNX Runtime accelerates ML model inference across CPUs, GPUs, and specialized accelerators — but when inference endpoints become unresponsive, latency degrades under load, model versions diverge between replicas, or GPU memory leaks accumulate across long-running sessions, your production ML serving pipeline degrades silently. Here's how to monitor ONNX Runtime inference availability, latency percentiles, model health, and hardware utilization with Vigilmon.

ONNX Runtime is a high-performance inference engine for machine learning models in ONNX format, used across production serving stacks at scale — from real-time NLP and computer vision APIs to recommendation systems running billions of requests per day. It supports hardware execution providers (CUDA, TensorRT, OpenVINO, DirectML) that dramatically accelerate inference beyond CPU-only throughput. When the inference service becomes unresponsive due to a model loading failure, when P99 latency climbs because GPU memory fragmentation forces fallback to CPU execution, when a model deployment pushes a shape-incompatible ONNX file that only fails on specific input shapes at runtime, or when a memory leak in a long-running session causes gradual OOM, production ML quality degrades silently — users experience slow or incorrect responses while metrics dashboards show the service as "up." Vigilmon provides external monitoring for ONNX Runtime inference availability, response latency, model version consistency, hardware utilization, and session health so ML serving incidents are detected before they impact users.

What You'll Set Up

  • Inference endpoint availability and response correctness
  • Inference latency percentile monitoring (P50, P95, P99)
  • Model version and metadata consistency across replicas
  • GPU/accelerator memory utilization and OOM detection
  • Session health and throughput rate monitoring

Prerequisites

  • ONNX Runtime serving via a REST or gRPC endpoint (e.g., Triton Inference Server, custom FastAPI/Flask wrapper, or ONNX Runtime Server)
  • Prometheus metrics endpoint enabled (if using Triton or a custom metrics shim)
  • A representative test input payload for your model (for inference probing)
  • A free Vigilmon account

Step 1: Monitor Inference Endpoint Availability

The most fundamental check: can your inference endpoint accept requests and return valid responses? This is not just a TCP connectivity check — a model loading failure can leave the HTTP server running while the inference handler panics on every request. Use an HTTP monitor with response content validation:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to your model's health endpoint (e.g., https://inference.your-org.internal/v2/health/ready for Triton, or your custom /health path).
  3. Set Check interval to 1 minute.
  4. Under Expected response, add keyword "ready" or "status":"ok" depending on your server.
  5. Set Timeout to 10 seconds.

For services using Triton Inference Server (which wraps ONNX Runtime):

# Triton exposes KServe v2 health endpoints:
# GET /v2/health/live  — server is running
# GET /v2/health/ready — server AND all models are loaded and ready
curl -sf https://inference.your-org.internal/v2/health/ready
# Returns HTTP 200 when all models are ready, 503 when any model fails to load

For custom ONNX Runtime wrappers, add an inference probe that runs an actual forward pass:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 2 minutes.
#!/bin/bash
# /usr/local/bin/onnx-inference-probe.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
INFERENCE_URL="https://inference.your-org.internal/v1/predict"
MODEL_NAME="your-model-name"
TIMEOUT=30

# Send a minimal valid test input to the model
# Replace with your actual model's input schema
RESPONSE=$(curl -sf \
  --max-time "$TIMEOUT" \
  -X POST "$INFERENCE_URL" \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "'"$MODEL_NAME"'",
    "inputs": [{"name": "input_ids", "shape": [1, 10], "datatype": "INT64",
                "data": [101, 7592, 1010, 2088, 999, 102, 0, 0, 0, 0]}]
  }' 2>/dev/null)

if echo "$RESPONSE" | python3 -c "
import sys, json
data = json.load(sys.stdin)
# Verify response has outputs and no error field
has_outputs = 'outputs' in data or 'predictions' in data
has_error = 'error' in data and data['error']
sys.exit(0 if has_outputs and not has_error else 1)
" 2>/dev/null; then
  echo "ONNX Runtime inference probe successful"
  curl -s "$HEARTBEAT_URL"
else
  echo "ONNX Runtime inference probe failed: $RESPONSE"
  exit 1
fi

Running an actual forward pass catches model loading failures, shape validation errors, and execution provider fallbacks that a simple HTTP health check misses.

Step 2: Monitor Inference Latency Percentiles

ONNX Runtime inference latency can degrade for several reasons: GPU memory fragmentation forcing CPU fallback, model input batching misconfiguration, TensorRT engine rebuild on shape change, or thermal throttling on GPU nodes. P50 latency degradation catches gradual issues; P99 spikes catch tail latency problems that affect users at the edge of your input distribution.

#!/bin/bash
# /usr/local/bin/onnx-latency-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
INFERENCE_URL="https://inference.your-org.internal/v1/predict"
MODEL_NAME="your-model-name"
MAX_P50_MS=50    # Alert if median latency exceeds 50ms
MAX_P99_MS=500   # Alert if P99 latency exceeds 500ms
SAMPLES=20       # Number of inference calls to sample
TIMEOUT=30

LATENCIES=()

for i in $(seq 1 $SAMPLES); do
  START=$(date +%s%N)
  STATUS=$(curl -sf \
    --max-time "$TIMEOUT" \
    -o /dev/null \
    -w "%{http_code}" \
    -X POST "$INFERENCE_URL" \
    -H "Content-Type: application/json" \
    -d '{"model_name": "'"$MODEL_NAME"'",
         "inputs": [{"name": "input_ids", "shape": [1, 10], "datatype": "INT64",
                     "data": [101, 7592, 1010, 2088, 999, 102, 0, 0, 0, 0]}]}' \
    2>/dev/null)
  END=$(date +%s%N)
  
  LATENCY_MS=$(( (END - START) / 1000000 ))
  LATENCIES+=("$LATENCY_MS")
done

# Calculate percentiles
printf '%s\n' "${LATENCIES[@]}" | sort -n | python3 -c "
import sys
latencies = sorted([int(x.strip()) for x in sys.stdin if x.strip()])
n = len(latencies)
if n == 0:
    print('No latency samples collected')
    sys.exit(1)

p50 = latencies[int(n * 0.50)]
p95 = latencies[int(n * 0.95)]
p99 = latencies[min(int(n * 0.99), n-1)]

print(f'Latency - P50: {p50}ms, P95: {p95}ms, P99: {p99}ms')

issues = []
if p50 > ${MAX_P50_MS}:
    issues.append(f'P50 {p50}ms exceeds threshold ${MAX_P50_MS}ms')
if p99 > ${MAX_P99_MS}:
    issues.append(f'P99 {p99}ms exceeds threshold ${MAX_P99_MS}ms')

if issues:
    print('LATENCY ISSUES: ' + '; '.join(issues))
    sys.exit(1)
else:
    sys.exit(0)
" && curl -s "$HEARTBEAT_URL"

Run this every 5 minutes. A P50 that climbs from 20ms to 200ms over several hours indicates gradual GPU memory pressure; a sudden P99 spike often points to a model shape change triggering a TensorRT engine rebuild on the first affected request.

Step 3: Monitor Model Version Consistency Across Replicas

When you serve ONNX Runtime behind a load balancer with multiple replicas, model version inconsistency is a subtle failure mode. A rolling deployment that updates 2 of 3 replicas means some requests receive predictions from the old model and some from the new — resulting in inconsistent output distributions that are hard to diagnose from user-visible metrics. Monitor model metadata across replicas:

#!/bin/bash
# /usr/local/bin/onnx-version-consistency.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
MODEL_NAME="your-model-name"

# List of replica endpoints — adjust for your deployment topology
REPLICAS=(
  "https://inference-0.your-org.internal"
  "https://inference-1.your-org.internal"
  "https://inference-2.your-org.internal"
)

VERSIONS=()
for REPLICA in "${REPLICAS[@]}"; do
  VERSION=$(curl -sf \
    --max-time 10 \
    "${REPLICA}/v2/models/${MODEL_NAME}" 2>/dev/null | \
    python3 -c "
import sys, json
data = json.load(sys.stdin)
# KServe v2 returns versions list; take the active one
versions = data.get('versions', [])
print(versions[-1] if versions else data.get('version', 'unknown'))
" 2>/dev/null || echo "unreachable")
  VERSIONS+=("$VERSION")
  echo "Replica ${REPLICA}: version=${VERSION}"
done

# Check all versions match
UNIQUE_VERSIONS=$(printf '%s\n' "${VERSIONS[@]}" | sort -u | grep -v "^$" | wc -l)

if [ "$UNIQUE_VERSIONS" -eq 1 ]; then
  DEPLOYED_VERSION="${VERSIONS[0]}"
  echo "All replicas running model version: ${DEPLOYED_VERSION}"
  curl -s "$HEARTBEAT_URL"
else
  echo "Model version mismatch across replicas:"
  printf '%s\n' "${VERSIONS[@]}" | sort | uniq -c
  exit 1
fi

For Triton-based deployments, the model repository polling interval controls how quickly version changes propagate. A version mismatch lasting more than your polling interval plus one check cycle indicates a replica that failed to pull the new model version.

Step 4: Monitor GPU Memory Utilization and OOM Risk

ONNX Runtime with CUDA or TensorRT execution providers allocates GPU memory for model weights, activations, and inference buffers. Memory leaks in long-running inference sessions — particularly when using dynamic input shapes with TensorRT or when session options are misconfigured — cause gradual OOM that eventually crashes the serving process. Monitor GPU memory before it causes an outage:

#!/bin/bash
# /usr/local/bin/onnx-gpu-memory-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
MAX_GPU_MEMORY_PERCENT=85   # Alert at 85% to catch trends before OOM

# Query GPU memory via nvidia-smi
GPU_STATS=$(nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu \
  --format=csv,noheader,nounits 2>/dev/null)

if [ -z "$GPU_STATS" ]; then
  echo "nvidia-smi not available — skipping GPU check"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

MEMORY_OK=true

while IFS=',' read -r GPU_INDEX MEM_USED MEM_TOTAL GPU_UTIL; do
  MEM_USED=$(echo "$MEM_USED" | tr -d ' ')
  MEM_TOTAL=$(echo "$MEM_TOTAL" | tr -d ' ')
  GPU_UTIL=$(echo "$GPU_UTIL" | tr -d ' ')
  
  if [ -z "$MEM_TOTAL" ] || [ "$MEM_TOTAL" -eq 0 ]; then
    continue
  fi
  
  MEM_PERCENT=$(( MEM_USED * 100 / MEM_TOTAL ))
  echo "GPU ${GPU_INDEX}: memory=${MEM_USED}/${MEM_TOTAL}MB (${MEM_PERCENT}%), util=${GPU_UTIL}%"
  
  if [ "$MEM_PERCENT" -gt "$MAX_GPU_MEMORY_PERCENT" ]; then
    echo "WARNING: GPU ${GPU_INDEX} memory at ${MEM_PERCENT}% (threshold: ${MAX_GPU_MEMORY_PERCENT}%)"
    MEMORY_OK=false
  fi
done <<< "$GPU_STATS"

if [ "$MEMORY_OK" = true ]; then
  echo "GPU memory utilization within limits"
  curl -s "$HEARTBEAT_URL"
else
  echo "GPU memory pressure detected — OOM risk"
  exit 1
fi

For containerized deployments, also monitor container-level memory limits:

# In Kubernetes, check GPU memory via the pod's environment
kubectl exec -n inference "$INFERENCE_POD" -- \
  nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader,nounits

Run GPU memory checks every 2 minutes. A GPU whose memory usage grows monotonically by even 50MB per hour will OOM within 24 hours — catching the trend early allows for a scheduled pod restart rather than an emergency incident.

Step 5: Monitor Inference Throughput and Session Count

ONNX Runtime sessions (InferenceSession objects) can accumulate if your serving layer leaks session handles or creates a new session per request rather than pooling sessions. Each leaked session holds model weights in memory — 1,000 leaked sessions for a 500MB model consumes 500GB of RAM. Monitor active session count and throughput rate if your serving framework exposes them:

#!/bin/bash
# /usr/local/bin/onnx-throughput-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
METRICS_URL="https://inference.your-org.internal/metrics"   # Prometheus endpoint
MIN_RPS=0.1          # Alert if throughput drops below 0.1 req/s (serving but idle for too long)
MAX_RPS=10000        # Alert if throughput exceeds expected ceiling (runaway load)
MAX_ACTIVE_SESSIONS=100

# Fetch Prometheus metrics
METRICS=$(curl -sf --max-time 15 "$METRICS_URL" 2>/dev/null)

if [ -z "$METRICS" ]; then
  echo "Metrics endpoint unavailable"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

# Parse inference throughput (metric names vary by serving framework)
# Triton: nv_inference_request_success
# Custom: onnx_inference_requests_total
echo "$METRICS" | python3 -c "
import sys, re

metrics = sys.stdin.read()

# Try Triton metric names first, then generic
patterns = [
    r'nv_inference_request_success\s+(\d+\.?\d*)',
    r'onnx_inference_requests_total\{[^}]*\}\s+(\d+\.?\d*)',
    r'inference_requests_total\s+(\d+\.?\d*)',
]

total_requests = None
for p in patterns:
    m = re.search(p, metrics)
    if m:
        total_requests = float(m.group(1))
        break

# Check active session count if available
session_match = re.search(r'onnx_active_sessions\s+(\d+)', metrics)
active_sessions = int(session_match.group(1)) if session_match else None

print(f'Total requests (counter): {total_requests}')
if active_sessions is not None:
    print(f'Active sessions: {active_sessions}')
    if active_sessions > ${MAX_ACTIVE_SESSIONS}:
        print(f'SESSION LEAK: {active_sessions} active sessions (max: ${MAX_ACTIVE_SESSIONS})')
        sys.exit(1)

sys.exit(0)
" && curl -s "$HEARTBEAT_URL"

Step 6: Set Up Alert Channels

Configure Vigilmon to route ONNX Runtime alerts to the right teams:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #ml-serving for all ONNX Runtime monitors.
  3. Add PagerDuty for the inference availability and GPU OOM monitors — these are production-impacting incidents.
  4. Add email for the model version consistency monitor — a version mismatch is important to resolve but rarely an immediate outage.
  5. Set Consecutive failures before alert to 2 for the latency monitor — a single slow inference probe can be a cold-start artifact.

Include model context in Slack notifications:

ALERT_MSG=":warning: ONNX Runtime: P99 latency ${P99_MS}ms on model *${MODEL_NAME}* (threshold: ${MAX_P99_MS}ms). Check GPU memory and execution provider status."

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-type: application/json' \
  --data "{\"text\": \"$ALERT_MSG\", \"channel\": \"#ml-serving\"}"

Add maintenance windows during model deployments:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "ONNX_INFERENCE_MONITOR_ID",
    "duration_minutes": 5,
    "reason": "ONNX model version deployment — inference restart for model reload"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (health) | /v2/health/ready or /health keyword | Model load failure, serving process crash | | Cron heartbeat (inference probe) | Actual forward pass with test input | Shape mismatch, execution provider errors, silent failures | | Cron heartbeat (latency) | P50/P99 of sampled inference calls | GPU fallback to CPU, memory fragmentation, throttling | | Cron heartbeat (version) | Model metadata per replica | Rolling deployment inconsistency, stale replicas | | Cron heartbeat (GPU memory) | nvidia-smi memory percent | OOM risk, session leaks, memory pressure trends | | Cron heartbeat (throughput) | Prometheus inference counters | Session leaks, throughput anomalies |

ONNX Runtime's performance and correctness depend on a healthy interaction between the inference engine, execution providers, hardware, and model artifacts. A simple HTTP health check only tells you the server is responding — it won't catch a P99 regression, a GPU memory leak, or a model version mismatch that silently degrades prediction quality for a subset of users. Vigilmon gives you external visibility into every layer of your inference serving stack so ML incidents are detected before they become user-visible quality problems.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles