tutorial

Monitoring NVIDIA Triton Inference Server with Vigilmon: Health API, gRPC Port, Metrics Endpoint & SSL Certificate Alerts

How to monitor NVIDIA Triton Inference Server with Vigilmon — health API checks, gRPC port TCP monitoring, Prometheus metrics endpoint availability, and SSL certificate alerts.

How to Monitor NVIDIA Triton Inference Server with Vigilmon

NVIDIA Triton Inference Server is the industry-standard platform for serving ML models at scale — supporting TensorFlow, PyTorch, ONNX, TensorRT, and custom backends with dynamic batching, concurrent model execution, and GPU/CPU scheduling. Production Triton deployments handle everything from recommendation models to real-time vision inference at millions of requests per day.

But Triton's internal metrics, model repository polling, and readiness endpoints only tell you what the server itself sees. External monitoring with Vigilmon tells you what your clients and applications actually experience — which is often very different.

Why Triton needs external monitoring

Triton has rich built-in health endpoints (/v2/health/ready and /v2/health/live) and Prometheus metrics. Why add external monitoring on top?

Because internal and external perspectives diverge in exactly the failure modes that matter most:

  • GPU driver update on the host kills Triton's CUDA context — Triton's process is alive, readiness probe might pass, but inference requests fail with CUDA errors
  • Model repository sync fails silently — a network mount for the model store becomes stale; Triton continues serving the last loaded version without alerting operators
  • Load balancer de-registers the Triton instance after a pod restart — the pod is healthy but traffic is still routing to the old IP
  • Backend plugin segfaults — a custom backend crashes Triton's inference thread without killing the main HTTP process; the liveness probe passes but /infer returns 500s
  • TLS termination breaks at the proxy layer — clients see SSL errors while Triton's internal HTTP is fine
  • gRPC port is bound but HTTP port failed to start — your monitoring watches one protocol while client traffic uses the other

External monitoring from a neutral vantage point catches what internal readiness probes miss.

What you'll need

  • A running Triton Inference Server instance (Docker, Kubernetes, or bare metal)
  • HTTP or gRPC endpoint accessible from outside the server
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Understand Triton's health endpoints

Triton exposes two health endpoints on its HTTP port (default 8000):

# Liveness check — is the server process running?
curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/v2/health/live

# Readiness check — is the server ready to handle inference requests?
curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/v2/health/ready

Both return 200 OK when healthy. The difference:

  • /live — Triton process is alive. Returns 200 even while models are still loading.
  • /ready — All models in the model repository have loaded successfully. Returns 503 until models are ready.

For external monitoring, use /v2/health/ready — it's the authoritative signal that Triton can serve inference requests.

Starting Triton with Docker

docker run --gpus=all \
  --name triton \
  -p 8000:8000 \
  -p 8001:8001 \
  -p 8002:8002 \
  -v /path/to/model-repository:/models \
  nvcr.io/nvidia/tritonserver:24.01-py3 \
  tritonserver \
    --model-repository=/models \
    --log-verbose=1

Ports:

  • 8000 — HTTP
  • 8001 — gRPC
  • 8002 — Prometheus metrics

Step 2: Create an HTTP monitor in Vigilmon

Log in to Vigilmon and create your primary monitor:

  1. Click Monitors → New Monitor
  2. Set the Type to HTTP
  3. Enter your endpoint: https://triton.yourdomain.com/v2/health/ready
  4. Set Check interval to 1 minute
  5. Set Expected status to 200
  6. Set Alert after to 1 failure — inference server failures are rarely transient
  7. Click Save

Vigilmon begins checking from multiple geographic regions every minute.

What this monitor catches

| Failure | Vigilmon response | |---------|-----------------| | Triton process crashed | Connection refused → alert | | Model failed to load | HTTP 503 → alert | | CUDA OOM during inference | HTTP 500 → alert | | Reverse proxy misconfigured | HTTP 502/504 → alert | | Load balancer routing broken | Timeout or connection refused → alert | | DNS resolution failed | DNS error → alert | | TLS certificate expired | SSL error → alert |

Step 3: Add a TCP monitor for the gRPC port

Many high-throughput Triton clients use gRPC (port 8001) rather than HTTP. Monitor the gRPC port separately:

  1. Click Monitors → New Monitor
  2. Set the Type to TCP
  3. Enter: triton.yourdomain.com:8001
  4. Set Check interval: 1 minute
  5. Click Save

This lets you distinguish "HTTP layer broken" from "gRPC port not listening" — different root causes with different recovery procedures.

Step 4: Monitor specific model availability

Triton's /v2/models/{model_name}/ready endpoint tells you whether a specific model is loaded and ready for inference. Create keyword monitors for your critical models:

  1. Click Monitors → New Monitor
  2. Set the Type to HTTP
  3. Enter: https://triton.yourdomain.com/v2/models/yolov8/ready
  4. Set Expected status: 200
  5. Set Check interval: 5 minutes
  6. Click Save

Repeat for each model that's critical to your production pipelines. This catches model-specific failures (OOM for a large model, corrupt checkpoint) without affecting the server-level health monitors.

You can also check model metadata to verify the correct version is loaded:

curl -s https://triton.yourdomain.com/v2/models/yolov8 | python3 -m json.tool
# Returns model name, versions, platform, and input/output tensor shapes

Add a keyword monitor on this endpoint looking for your expected model version string.

Step 5: Configure alert channels

Navigate to Alerts → New Alert Channel:

Slack integration:

# In Slack, create an incoming webhook for #ml-ops or #triton-alerts
# In Vigilmon, paste the webhook URL under Alerts → Slack

PagerDuty / Opsgenie: Use Vigilmon's generic webhook output to call PagerDuty Events API v2:

POST https://events.pagerduty.com/v2/enqueue
{
  "routing_key": "YOUR_INTEGRATION_KEY",
  "event_action": "trigger",
  "payload": {
    "summary": "Triton Inference Server is DOWN",
    "severity": "critical",
    "source": "vigilmon"
  }
}

Email: Configure email alerts to your ML platform team's on-call rotation.

Example Vigilmon webhook payload:

{
  "monitor_name": "Triton /v2/health/ready",
  "status": "down",
  "url": "https://triton.yourdomain.com/v2/health/ready",
  "started_at": "2026-03-10T14:52:00Z",
  "duration_seconds": 180
}

Step 6: Correlate Vigilmon alerts with Triton diagnostics

When an alert fires, run this triage sequence:

# 1. Is the Triton process running?
docker ps | grep triton
# or for systemd:
systemctl status triton

# 2. Check Triton logs for model loading or CUDA errors
docker logs triton --tail 100 | grep -E "ERROR|FAIL|OOM|cuda" -i

# 3. Check GPU health
nvidia-smi
nvidia-smi dmon -s u -d 1 -c 5  # Watch utilization over 5 seconds

# 4. Test health endpoints directly (bypass proxy)
curl -v http://localhost:8000/v2/health/live
curl -v http://localhost:8000/v2/health/ready

# 5. Check model repository status
curl -s http://localhost:8000/v2/models | python3 -m json.tool

# 6. Check Prometheus metrics for error rates
curl -s http://localhost:8002/metrics | grep -E "nv_inference_request_failure|nv_gpu_utilization"

Triage decision tree:

  • Process not running → check host OOM killer (dmesg | grep killed), restart Triton, investigate VRAM requirements
  • Process running, /live returns 200, /ready returns 503 → models still loading or a model failed to load; check logs for specific model errors
  • Both return 200 from localhost, but Vigilmon shows down → proxy or network path issue; check nginx/load balancer logs
  • 503 persisting after restart → model repository may be inaccessible (NFS mount dead, S3 credentials expired); verify model store connectivity

Step 7: Kubernetes deployment monitoring

For Triton running in Kubernetes, external monitoring from Vigilmon is especially important because Kubernetes health probes only report what the scheduler sees:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-inference-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: triton
  template:
    metadata:
      labels:
        app: triton
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8002"
    spec:
      containers:
        - name: triton
          image: nvcr.io/nvidia/tritonserver:24.01-py3
          args:
            - tritonserver
            - --model-repository=s3://my-bucket/models
          ports:
            - containerPort: 8000  # HTTP
            - containerPort: 8001  # gRPC
            - containerPort: 8002  # Metrics
          resources:
            limits:
              nvidia.com/gpu: 1
          livenessProbe:
            httpGet:
              path: /v2/health/live
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /v2/health/ready
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
            failureThreshold: 6
---
apiVersion: v1
kind: Service
metadata:
  name: triton-inference-server
spec:
  selector:
    app: triton
  ports:
    - name: http
      port: 80
      targetPort: 8000
    - name: grpc
      port: 8001
      targetPort: 8001
    - name: metrics
      port: 8002
      targetPort: 8002
  type: LoadBalancer

Monitor the LoadBalancer external IP and hostname with Vigilmon — the readiness probe tells you what the cluster sees, Vigilmon tells you what the world sees.

Step 8: Create a status page for ML inference

  1. Go to Status Pages → New Status Page
  2. Name it: "ML Inference Platform"
  3. Add your monitors grouped by model or pipeline:
    • Core Models: Triton ready check, per-model ready checks
    • Object Detection: YOLOv8 model, ResNet50 classifier
    • NLP: BERT, embedding model
  4. Publish and share with:
    • Application teams calling your inference APIs
    • Data science teams who need to know when their pipelines are affected
    • Leadership tracking ML platform uptime SLAs

Putting it all together

| Monitor | Type | What it catches | |---------|------|----------------| | https://triton.yourdomain.com/v2/health/ready | HTTP | Server-level failures, model loading failures, proxy issues | | triton.yourdomain.com:8001 | TCP | gRPC port availability | | https://triton.yourdomain.com/v2/models/yolov8/ready | HTTP | Per-model failures without affecting server-level health | | https://triton.yourdomain.com/v2/models | HTTP keyword | Verify expected models are loaded |

What's next

  • Prometheus + Grafana — use Triton's metrics port (8002) for deep performance dashboards; use Vigilmon for the external availability signal that Prometheus can't provide
  • SSL certificate monitoring — Vigilmon tracks TLS cert expiry for your inference endpoint
  • Heartbeat monitors — use Vigilmon heartbeats to monitor offline batch inference jobs that run on a schedule

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles