Monitoring KServe with Vigilmon: Model Serving Health, Inference Endpoint Alerts & Controller Status

KServe is the Kubernetes-native platform for serving machine learning models at scale. Every inference request — whether from a real-time recommendation engine, a fraud detection pipeline, or a computer vision API — passes through KServe's InferenceService pods. When KServe is healthy, model predictions arrive with sub-100ms latency. When it fails, one of two things happens: either model endpoints return 503s and downstream services silently degrade (often with no error surface to end users), or the controller stops reconciling new model deployments while existing traffic appears unaffected. Vigilmon monitors KServe's health endpoints and inference service availability from outside the cluster, alerting your ML platform team the moment either failure mode appears.

What You'll Build

• HTTP monitors for KServe's controller manager health endpoint
• HTTP monitors for each InferenceService's predictor pod
• A heartbeat monitor to verify model serving requests are completing successfully
• Tiered alert routing: inference endpoint failures page on-call, controller degradation notifies the ML platform Slack channel

Prerequisites

• KServe installed in a Kubernetes cluster (version 0.11+)
• At least one InferenceService deployed and healthy
• An Ingress controller exposing KServe endpoints over HTTPS
• A free account at vigilmon.online

Why KServe Monitoring Is Not Optional

KServe's failure modes are uniquely difficult to detect:

Silent serving degradation. An InferenceService can report Ready: True in kubectl get inferenceservice while its predictor pod is OOMKilled and restarting. During the restart window, every inference request returns 503 — but your Kubernetes control plane shows the service as healthy because it checks the resource status, not the actual endpoint.

Canary and traffic split failures are invisible. KServe supports traffic splitting between model versions using its spec.predictor.canaryTrafficPercent. If the canary predictor crashes, traffic silently routes 100% to the previous version. You won't know the canary failed unless you're watching the predictor pod health directly.

Controller failure blocks new deployments without affecting existing ones. The KServe controller manager handles InferenceService reconciliation. If it crashes, existing model servers keep running, but new model deployments stall indefinitely with no error signal beyond a hung kubectl get inferenceservice status.

Transformer and explainer components fail independently. KServe's architecture separates the predictor, transformer (pre/post-processing), and explainer into separate pods. Predictor health doesn't imply transformer health — and a crashed transformer means your model API silently drops any requests that require pre-processing.

Step 1: Verify KServe Health Endpoints

KServe's controller manager exposes a health endpoint, and each InferenceService predictor exposes its own:

# Check controller manager health
kubectl exec -n kserve deploy/kserve-controller-manager -- \
  wget -qO- http://localhost:8081/healthz

# List your InferenceServices
kubectl get inferenceservice -A

# Check a specific predictor pod health
PREDICTOR_POD=$(kubectl get pods -n <namespace> \
  -l serving.kserve.io/inferenceservice=<model-name> \
  -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n <namespace> $PREDICTOR_POD -- \
  wget -qO- http://localhost:8080/v2/health/ready

InferenceService predictors implementing the V2 Inference Protocol expose:

• GET /v2/health/live — liveness check
• GET /v2/health/ready — readiness check (returns 200 only when model is loaded)

# Verify via the external endpoint
curl https://your-model.yourdomain.com/v2/health/ready
# Expected: 200 OK with {"ready": true}

Step 2: Expose Health Endpoints via Ingress

KServe typically creates its own Ingress via the knative-serving gateway or a custom ClusterServingRuntime. To monitor from outside the cluster, ensure your Ingress controller routes health paths:

# kserve-monitoring-ingress.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: kserve-controller-health
  namespace: kserve
spec:
  selector:
    control-plane: kserve-controller-manager
  ports:
    - name: health
      port: 8081
      targetPort: 8081
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kserve-health
  namespace: kserve
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /healthz
spec:
  rules:
    - host: kserve-health.internal.yourdomain.com
      http:
        paths:
          - path: /controller/healthz
            pathType: Prefix
            backend:
              service:
                name: kserve-controller-health
                port:
                  number: 8081

For InferenceService endpoints, KServe creates these automatically if you configure your gateway correctly:

kubectl apply -f kserve-monitoring-ingress.yaml
curl https://kserve-health.internal.yourdomain.com/controller/healthz
# Expected: 200 OK

Step 3: Add Vigilmon HTTP Monitors

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Configure the following monitors:

| Monitor Name | URL | Priority | |---|---|---| | kserve controller health | https://kserve-health.yourdomain.com/controller/healthz | High | | kserve model ready <model-name> | https://<model-name>.yourdomain.com/v2/health/ready | Critical | | kserve model live <model-name> | https://<model-name>.yourdomain.com/v2/health/live | Critical |

Settings for all monitors:

• Check interval: 1 minute
• Timeout: 15 seconds (model servers can be slow to respond under load)
• Expected status code: 200
• Alert after: 2 consecutive failures

Add one set of monitors per production InferenceService. Model serving endpoints are customer-facing — treat them with the same priority as your API tier.

Step 4: Monitor Model Loading and Canary Health

Beyond basic health checks, verify that model loading completes successfully after deployments. KServe exposes model-level readiness via the V2 protocol:

# Check model metadata (indicates model is loaded)
curl https://your-model.yourdomain.com/v2/models/your-model-name
# Expected: 200 with model metadata JSON

# Check canary predictor separately if using traffic split
kubectl get inferenceservice your-model -o jsonpath='{.status.components}'

Create a Vigilmon monitor that checks the model metadata endpoint:

# In Vigilmon dashboard:
# Monitor Name: kserve model metadata <model-name>
# URL: https://your-model.yourdomain.com/v2/models/your-model-name
# Method: GET
# Expected status: 200
# Keyword check: "name" (asserts the metadata JSON contains a model name field)

For SSL certificate monitoring, enable it on all InferenceService domains — KServe serves all inference traffic over HTTPS, and an expired cert blocks every prediction request.

Step 5: Heartbeat Monitor for Active Inference Verification

A healthy /v2/health/ready endpoint doesn't guarantee inference requests succeed. Verify end-to-end model serving with a synthetic prediction CronJob:

# kserve-inference-heartbeat.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kserve-inference-heartbeat
  namespace: kserve
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: heartbeat
              image: curlimages/curl:latest
              env:
                - name: MODEL_URL
                  value: "https://your-model.yourdomain.com"
                - name: MODEL_NAME
                  value: "your-model-name"
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: kserve-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Send a minimal inference request
                  RESPONSE=$(curl -sf -X POST \
                    "${MODEL_URL}/v2/models/${MODEL_NAME}/infer" \
                    -H "Content-Type: application/json" \
                    -d '{"inputs": [{"name": "input", "shape": [1, 4], "datatype": "FP32", "data": [1.0, 2.0, 3.0, 4.0]}]}')
                  if [ -z "$RESPONSE" ]; then
                    echo "FAIL: No inference response"
                    exit 1
                  fi
                  echo "Inference successful, sending heartbeat"
                  curl -sf "$HEARTBEAT_URL"

Store the heartbeat URL:

kubectl create secret generic vigilmon-secrets \
  --from-literal=kserve-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  -n kserve

In Vigilmon, create a Heartbeat monitor:

• Name: kserve active inference <model-name>
• Expected interval: 5 minutes
• Grace period: 3 minutes

Adjust the inference payload to match your model's actual input schema.

Step 6: Monitor KServe's Knative Dependencies

KServe on serverless mode depends on Knative Serving. A Knative control plane failure cascades into KServe. Monitor the Knative activator, which handles cold-start traffic:

# Check Knative activator health
kubectl exec -n knative-serving deploy/activator -- \
  wget -qO- http://localhost:8012/healthz

Expose and monitor:

# In Vigilmon: add a monitor for the Knative activator
# URL: https://knative-health.yourdomain.com/activator/healthz
# Priority: High (dependency of KServe)

If running KServe in raw deployment mode (without Knative), skip this step — the dependency doesn't apply.

Step 7: Configure Alert Routing

Route alerts by impact:

Critical — Model inference endpoint down (page on-call):

1. Alert Channels → Add Channel → PagerDuty
2. Assign to: kserve model ready <model-name>, kserve active inference <model-name> heartbeat

High — Controller or infrastructure dependency failing (notify ML platform Slack):

1. Alert Channels → Add Channel → Slack Webhook
2. Paste your #ml-platform-alerts webhook URL
3. Assign to: kserve controller health, Knative activator monitors

This mirrors operational reality: a model serving outage is a production incident; a controller crash is a deployment blocker that needs the ML platform team.

What You're Now Monitoring

| Component | Monitor Type | Failure Detected | |---|---|---| | KServe controller manager | HTTP health | Controller crash blocks new model deployments | | InferenceService predictor liveness | HTTP live | Pod crash or OOMKill during traffic | | InferenceService predictor readiness | HTTP ready | Model not loaded, serving 503s | | Model metadata endpoint | HTTP + keyword | Model failed to load after deployment | | Active inference (synthetic) | Heartbeat | End-to-end prediction pipeline broken | | Knative activator (serverless mode) | HTTP health | Cold-start traffic routing failure |

Conclusion

KServe sits at the critical intersection of ML and production reliability — a failure doesn't just break an API, it silently degrades model-driven features across your product. Vigilmon gives your ML platform and SRE teams the external visibility to catch KServe problems within minutes, from the same perspective as the services and users making inference requests.

Start monitoring KServe for free at vigilmon.online. Your first monitor is running in under two minutes.