How to Monitor Kubeflow Pipelines with Vigilmon

Kubeflow is the machine learning platform built on Kubernetes — providing a suite of tools for managing ML workflows, serving models, and running experiments at scale. When Kubeflow components fail, model training stops, pipeline runs hang indefinitely, and model serving endpoints go dark without any immediate alert.

In this tutorial you'll learn how to set up reliable external monitoring for Kubeflow using Vigilmon, so you catch failures before your data scientists do.

Why Kubeflow needs external monitoring

Kubeflow has a complex, multi-component architecture with several distinct failure patterns:

• Pipeline backend failures — the Kubeflow Pipelines API server (port 8888) can fail independently of the rest of the Kubernetes cluster; pipeline runs stop scheduling but existing pods keep running
• ML Metadata (MLMD) disconnects — if the MLMD gRPC service goes down, pipeline steps cannot log artifacts or metrics, causing runs to hang at transition points
• Namespace-scoped profile failures — Kubeflow profiles manage per-team namespaces; a profile controller failure prevents new namespace creation but existing workloads are unaffected
• Central dashboard 502s — the Istio ingress gateway fronting the Kubeflow central dashboard frequently returns 502 during upgrades or network policy changes, leaving users locked out
• Katib experiment stalls — the hyperparameter tuning controller can stop processing experiments silently when etcd has high latency

Internal Kubernetes liveness probes only restart individual pods. External monitoring is what tells you when the user-visible layer has failed.

What you'll need

• A Kubeflow installation (v1.7+ recommended)
• The external URL or LoadBalancer IP for your Kubeflow deployment
• A free Vigilmon account

Step 1: Identify Kubeflow's external endpoints

Kubeflow typically exposes a central dashboard via an Istio ingress gateway. Find your external IP or hostname:

# Get the Istio ingress gateway external IP
kubectl get svc -n istio-system istio-ingressgateway \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# Or for hostname-based LoadBalancers (EKS/GKE)
kubectl get svc -n istio-system istio-ingressgateway \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

The Kubeflow Pipelines API is also accessible directly:

# Port-forward for local testing
kubectl port-forward -n kubeflow svc/ml-pipeline 8888:8888

# Test the API health
curl http://localhost:8888/apis/v1beta1/healthz
# Returns: {"status":"ok"}

For external access, the Pipelines API is typically routed through the ingress gateway at /pipeline/.

Step 2: Add health check endpoints to your pipeline components

Kubeflow Pipelines exposes a health endpoint, but it's useful to add a lightweight check that verifies end-to-end connectivity:

# kfp_health_check.py - deploy as a separate service or sidecar
from http.server import HTTPServer, BaseHTTPRequestHandler
import kfp
import json
import os

KFP_ENDPOINT = os.getenv("KFP_ENDPOINT", "http://ml-pipeline.kubeflow:8888")

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return

        try:
            client = kfp.Client(host=KFP_ENDPOINT)
            # List pipelines — exercises the full API + MLMD stack
            pipelines = client.list_pipelines(page_size=1)
            
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({
                "status": "ok",
                "pipeline_count": pipelines.total_size
            }).encode())

        except Exception as e:
            self.send_response(503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({
                "status": "error",
                "detail": str(e)
            }).encode())

    def log_message(self, format, *args):
        pass

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8080), HealthHandler)
    print("Kubeflow health check running on :8080")
    server.serve_forever()

Deploy this as a Kubernetes Deployment with a Service to expose port 8080:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kfp-healthcheck
  namespace: kubeflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kfp-healthcheck
  template:
    metadata:
      labels:
        app: kfp-healthcheck
    spec:
      containers:
      - name: healthcheck
        image: python:3.11-slim
        command: ["python", "/app/kfp_health_check.py"]
        env:
        - name: KFP_ENDPOINT
          value: "http://ml-pipeline.kubeflow:8888"
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 60
---
apiVersion: v1
kind: Service
metadata:
  name: kfp-healthcheck
  namespace: kubeflow
spec:
  selector:
    app: kfp-healthcheck
  ports:
  - port: 8080
    targetPort: 8080
  type: LoadBalancer

Step 3: Monitor the Kubeflow central dashboard

The central dashboard is the primary user-visible surface. Set up an HTTP monitor for it:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to https://your-kubeflow-domain.example.com
4. Check interval: 1 minute
5. Expected response:
   • Status code: 200 (or 302 if the dashboard redirects to a login page)
   • (Optional) Response body contains: Kubeflow
6. Save the monitor

This catches Istio ingress failures, certificate issues, and full platform outages.

Step 4: Monitor the Pipelines API directly

Add a separate monitor for the Pipelines API health endpoint:

1. Go to Monitors → New Monitor → HTTP / HTTPS
2. URL: https://your-kubeflow-domain.example.com/pipeline/apis/v1beta1/healthz
3. Expected response:
   • Status code: 200
   • Body contains: "status":"ok"
4. Save the monitor

This is more specific than the dashboard monitor — the ingress can be up while the Pipelines API backend is crashed.

Step 5: Set up TCP port monitoring

Add TCP checks for the core service ports to catch network-level failures:

| Service | Port | What it monitors | |---------|------|-----------------| | Istio ingress (HTTP) | 80 | Ingress gateway reachability | | Istio ingress (HTTPS) | 443 | TLS termination | | KFP API (direct) | 8888 | Pipeline API port (if exposed) | | Jupyter gateway | 8080 | Notebook spawning |

For each:

1. Monitors → New Monitor → TCP Port
2. Enter hostname and port
3. Save

Step 6: Set up heartbeat monitors for pipeline runs

Kubeflow Pipelines can run scheduled recurring pipelines (using KFP's recurring run feature or an external scheduler). Vigilmon's heartbeat monitors let you verify that scheduled ML jobs actually complete.

Add a pipeline step that pings a Vigilmon heartbeat URL after each successful run:

# In your KFP pipeline component
import kfp
from kfp import dsl
import requests
import os

@dsl.component(
    base_image="python:3.11-slim",
    packages_to_install=["requests"]
)
def notify_completion(vigilmon_heartbeat_url: str):
    """Ping Vigilmon heartbeat to confirm pipeline completed."""
    import requests
    try:
        response = requests.get(vigilmon_heartbeat_url, timeout=10)
        print(f"Heartbeat sent: {response.status_code}")
    except Exception as e:
        print(f"Heartbeat failed: {e}")
        # Don't raise — monitoring failure shouldn't fail the pipeline

@dsl.pipeline(name="training-pipeline")
def training_pipeline(
    vigilmon_heartbeat_url: str = "https://api.vigilmon.online/beat/YOUR_HEARTBEAT_ID"
):
    # ... your training steps ...
    
    notify = notify_completion(vigilmon_heartbeat_url=vigilmon_heartbeat_url)
    notify.after(your_last_training_step)

In Vigilmon, create the heartbeat monitor:

1. Monitors → New Monitor → Heartbeat
2. Set the expected ping interval to match your pipeline schedule (e.g., every 24 hours)
3. Copy the heartbeat URL and paste it into your pipeline configuration
4. Save

If the pipeline fails mid-run or stops scheduling entirely, Vigilmon alerts you when the heartbeat goes missing.

Step 7: Configure alerts for the ML team

ML infrastructure failures usually affect multiple teams simultaneously. Configure alert routing to reach the right people:

1. Go to Alert Channels → Add Channel
2. Set up Slack with your #ml-infra-alerts channel webhook
3. Set up Email with on-call distribution list
4. Assign channels to monitors with priority routing:
   • Pipelines API down → immediate Slack + email
   • Dashboard down → Slack only
   • Heartbeat missed → email after 15-minute grace period

Complete monitoring coverage

| Monitor | Type | What it catches | |---------|------|-----------------| | Central dashboard | HTTP | Platform-wide outage, Istio failures | | Pipelines API /healthz | HTTP | KFP backend crash, MLMD disconnect | | Custom health wrapper | HTTP | End-to-end API connectivity | | Istio ingress :443 | TCP | TLS/network-level failures | | Pipeline heartbeat | Heartbeat | Scheduled run failures |

What's next

• SSL certificate monitoring — Vigilmon checks your Kubeflow TLS certificate expiry automatically, preventing login failures when certificates lapse
• Multi-region monitoring — verify that your Kubeflow cluster is reachable from multiple geographic locations, important for globally distributed teams
• Status page — create an AI Platform status page in Vigilmon so your data scientists can self-diagnose before raising infra tickets

Monitor your Kubeflow platform for free at vigilmon.online — no credit card, running in under a minute.