How to Monitor Metaflow Workflows with Vigilmon

Metaflow is Netflix's open-source Python framework for building and managing real-life ML workflows. It handles everything from local experimentation to cloud-scale training runs on AWS Batch, Kubernetes, and beyond. But when a production Metaflow deployment fails — whether the metadata service goes down or a scheduled flow stops running — the silence is deafening. No alerts, no notifications, just missing model updates and stale predictions.

In this tutorial you'll set up robust monitoring for Metaflow using Vigilmon to catch infrastructure failures and silent workflow stoppages before they affect your ML applications.

Why Metaflow deployments need external monitoring

Metaflow's failure modes are particularly hard to detect because the framework is designed to be transparent:

• Metadata service outages — Metaflow's metadata service (typically hosted as a REST API) tracks flow runs, steps, and artifacts. When it goes down, new flows fail immediately but existing data is safe — you just have no visibility into what's running.
• Silent flow scheduling failures — if you use Argo Workflows, AWS Step Functions, or a custom cron to schedule flows, a scheduler failure means flows simply stop running with no error surfaced in Metaflow itself.
• AWS Batch queue saturation — jobs can get stuck in RUNNABLE state indefinitely when Batch compute environments are misconfigured or have capacity issues.
• S3 datastore disconnects — Metaflow stores all artifacts in S3 (or a local store). A broken S3 connection causes all artifact read/write operations to fail silently until they hit a timeout.
• Conda environment build failures — flows using @conda or @pypi decorators can fail during the environment setup phase, which looks like a hung step rather than an immediate error.

What you'll need

• A Metaflow deployment with the metadata service running
• A free Vigilmon account

Step 1: Identify your Metaflow service endpoints

Metaflow's hosted metadata service exposes an HTTP API. Find your endpoint:

# Check current Metaflow configuration
python -c "from metaflow.metaflow_config import METADATA_SERVICE_URL; print(METADATA_SERVICE_URL)"

# Or inspect the config file
cat ~/.metaflowconfig | python -m json.tool

The metadata service typically runs at a URL like https://metaflow-metadata.your-domain.com. It exposes:

# Health check endpoint
curl https://metaflow-metadata.your-domain.com/ping
# Returns: "pong"

# Service info
curl https://metaflow-metadata.your-domain.com/metadata/version

For the open-source self-hosted version, the default port is 8080:

curl http://localhost:8080/ping

Step 2: Deploy the Metaflow metadata service with health endpoints

If you're running the metadata service yourself, ensure it's deployed with proper health exposure:

# metaflow-metadata-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metaflow-metadata
  namespace: metaflow
spec:
  replicas: 2
  selector:
    matchLabels:
      app: metaflow-metadata
  template:
    metadata:
      labels:
        app: metaflow-metadata
    spec:
      containers:
      - name: metadata
        image: netflixoss/metaflow_metadata_service:latest
        env:
        - name: MF_METADATA_DB_HOST
          valueFrom:
            secretKeyRef:
              name: metaflow-db-secret
              key: host
        - name: MF_METADATA_DB_PORT
          value: "5432"
        - name: MF_METADATA_DB_USER
          valueFrom:
            secretKeyRef:
              name: metaflow-db-secret
              key: username
        - name: MF_METADATA_DB_PSWD
          valueFrom:
            secretKeyRef:
              name: metaflow-db-secret
              key: password
        - name: MF_METADATA_DB_NAME
          value: "metaflow"
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /ping
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /ping
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: metaflow-metadata
  namespace: metaflow
spec:
  selector:
    app: metaflow-metadata
  ports:
  - port: 8080
    targetPort: 8080
  type: LoadBalancer

Step 3: Add heartbeat pings to your Metaflow flows

The most important thing to monitor in a Metaflow deployment isn't the infrastructure — it's whether your flows actually complete. Use Vigilmon's heartbeat monitors to track flow execution:

# flows/training_flow.py
from metaflow import FlowSpec, step, Parameter
import requests
import os

VIGILMON_HEARTBEAT_URL = os.getenv(
    "VIGILMON_HEARTBEAT_URL",
    "https://api.vigilmon.online/beat/YOUR_HEARTBEAT_ID"
)

class TrainingFlow(FlowSpec):
    
    learning_rate = Parameter("learning_rate", default=0.001)
    epochs = Parameter("epochs", default=10)
    
    @step
    def start(self):
        print("Starting training flow")
        self.next(self.preprocess)
    
    @step
    def preprocess(self):
        # Your data preprocessing logic
        self.next(self.train)
    
    @step
    def train(self):
        # Your model training logic
        self.next(self.evaluate)
    
    @step
    def evaluate(self):
        # Your model evaluation logic
        self.next(self.end)
    
    @step
    def end(self):
        """Final step — ping Vigilmon to confirm successful completion."""
        try:
            response = requests.get(VIGILMON_HEARTBEAT_URL, timeout=10)
            print(f"Vigilmon heartbeat sent: HTTP {response.status_code}")
        except Exception as e:
            # Log but don't fail — monitoring shouldn't affect the flow
            print(f"Warning: Vigilmon heartbeat failed: {e}")

if __name__ == "__main__":
    TrainingFlow()

For flows deployed on Argo Workflows or AWS Step Functions, add the heartbeat ping as the final step in the same pattern — it runs only when the full flow succeeds.

Step 4: Set up Vigilmon heartbeat monitors for each flow

For each critical flow you want to track:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose Heartbeat as the type
3. Name it clearly, e.g. "Daily Model Training Flow"
4. Set the expected ping period to match your flow schedule:
   • Hourly flow → 65-minute grace period
   • Daily flow → 25-hour grace period
   • Weekly flow → 8-day grace period
5. Copy the heartbeat URL and add it to your flow as VIGILMON_HEARTBEAT_URL
6. Save the monitor

Vigilmon will alert you if the heartbeat isn't received within the grace period, which means either the flow failed, the scheduler stopped, or the infrastructure underneath collapsed.

Step 5: Monitor the metadata service via HTTP

1. Go to Monitors → New Monitor → HTTP / HTTPS
2. URL: https://metaflow-metadata.your-domain.com/ping
3. Check interval: 1 minute
4. Expected response:
   • Status code: 200
   • Body contains: pong
5. Save the monitor

Add a second HTTP monitor for the service info endpoint:

• URL: https://metaflow-metadata.your-domain.com/metadata/version
• Expected status: 200

Step 6: Monitor upstream infrastructure

Metaflow workflows depend on AWS or cloud resources that can fail independently. Add monitors for:

PostgreSQL metadata database (if self-hosted):

# TCP check to confirm DB is reachable
# Add TCP monitor: your-db-host:5432

In Vigilmon: Monitors → New Monitor → TCP Port → your-db-host:5432

Argo Workflows UI (if using Argo for scheduling):

• HTTP monitor: https://argo.your-domain.com/
• Expected: 200, body contains Argo

AWS Batch queue monitoring: While Vigilmon can't directly query AWS Batch, you can write a lightweight healthcheck Lambda that checks queue depths and pings a heartbeat:

# lambda_metaflow_monitor.py
import boto3
import requests
import os

VIGILMON_HEARTBEAT = os.environ["VIGILMON_HEARTBEAT_URL"]
BATCH_QUEUE_NAME = os.environ["BATCH_QUEUE_NAME"]

def handler(event, context):
    batch = boto3.client("batch")
    
    # Check for stuck RUNNABLE jobs (> 30 min)
    response = batch.list_jobs(
        jobQueue=BATCH_QUEUE_NAME,
        jobStatus="RUNNABLE"
    )
    
    runnable_jobs = response.get("jobSummaryList", [])
    
    if not runnable_jobs:
        # Queue is healthy — ping heartbeat
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
        return {"status": "healthy", "runnable_jobs": 0}
    
    # Check if any job has been RUNNABLE for more than 30 minutes
    import time
    now = int(time.time() * 1000)
    stuck = [
        j for j in runnable_jobs
        if now - j.get("createdAt", now) > 30 * 60 * 1000
    ]
    
    if not stuck:
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
        return {"status": "healthy", "runnable_jobs": len(runnable_jobs)}
    
    # Don't ping — trigger heartbeat alert
    return {"status": "stuck_jobs", "count": len(stuck)}

Schedule this Lambda to run every 10 minutes with a Vigilmon heartbeat set to a 15-minute grace period.

Step 7: Configure alert routing

1. Go to Alert Channels → Add Channel
2. Add Slack with your #ml-ops webhook URL
3. Add Email for your on-call distribution list
4. Assign channels:
   • Metadata service down → Slack + email (immediate)
   • Flow heartbeat missed → Slack + email after grace period
   • Database TCP check → email only

Complete monitoring setup

| Monitor | Type | Detects | |---------|------|---------| | Metadata service /ping | HTTP | Service crashes, DB disconnects | | Metadata service /metadata/version | HTTP | API degradation | | Daily training flow | Heartbeat | Flow failures, scheduler outages | | Weekly retraining flow | Heartbeat | Silent flow stoppage | | PostgreSQL :5432 | TCP | Database reachability | | Argo Workflows UI | HTTP | Scheduler UI availability | | Batch queue (via Lambda) | Heartbeat | Stuck compute jobs |

What's next

• Status page — build a "ML Platform Status" page in Vigilmon that consolidates all your Metaflow infrastructure monitors into a single view for your data science team
• SSL monitoring — Vigilmon automatically tracks certificate expiry for your metadata service HTTPS endpoint
• Response time history — track metadata service latency over time to catch database performance degradation before it causes flow timeouts

Get started monitoring your Metaflow deployment for free at vigilmon.online.