How to Monitor Apache Submarine with Vigilmon (Uptime, APIs, and Alerts)

Apache Submarine is a unified machine learning platform designed for running distributed ML workloads on Kubernetes and YARN. It provides a consistent environment for training, serving, and experimenting with ML models across cloud and on-premise infrastructure — handling job scheduling, experiment tracking, notebook servers, and model serving from a single control plane.

When your Submarine server goes down, data scientists lose access to notebooks, training jobs stall without error, and model-serving endpoints return 503s to production applications. This tutorial shows you how to set up comprehensive uptime monitoring for Apache Submarine using Vigilmon — free tier, no credit card.

Why Submarine deployments need external monitoring

Submarine coordinates multiple components that can fail independently:

• API server outages — the Submarine server REST API is the single control plane for jobs, notebooks, and models; losing it silently breaks all automation
• Model serving endpoint failures — a served model can crash or OOM-kill without Submarine's scheduler noticing; external probes catch it before users do
• Notebook server stalls — Jupyter notebook servers managed by Submarine can enter unresponsive states while appearing "Running" in the Submarine UI
• Storage backend failures — Submarine depends on MinIO or S3 for artifact storage; losing it causes training jobs to fail at checkpointing, not at start

External HTTP and TCP monitoring from Vigilmon provides the independent layer of visibility that catches these failures.

What you'll need

• A running Apache Submarine deployment (we'll use the default API port 8080 and model server port 8000)
• A free Vigilmon account (sign up takes 30 seconds)

Step 1: Verify your Submarine health endpoint

Apache Submarine exposes a REST API with built-in health and liveness routes. Confirm these are accessible:

# Check Submarine server health
curl http://submarine.example.com:8080/api/v1/

# Should return something like:
# {"status":"OK","code":200,"result":{"version":"0.8.0",...}}

For Kubernetes deployments, ensure the service is exposed outside the cluster:

apiVersion: v1
kind: Service
metadata:
  name: submarine-server
  namespace: submarine
spec:
  type: LoadBalancer
  selector:
    app: submarine-server
  ports:
    - name: http
      port: 8080
      targetPort: 8080

If you've deployed a custom model serving layer, add a /health route:

# FastAPI model server wrapping a Submarine-managed model
from fastapi import FastAPI
import time

app = FastAPI()

@app.get("/health")
async def health():
    return {"status": "ok", "timestamp": time.time()}

@app.get("/predict")
async def predict(data: dict):
    # inference logic here
    return {"prediction": model.infer(data)}

Step 2: Set up HTTP monitoring for the Submarine API

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS as the type
3. URL: https://submarine.example.com/api/v1/
4. Set check interval to 1 minute
5. Under Expected response, set:
   • Status code: 200
   • Response body contains: "status":"OK"
6. Save the monitor

Vigilmon probes this endpoint from multiple regions every minute. A non-200 response or timeout triggers an incident immediately.

Step 3: Monitor served model endpoints

Training and serving are separate concerns in Submarine. Your API server can be healthy while every served model is down. Add a monitor for each critical model endpoint:

1. Go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: https://submarine.example.com/model/your-model/health (adjust to your serving path)
4. Expected status: 200
5. Response time threshold: 3 seconds — slow inference responses indicate resource contention before full failure

Repeat this for each production model you're serving. It takes about 30 seconds per model to add.

Step 4: Add TCP port monitoring for Submarine components

Submarine relies on several backend services that speak non-HTTP protocols. Monitor them at the TCP layer to catch port-binding failures and network-level outages:

For the Submarine database (MySQL by default):

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Host: submarine-db.example.com, Port: 3306
4. Save the monitor

For MinIO object storage:

1. Repeat for host minio.example.com, port 9000

These TCP checks ensure the full Submarine stack is verified, not just the API surface.

Step 5: Configure webhook alerts

ML job outages have cascading effects — a training job that silently stalls wastes expensive GPU hours and delays model updates. Get alerted immediately.

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your Slack, Teams, or PagerDuty webhook URL
3. Assign the alert channel to all your Submarine monitors

Example alert payload from Vigilmon:

{
  "monitor_name": "Submarine API /api/v1/",
  "status": "down",
  "url": "https://submarine.example.com/api/v1/",
  "started_at": "2026-03-15T09:12:00Z",
  "duration_seconds": 120
}

Add an email channel as well for escalations that need a paper trail or reach on-call engineers outside of Slack.

Step 6: Set up a status page for your ML platform

If multiple teams submit jobs to Submarine, give them a shared status page instead of having them file support tickets when jobs stop running:

1. Go to Status Pages → New Status Page
2. Name it (e.g. "ML Platform Status")
3. Add your monitors — Submarine API, model server, MySQL TCP, MinIO TCP
4. Publish the page

You get a public URL at https://status.vigilmon.online/your-page with real-time component status. Pin it in your team's Slack channel or internal wiki.

Monitor coverage summary

| Monitor | Type | What it catches | |---------|------|-----------------| | https://submarine.example.com/api/v1/ | HTTP | Submarine API server crash | | https://submarine.example.com/model/*/health | HTTP | Served model outages, slow inference | | submarine-db.example.com:3306 | TCP | Database port loss (blocks all operations) | | minio.example.com:9000 | TCP | Object storage loss (blocks checkpointing) |

What's next

• Heartbeat monitors — wrap your recurring training jobs to ping a Vigilmon heartbeat URL on completion; if the heartbeat stops arriving, you know the job silently died
• SSL certificate monitoring — Vigilmon tracks your TLS certificate expiry and alerts you before it causes client errors in your model-serving APIs

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.