tutorial

How to Monitor Apache SystemDS Uptime and Health with Vigilmon

Apache SystemDS is a declarative machine learning system for large-scale data — but long-running ML jobs, REST API failures, and backend resource exhaustion can stall your entire ML pipeline silently. Here's how to monitor SystemDS with Vigilmon.

Apache SystemDS (formerly SystemML) is a declarative machine learning system designed to run large-scale ML algorithms over data stored in HDFS, Apache Spark, or local filesystems. Data scientists write algorithms in DML (Declarative Machine Learning language) or using the Python/R/Java APIs, and SystemDS compiles and optimizes their execution plans for the available compute environment. SystemDS can run as a REST API server (MLContext REST), accepting ML job submissions and returning results asynchronously. When that API server becomes unavailable, when submitted jobs silently time out, or when Spark executor failures cause jobs to hang indefinitely, your ML pipeline stalls without a clear error visible to operators.

Vigilmon gives you external visibility into SystemDS availability through HTTP probe monitoring and heartbeat monitoring for ML job completion. This tutorial walks through both.

Why Apache SystemDS Monitoring Matters

SystemDS introduces failure modes specific to long-running ML computation that traditional uptime monitoring misses:

  • REST API unavailability blocks all ML job submissions — pipelines that depend on SystemDS for model training or inference silently stop receiving results
  • Spark executor exhaustion causes submitted jobs to queue indefinitely — the API accepts new submissions but no job ever completes
  • HDFS connectivity loss causes DML scripts that read training data to fail with obscure I/O errors — jobs appear to start but never produce output
  • Out-of-memory kills on the SystemDS driver JVM cause the process to restart silently — in-flight jobs are lost and callers receive no response
  • Long-running jobs masking hangs — a 6-hour training job and a hung job look identical from the outside; only a failure to complete within expected time reveals the hang

External monitoring through Vigilmon verifies whether SystemDS is accepting requests and completing work — not just whether the JVM process is alive.

Step 1: Build a SystemDS Health Endpoint

SystemDS can run as a REST API server. The default port is 8080. Test whether it is accepting requests:

curl -i http://your-systemds-host:8080/api/v1/dml
# With a trivial DML script to verify execution

For a robust health endpoint that verifies end-to-end DML execution, build a sidecar that submits a minimal DML probe script:

Node.js Sidecar Health Endpoint

// systemds-health.js — verifies SystemDS DML execution end-to-end
const express = require('express');
const axios = require('axios');
const FormData = require('form-data');

const app = express();
const SDS_URL = process.env.SYSTEMDS_URL || 'http://localhost:8080';

// Minimal probe DML script — computes a scalar, returns immediately
const PROBE_DML = `
x = 1 + 1;
print(x);
`;

app.get('/health/systemds', async (req, res) => {
  try {
    const form = new FormData();
    form.append('scriptContents', PROBE_DML);
    form.append('outputVariables', JSON.stringify([]));

    const response = await axios.post(`${SDS_URL}/api/v1/dml`, form, {
      headers: form.getHeaders(),
      timeout: 30000,
    });

    if (response.status === 200) {
      return res.status(200).json({ status: 'ok', dml: 'executed' });
    }

    return res.status(503).json({ status: 'degraded', code: response.status });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Python Sidecar Health Endpoint

# systemds_health.py — FastAPI sidecar for SystemDS health
import os
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
SYSTEMDS_URL = os.environ.get("SYSTEMDS_URL", "http://localhost:8080")

# Minimal probe DML — trivial scalar computation, completes in milliseconds
PROBE_DML = "x = 1 + 1; print(x);"

@app.get("/health/systemds")
async def systemds_health():
    async with httpx.AsyncClient(timeout=30.0) as client:
        # First check: can we reach the API at all?
        try:
            r = await client.get(f"{SYSTEMDS_URL}/api/v1/")
            if r.status_code >= 500:
                return JSONResponse(status_code=503,
                                    content={"status": "down", "check": "api_root",
                                             "code": r.status_code})
        except Exception as e:
            return JSONResponse(status_code=503,
                                content={"status": "down", "check": "api_root", "error": str(e)})

        # Second check: submit a minimal DML script and verify execution
        try:
            r = await client.post(
                f"{SYSTEMDS_URL}/api/v1/dml",
                data={
                    "scriptContents": PROBE_DML,
                    "outputVariables": "[]",
                },
                timeout=30.0,
            )
            if r.status_code != 200:
                return JSONResponse(status_code=503, content={
                    "status": "degraded",
                    "check": "dml_execution",
                    "code": r.status_code,
                })
        except Exception as e:
            return JSONResponse(status_code=503,
                                content={"status": "down", "check": "dml_execution", "error": str(e)})

    return {"status": "ok", "dml": "executed"}

Deploy and verify:

curl -i https://your-app.example.com/health/systemds
# HTTP/1.1 200 OK
# {"status":"ok","dml":"executed"}

Step 2: Configure a Vigilmon HTTP Monitor for SystemDS

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-app.example.com/health/systemds
  4. Set the check interval to 2 minutes (DML execution probe takes a few seconds)
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 30000ms (allow for JVM warm-up and trivial DML execution)
  6. Under Alert channels, assign your Slack or email channel
  7. Save the monitor

For a lighter-weight check against the raw SystemDS API endpoint:

  1. Create a second monitor pointing to http://your-systemds-host:8080/api/v1/
  2. Set the expected status code to 200
  3. Set the response time threshold to 5000ms

Failure Coverage

| Failure Mode | Process Monitor | Vigilmon | |---|---|---| | SystemDS JVM crash | ✓ | ✓ | | REST API accepts requests but DML fails | ✗ | ✓ | | Spark executor exhaustion (jobs queue forever) | ✗ | ✓ (via heartbeat) | | HDFS connectivity loss (training jobs stall) | ✗ | ✓ (via heartbeat) | | OOM kill and restart (in-flight jobs lost) | ✗ | ✓ |

Step 3: Heartbeat Monitoring for ML Job Completion

Long-running SystemDS training jobs are the best candidates for heartbeat monitoring. When a Spark executor becomes unavailable mid-training or HDFS data reads time out, jobs hang indefinitely — no error is raised, no result is returned, and the API continues to accept new submissions. Vigilmon's heartbeat monitor detects this silent hang.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: systemds-ml-pipeline
  3. Set the expected interval to match your ML job cadence (e.g., 30 minutes for a regular training pipeline)
  4. Set the grace period: 1 hour (allow for legitimate long runs)
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your SystemDS DML Script

Call an external URL from within DML using SystemDS's externalFunction or wrap the job submission in a Python script:

# systemds_ml_job.py — submits a training job and pings Vigilmon on success
import os
import requests
import systemds  # Apache SystemDS Python API

HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]
TRAINING_DATA = os.environ.get("TRAINING_DATA_PATH", "/data/training.csv")

def run_training():
    from systemds.context import SystemDSContext

    with SystemDSContext() as sds:
        X = sds.read(TRAINING_DATA)
        # Run logistic regression (example)
        model = sds.functions.logReg(X=X[:, :-1], Y=X[:, -1:], maxii=100)
        model.save("/models/logistic_output")

    # Only ping Vigilmon after the job completes successfully
    requests.get(HEARTBEAT_URL, timeout=5)
    print("Training complete. Vigilmon pinged.")

if __name__ == "__main__":
    run_training()

Java API Integration

// SystemDsJobRunner.java — runs SystemDS job and pings Vigilmon on completion
import org.apache.sysds.api.mlcontext.*;
import java.net.HttpURLConnection;
import java.net.URL;

public class SystemDsJobRunner {
    private static final String HEARTBEAT_URL = System.getenv("VIGILMON_HEARTBEAT_URL");

    public static void runJob(String dmlScript) throws Exception {
        SparkConf conf = new SparkConf().setAppName("SystemDS-Job");
        JavaSparkContext sc = new JavaSparkContext(conf);
        MLContext ml = new MLContext(sc);

        Script script = ScriptFactory.dmlFromString(dmlScript);
        ml.execute(script);

        // Ping Vigilmon after successful completion
        pingVigilmon();
        sc.stop();
    }

    private static void pingVigilmon() {
        if (HEARTBEAT_URL == null) return;
        try {
            HttpURLConnection conn = (HttpURLConnection) new URL(HEARTBEAT_URL).openConnection();
            conn.setRequestMethod("GET");
            conn.setConnectTimeout(3000);
            conn.setReadTimeout(3000);
            conn.getResponseCode();
            conn.disconnect();
        } catch (Exception ignored) {}
    }
}

Step 4: Monitor Multiple SystemDS Deployment Components

A production SystemDS deployment spans multiple services. Monitor each independently:

[systemds-api]       :8080/api/v1/
[systemds-health]    /health/systemds (sidecar)
[systemds-spark]     Spark master UI :8080 (separate instance)
[systemds-hdfs]      HDFS NameNode :9870
[systemds-pipeline]  (heartbeat monitor)

Name monitors using the bracket convention so alert notifications immediately identify the failing component.

Group all SystemDS monitors into a Vigilmon Status Page for your data science and MLOps teams.

Step 5: Alert Routing for SystemDS Outages

SystemDS outages block model training and batch inference pipelines — data science teams cannot produce new models, and production ML services that depend on freshly trained models begin serving stale predictions.

Configure alert channels in Vigilmon:

  1. SystemDS API monitor → Slack + PagerDuty (P1 — ML platform unavailable)
  2. DML execution health sidecar → Slack + PagerDuty (P1 — execution engine broken)
  3. ML pipeline heartbeat → Slack + PagerDuty (P1 — training jobs not completing)
  4. Spark/HDFS infrastructure monitors → Slack (P2 — compute/storage degraded)

Set response time thresholds carefully:

  • Alert at 25000ms on the DML execution health sidecar — a trivial DML script taking more than 25 seconds indicates JVM startup failure or Spark scheduler congestion
  • Alert at 3000ms on the raw API endpoint — the REST API root should respond instantly; slowness here signals GC pressure or network issues

Summary

Apache SystemDS failures are silent and expensive — long-running ML jobs hang without errors, training pipelines stop producing models, and the REST API may appear available while no DML script can execute to completion. Vigilmon monitors the full execution path.

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /api/v1/ | API reachability | | HTTP monitor on /health/systemds | End-to-end DML execution | | Heartbeat monitor | ML job completion liveness | | Infrastructure monitors (Spark, HDFS) | Compute and storage availability |

Get started free at vigilmon.online — your first SystemDS monitor takes under two minutes to configure.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles