tutorial

How to Monitor Scholar with Vigilmon

Monitor your Scholar ML pipelines in Elixir with Vigilmon — detect stalled training runs, data ingestion failures, and model drift before they corrupt your predictions or delay your data science releases.

How to Monitor Scholar with Vigilmon

Scholar is the scientific computing and machine learning library for Elixir, built on top of the Nx tensor library. It provides battle-tested implementations of linear regression, k-means clustering, PCA, naive Bayes classifiers, decision trees, and other ML primitives — letting Elixir data pipelines train, evaluate, and serve models without leaving the BEAM. When a Scholar training job stalls on a large dataset, a feature extraction step silently produces NaN values, or a model serving process crashes between predictions, there is no built-in external alert — only confused downstream consumers wondering why results changed.

Vigilmon adds the external watchdog: heartbeat monitors confirm training pipelines are progressing, HTTP checks verify model-serving endpoints are responding, and immediate alerts fire when a pipeline job misses its expected completion window.

This tutorial covers:

  • Heartbeat monitoring for Scholar training pipelines and batch jobs
  • HTTP monitoring for Elixir model-serving APIs
  • Alerting on data ingestion gaps and training run failures
  • Status pages for data science teams

What to Monitor in a Scholar Deployment

| Layer | What it is | How to monitor | |---|---|---| | Training pipeline | GenServer or Task running Scholar.fit/2 | Heartbeat on training completion | | Batch inference | Scheduled job applying model to new data | Heartbeat per processed batch | | Model serving API | HTTP endpoint returning Scholar predictions | HTTP uptime check | | Data ingestion | Upstream data flowing into feature store | Heartbeat on data receipt |

Step 1: Add heartbeats to training pipelines

Training runs are the most critical signals to instrument. A long-running Scholar pipeline that stalls on a memory pressure spike or a corrupt dataset row simply stops producing output — with no signal unless you built one in.

# lib/my_app/ml/classifier_trainer.ex
defmodule MyApp.ML.ClassifierTrainer do
  use GenServer
  require Logger

  alias Scholar.NaiveBayes.Multinomial
  alias Scholar.Preprocessing

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)

  @impl true
  def init(opts) do
    schedule_training()
    {:ok, %{opts: opts, last_trained_at: nil}}
  end

  @impl true
  def handle_info(:train, state) do
    Logger.info("Starting Scholar classifier training")

    with {:ok, raw} <- MyApp.DataStore.fetch_training_data(),
         {:ok, {x, y}} <- preprocess(raw),
         model <- Multinomial.fit(x, y, num_classes: 5) do
      MyApp.ModelStore.save(:classifier, model)
      ping_vigilmon(:training_heartbeat)
    else
      {:error, reason} ->
        Logger.error("Training failed: #{inspect(reason)}")
        # No heartbeat → Vigilmon alerts after grace period
    end

    schedule_training()
    {:noreply, %{state | last_trained_at: DateTime.utc_now()}}
  end

  defp preprocess(raw_data) do
    x =
      raw_data
      |> Enum.map(& &1.features)
      |> Nx.tensor()
      |> Preprocessing.min_max_scale()

    y = raw_data |> Enum.map(& &1.label) |> Nx.tensor()
    {:ok, {x, y}}
  rescue
    e -> {:error, Exception.message(e)}
  end

  defp schedule_training do
    # Retrain every 6 hours
    Process.send_after(self(), :train, :timer.hours(6))
  end

  defp ping_vigilmon(key) do
    url = Application.get_env(:my_app, [:vigilmon, key])

    if url do
      Task.start(fn ->
        :httpc.request(:get, {String.to_charlist(url), []}, [{:timeout, 5000}], [])
      end)
    end
  end
end

In Vigilmon:

  1. Click New Monitor → Heartbeat.
  2. Name it Scholar classifier training.
  3. Set expected interval to 8 hours (6h training cycle + 2h grace).
  4. Add the heartbeat URL to your app config.

Step 2: Monitor batch inference jobs

Scheduled inference runs — re-scoring a user base, refreshing recommendations — are easy to lose track of once they're in production. A heartbeat ensures you know when the nightly job skips.

# lib/my_app/ml/batch_scorer.ex
defmodule MyApp.ML.BatchScorer do
  use Oban.Worker, queue: :ml_jobs

  require Logger

  alias Scholar.Linear.RidgeRegression

  @impl true
  def perform(%Oban.Job{args: %{"dataset_id" => dataset_id}}) do
    Logger.info("Scoring dataset #{dataset_id}")

    model = MyApp.ModelStore.get(:regression)

    with {:ok, features} <- MyApp.DataStore.fetch_features(dataset_id),
         x <- Nx.tensor(features),
         predictions <- RidgeRegression.predict(model, x),
         :ok <- MyApp.DataStore.save_predictions(dataset_id, Nx.to_list(predictions)) do
      ping_vigilmon()
      :ok
    else
      {:error, reason} ->
        Logger.error("Batch scoring failed for #{dataset_id}: #{inspect(reason)}")
        {:error, reason}
    end
  end

  defp ping_vigilmon do
    url = Application.get_env(:my_app, [:vigilmon, :batch_scoring_heartbeat])
    if url, do: :httpc.request(:get, {String.to_charlist(url), []}, [{:timeout, 5000}], [])
  end
end

In Vigilmon, create a second heartbeat monitor named Scholar batch scoring with interval matching your job schedule (e.g., 25 hours for daily jobs).

Step 3: Expose a model-serving health endpoint

Model-serving APIs built with Phoenix or Plug should expose a health endpoint that validates the model is loaded and the serving process is alive:

# lib/my_app_web/controllers/ml_health_controller.ex
defmodule MyAppWeb.MLHealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    model_ok = MyApp.ModelStore.model_loaded?(:classifier)
    last_trained = MyApp.ModelStore.last_trained_at(:classifier)
    stale = stale?(last_trained)

    status_code = if model_ok and not stale, do: 200, else: 503

    conn
    |> put_status(status_code)
    |> json(%{
      status: if(status_code == 200, do: "ok", else: "degraded"),
      model_loaded: model_ok,
      last_trained_at: last_trained && DateTime.to_iso8601(last_trained),
      stale: stale
    })
  end

  # Alert if the model is more than 25 hours old
  defp stale?(nil), do: true
  defp stale?(dt), do: DateTime.diff(DateTime.utc_now(), dt, :hour) > 25
end

Add the route:

# lib/my_app_web/router.ex
scope "/ml", MyAppWeb do
  pipe_through :api
  get "/health", MLHealthController, :index
end

Point a Vigilmon HTTP monitor at https://your-app.example.com/ml/health. Set keyword match to "ok" so a stale model triggers an alert even if the server returns 200.

Step 4: Configure alerts for ML pipeline failures

| Monitor | Condition | Action | |---|---|---| | Classifier training heartbeat | Missed ≥ 1× | Alert data engineering team | | Batch scoring heartbeat | Missed ≥ 1× | Alert data science on-call | | Model serving HTTP check | Non-200 or timeout | PagerDuty — model down | | Data ingestion heartbeat | Missed ≥ 2× | Alert data engineering |

Set up alerts in Vigilmon → Alerts:

  • Slack (#ml-alerts) for training failures and stale model warnings
  • Email for daily batch scoring results
  • PagerDuty for the model-serving HTTP check — a down serving endpoint means live predictions are failing

Step 5: Status page for data science stakeholders

Data science pipelines typically have non-engineering stakeholders: product managers, analysts, and business teams who rely on model outputs. A public or team-level status page lets them self-serve:

  1. Go to Status Pages → New Page.
  2. Add monitors: model serving, training pipeline, batch scoring.
  3. Label components for a non-technical audience: "Recommendation Engine", "Classifier Training", "Nightly Score Refresh".
  4. Share the URL with the business team in your model release runbook.

Key Metrics to Watch

| Metric | Threshold | Meaning | |---|---|---| | Training heartbeat | < 1 missed cycle | Pipeline completing successfully | | Batch scoring heartbeat | < 1 missed cycle | Daily inference running on schedule | | Model serving HTTP | Must return 200 | Predictions available | | Model age | < 25 hours | Not serving stale predictions | | Preprocessing error rate | < 0.01% | Input data quality acceptable |

Why Monitor Scholar?

ML pipelines in Elixir fail in ways that are harder to detect than typical web application failures:

  • Silent NaN propagation — a corrupted feature produces NaN tensors that flow through Scholar pipelines without exceptions, producing nonsense predictions
  • Training data schema drift — upstream data sources add or remove columns, causing preprocessing to silently drop features
  • Memory pressure OOM — large Nx tensors on nodes with insufficient RAM cause NIFs to crash in ways that may not bubble up to your supervision tree cleanly
  • Stale model serving — a crashed trainer leaves the serving endpoint returning predictions from an old model with no visible error

Vigilmon's heartbeat model treats every successful training run as a positive signal. If the pipeline runs, Vigilmon knows. If it doesn't, you hear about it before the business does.

Conclusion

Scholar brings mature ML primitives to Elixir without sacrificing the BEAM's concurrency and fault-tolerance properties. Vigilmon brings the external monitoring layer that makes Scholar pipelines production-grade: heartbeats on training runs, HTTP checks on serving endpoints, and instant alerts when your data science pipelines go dark. Set it up once, and your models stay observable even while you're focused elsewhere.

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles