How to Monitor Nx (Numerical Elixir) with Vigilmon

Nx brings multi-dimensional arrays and numerical computing to Elixir, enabling machine learning model training and inference directly on the BEAM. Combined with Axon for neural networks and Bumblebee for pre-trained models, Elixir applications can run ML inference in production without leaving the ecosystem.

But an ML inference service has different failure modes from a typical web app. The model might be loading. GPU memory might be exhausted. Inference latency might have degraded while the HTTP endpoint still returns 200. Numerical errors in a batch can be silent.

This tutorial adds production observability to an Nx-powered application:

• A health check endpoint that reports model and backend state
• Heartbeat monitoring for batch inference jobs and data pipelines
• Vigilmon uptime and latency monitoring
• Alerts for inference failures and model-serving degradation
• A status page for your ML infrastructure

Why Monitor Nx Applications?

Nx and Bumblebee applications have failure modes unique to ML systems:

• Model loading failures — the Serving process crashed during model load; inference requests queue forever
• Backend unavailability — EXLA or EMLX backend fails (GPU OOM, driver crash, CUDA version mismatch); Nx falls back to BinaryBackend silently
• Inference latency regression — a model update or backend change increased p99 latency; users notice degraded response times before any threshold is crossed
• Batch job stalls — a long-running numerical computation or training loop stalls without crashing
• Memory exhaustion — large tensor allocations push BEAM memory usage high; GC pressure increases
• Serving queue buildup — inference requests pile up faster than the model can process them

Standard process monitoring sees a healthy GenServer. External monitoring sees request failures and latency spikes.

Step 1: Add a health check endpoint for your Nx Serving

If you're using Nx.Serving (the standard way to serve Bumblebee models), expose its state over HTTP:

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def ml(conn, _params) do
    checks = %{
      serving: check_serving(),
      backend: check_backend(),
      memory: check_memory()
    }

    all_ok = Enum.all?(checks, fn {_, v} -> v == "ok" end)
    status = if all_ok, do: :ok, else: :service_unavailable

    conn
    |> put_status(status)
    |> json(%{
      status: if(all_ok, do: "ok", else: "degraded"),
      checks: checks,
      backend: Nx.default_backend() |> inspect()
    })
  end

  defp check_serving do
    # Check that the Serving process is alive and responsive
    case GenServer.whereis(MyApp.TextClassifier) do
      nil -> "not_started"
      pid ->
        if Process.alive?(pid), do: "ok", else: "dead"
    end
  rescue
    _ -> "error"
  end

  defp check_backend do
    # Verify the backend can execute a trivial operation
    try do
      Nx.tensor([1, 2, 3]) |> Nx.sum() |> Nx.to_number()
      "ok"
    rescue
      _ -> "error"
    end
  end

  defp check_memory do
    case :memsup.get_system_memory_data() do
      [] -> "ok"
      data ->
        total = Keyword.get(data, :total_memory, 1)
        free = Keyword.get(data, :free_memory, total)
        used_pct = (total - free) / total * 100
        if used_pct < 90, do: "ok", else: "high"
    end
  end
end

Add the route:

# lib/my_app_web/router.ex
scope "/health", MyAppWeb do
  pipe_through :api
  get "/ml", HealthController, :ml
end

Test it:

curl http://localhost:4000/health/ml
# {"status":"ok","checks":{"serving":"ok","backend":"ok","memory":"ok"},
#  "backend":"Nx.BinaryBackend"}

The backend field tells you whether EXLA/GPU or binary CPU fallback is active — which matters for both latency and cost.

Step 2: Smoke test inference in the health check

A process being alive doesn't mean inference works. Add a quick end-to-end smoke test:

defp check_inference do
  # Run a minimal inference request against your serving
  test_input = "This is a health check test."

  task = Task.async(fn ->
    Nx.Serving.run(MyApp.TextClassifier, test_input)
  end)

  case Task.yield(task, 5_000) do
    {:ok, _result} -> "ok"
    nil ->
      Task.shutdown(task, :brutal_kill)
      "timeout"
  end
rescue
  _ -> "error"
end

Include inference: check_inference() in your checks map. This catches model load failures and backend problems that wouldn't surface from a process-alive check alone.

Keep the test input small. Latency for the smoke test should be under your HTTP response timeout.

Step 3: Heartbeat monitoring for batch jobs

Nx is commonly used for batch inference pipelines — processing queues of images, text, or time-series data on a schedule. Use heartbeat monitoring to confirm these jobs complete successfully.

# lib/my_app/workers/batch_inference_worker.ex
defmodule MyApp.Workers.BatchInferenceWorker do
  use Oban.Worker, queue: :inference, max_attempts: 3

  require Logger

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"batch_id" => batch_id}}) do
    with {:ok, inputs} <- load_batch(batch_id),
         {:ok, results} <- run_inference(inputs),
         :ok <- store_results(batch_id, results) do
      ping_vigilmon(:batch_complete)
      :ok
    else
      {:error, reason} ->
        Logger.error("Batch inference failed: #{inspect(reason)}")
        {:error, reason}
    end
  end

  defp run_inference(inputs) do
    try do
      results = Nx.Serving.run(MyApp.ImageClassifier, inputs)
      {:ok, results}
    rescue
      e ->
        {:error, Exception.message(e)}
    end
  end

  defp ping_vigilmon(monitor_key) do
    url = Application.get_env(:my_app, :vigilmon_heartbeats)[monitor_key]
    if url do
      case Req.get(url, receive_timeout: 5_000) do
        {:ok, _} -> :ok
        {:error, reason} ->
          Logger.warning("Vigilmon heartbeat failed: #{inspect(reason)}")
      end
    end
  end

  defp load_batch(_id), do: {:ok, []}
  defp store_results(_id, _results), do: :ok
end

Configure heartbeat URLs:

# config/runtime.exs
config :my_app, :vigilmon_heartbeats,
  batch_complete: System.get_env("VIGILMON_BATCH_INFERENCE_HEARTBEAT_URL"),
  training_epoch: System.get_env("VIGILMON_TRAINING_EPOCH_HEARTBEAT_URL")

For long-running training loops, ping after each epoch:

defmodule MyApp.TrainingLoop do
  require Logger

  def train(model, data, epochs) do
    Enum.reduce(1..epochs, model, fn epoch, current_model ->
      updated = train_epoch(current_model, data)
      Logger.info("Epoch #{epoch}/#{epochs} complete")
      ping_epoch_heartbeat()
      updated
    end)
  end

  defp ping_epoch_heartbeat do
    url = Application.get_env(:my_app, :vigilmon_heartbeats)[:training_epoch]
    if url, do: Req.get(url, receive_timeout: 5_000)
  end

  defp train_epoch(model, _data), do: model
end

In Vigilmon, create Heartbeat Monitors for each:

1. Click New Monitor → Heartbeat
2. For batch jobs: expected interval matches your job schedule
3. For training: expected interval = your per-epoch time + buffer
4. Enable Slack notifications

Step 4: Latency monitoring for inference endpoints

Inference latency is as important as availability. An endpoint that returns 200 but in 10 seconds instead of 500ms is effectively down for real-time use cases.

In Vigilmon:

1. After creating your HTTP monitor, go to Monitor Settings → Advanced
2. Enable Response Time Alerts
3. Set a threshold: 2000ms (adjust for your model's expected latency)
4. Enable alerting when latency exceeds the threshold

Vigilmon tracks response time history, so you can see when a model update or backend change introduced a latency regression.

Step 5: Set up HTTP monitoring in Vigilmon

Point Vigilmon at your ML health endpoint:

1. Sign up at vigilmon.online
2. Click New Monitor → HTTP
3. Enter https://yourdomain.com/health/ml
4. Set check interval: 1 minute
5. Add body keyword: "status":"ok"
6. Save

The keyword check catches the case where your endpoint returns 200 but reports a degraded backend state.

Step 6: Telemetry for inference metrics

Nx.Serving supports telemetry events. Attach a handler to track inference timing:

defmodule MyApp.NxTelemetryHandler do
  require Logger

  def attach do
    :telemetry.attach_many(
      "nx-serving-handler",
      [
        [:nx, :serving, :execute, :stop],
        [:nx, :serving, :execute, :exception]
      ],
      &handle_event/4,
      nil
    )
  end

  def handle_event([:nx, :serving, :execute, :stop], measurements, _meta, _config) do
    duration_ms = System.convert_time_unit(measurements.duration, :native, :millisecond)

    if duration_ms > 2_000 do
      Logger.warning("Slow inference: #{duration_ms}ms")
    end
  end

  def handle_event([:nx, :serving, :execute, :exception], _measurements, meta, _config) do
    Logger.error("Inference exception: #{inspect(meta.reason)}")
  end
end

Start it in your application:

def start(_type, _args) do
  MyApp.NxTelemetryHandler.attach()
  # ...
end

Step 7: Slack alerts

In Vigilmon, go to Notifications → New Channel → Slack and paste your Slack incoming webhook URL.

Enable it on your HTTP monitor, heartbeat monitors, and latency threshold alerts:

🔴 DOWN: yourdomain.com/health/ml
Body keyword "status":"ok" not found
Response: {"status":"degraded","checks":{"serving":"dead",...}}

⚠️ SLOW: yourdomain.com/health/ml
Response time: 4,821ms (threshold: 2,000ms)

🔴 MISSED: Batch Inference Heartbeat
Last job: 3 hours ago (expected: every 1 hour)

Step 8: Status page

1. In Vigilmon, go to Status Pages → New Status Page
2. Add your ML health monitor, batch heartbeats, and training monitors
3. Label them: "Inference API", "Batch Processing", "Model Training"
4. Share the URL with your data and product teams

When a data scientist asks "is inference down?", they check the status page first.

What you've built

| What | How | |------|-----| | ML health endpoint | Phoenix controller checking Serving + backend | | Inference smoke test | End-to-end test run inside health check | | Heartbeat monitoring | Post-epoch and post-batch pings to Vigilmon | | Latency monitoring | Vigilmon response time threshold alerts | | HTTP uptime monitoring | Vigilmon HTTP monitor with keyword check | | Inference telemetry | :telemetry handler for slow/failed inferences | | Slack ML alerts | Vigilmon Slack notification channel | | Status page | Vigilmon public status page |

Nx runs the numbers. Vigilmon tells you when the numbers stop running.

Next steps

• Add GPU memory utilization to your health response if using EXLA with CUDA
• Monitor Bumblebee model download endpoints during cold starts (Hugging Face Hub)
• Use Vigilmon response time history to track model latency across deployments
• Set up separate heartbeat monitors per model when serving multiple models in production

Get started free at vigilmon.online.