How to Monitor Axon with Vigilmon

Axon brings deep learning to the BEAM ecosystem — define neural networks in Elixir, compile them to CPU or GPU backends via Nx, and train or infer at production scale. But ML workloads have failure modes that HTTP uptime checks never see: a training loop can appear alive while loss diverges to NaN, a batch inference service can silently time out while returning HTTP 200, and GPU memory can exhaust mid-epoch with no process crash.

This tutorial sets up layered monitoring for Axon workloads:

• A health endpoint that verifies the Nx backend is reachable
• HTTP uptime monitoring with Vigilmon
• Heartbeat monitoring for long-running training jobs
• Loss and metric alerting via telemetry
• Slack alerts when inference latency degrades

Why Monitor Axon?

| Signal | What it catches | |---|---| | Nx backend availability | EXLA / CUDA driver failures, Metal backend crashes | | Training loop liveness | Hung epochs, deadlocked data pipelines | | Loss divergence | NaN loss, exploding gradients, misconfigured learning rate | | Inference latency | Model load time regression, batch size misconfiguration | | Memory utilization | GPU OOM before process crash |

Standard process monitoring misses all of these. You need signals from inside the training loop.

Step 1: Add a Health Endpoint for Your Inference Service

If you expose Axon models via a Phoenix or Plug HTTP service, add a health endpoint that runs a minimal forward pass:

# lib/my_app_web/plugs/axon_health.ex
defmodule MyAppWeb.Plugs.AxonHealth do
  import Plug.Conn

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health/axon"} = conn, _opts) do
    {status, body} = case run_health_inference() do
      {:ok, _output} ->
        {200, %{status: "ok", backend: Nx.default_backend() |> inspect()}}
      {:error, reason} ->
        {503, %{status: "error", reason: inspect(reason)}}
    end

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, Jason.encode!(body))
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp run_health_inference do
    try do
      # Run a trivial forward pass on the loaded model
      input = Nx.tensor([[1.0, 0.0, 0.0]], type: :f32)
      output = MyApp.Model.predict(input)
      {:ok, output}
    rescue
      e -> {:error, e}
    end
  end
end

Register the plug in your endpoint:

# lib/my_app_web/endpoint.ex
plug MyAppWeb.Plugs.AxonHealth
plug MyAppWeb.Router

Test it:

curl http://localhost:4000/health/axon
# => {"status":"ok","backend":"EXLA.Backend"}

Step 2: Add Vigilmon Uptime Monitoring

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your health endpoint URL: https://inference.yourapp.com/health/axon.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Advanced, add a Response body contains check: "status":"ok".
7. Click Save.

Vigilmon now verifies your Nx backend and model are reachable every minute.

Step 3: Heartbeat Monitoring for Training Jobs

Long-running Axon training loops can stall silently — a deadlocked data pipeline or hung Nx operation keeps the process alive while progress stops. Use a heartbeat monitor to detect this.

Add a heartbeat ping inside your training loop's epoch callback:

# lib/my_app/training/axon_trainer.ex
defmodule MyApp.Training.AxonTrainer do
  @vigilmon_heartbeat_url "https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_ID"

  def train(model, data, opts \\ []) do
    epochs = Keyword.get(opts, :epochs, 10)

    model
    |> Axon.Loop.trainer(:categorical_cross_entropy, :adam)
    |> Axon.Loop.handle_event(:epoch_completed, &on_epoch_complete/1)
    |> Axon.Loop.run(data, %{}, epochs: epochs)
  end

  defp on_epoch_complete(state) do
    ping_vigilmon(state)
    {:continue, state}
  end

  defp ping_vigilmon(state) do
    epoch = state.epoch
    loss = state.step_state[:loss] |> Nx.to_number()

    # Only ping when loss is finite and training is healthy
    if is_float(loss) and not is_nan(loss) and loss < 100.0 do
      Task.start(fn ->
        HTTPoison.get!(@vigilmon_heartbeat_url)
      end)
    end

    :ok
  end

  defp is_nan(x), do: x != x
end

Create the heartbeat in Vigilmon:

1. Click Add Monitor → Cron / Heartbeat.
2. Set the expected ping interval to match your epoch duration (e.g., 10 minutes).
3. Set grace period to 2x the typical epoch time.
4. Copy the generated heartbeat URL into @vigilmon_heartbeat_url above.

If an epoch takes too long or loss diverges, the heartbeat stops and Vigilmon alerts you.

Step 4: Track Loss and Metrics via Telemetry

Emit Axon training metrics as telemetry events for structured monitoring:

# lib/my_app/training/axon_telemetry.ex
defmodule MyApp.Training.AxonTelemetry do
  def setup do
    :telemetry.attach_many(
      "axon-training",
      [
        [:my_app, :training, :epoch, :stop],
        [:my_app, :training, :batch, :stop]
      ],
      &handle_event/4,
      nil
    )
  end

  def handle_event([:my_app, :training, :epoch, :stop], measurements, metadata, _config) do
    loss = Map.get(measurements, :loss, nil)
    epoch = Map.get(metadata, :epoch, 0)

    if loss && loss > 10.0 do
      # Loss is diverging — alert via external webhook or log
      require Logger
      Logger.error("Axon training loss diverging: epoch=#{epoch} loss=#{loss}")
    end
  end

  def handle_event([:my_app, :training, :batch, :stop], measurements, _metadata, _config) do
    duration_ms = Map.get(measurements, :duration, 0) / 1_000_000
    if duration_ms > 5000 do
      require Logger
      Logger.warning("Slow Axon batch: #{duration_ms}ms")
    end
  end
end

Attach it in application.ex:

def start(_type, _args) do
  MyApp.Training.AxonTelemetry.setup()
  # ...
end

Step 5: Monitor Inference API Latency

For production inference services, add a Vigilmon monitor that checks response time:

1. In your Axon monitor, go to Settings → Response Time Alerting.
2. Set a warning threshold: alert if P95 latency exceeds 500ms.
3. Set a critical threshold: alert if P95 latency exceeds 2000ms.

Also add a dedicated latency health endpoint:

# lib/my_app_web/controllers/inference_controller.ex
defmodule MyAppWeb.InferenceController do
  use MyAppWeb, :controller

  def health(conn, _params) do
    start = System.monotonic_time(:millisecond)
    {:ok, _} = MyApp.Model.predict(Nx.tensor([[0.5, 0.5, 0.5]]))
    latency = System.monotonic_time(:millisecond) - start

    status = if latency < 1000, do: "ok", else: "degraded"
    json(conn, %{status: status, inference_latency_ms: latency})
  end
end

Step 6: Set Up Alerts

In Vigilmon, configure alert channels for your Axon monitors:

1. Go to Alert Channels → Add Channel.
2. Choose Slack, Email, or PagerDuty.
3. For training heartbeats, alert immediately on first miss (training stalls are always urgent).
4. For inference HTTP monitors, alert after 2 consecutive failures.
5. Assign alert channels to all Axon monitors.

Key Metrics to Watch

| Metric | Vigilmon feature | What to alert on | |---|---|---| | Inference service availability | HTTP monitor on /health/axon | Any 5xx or timeout | | Training loop liveness | Heartbeat monitor | Missing for > 2× epoch duration | | Inference latency | Response time chart | P95 > 500ms | | SSL certificate | Cert expiry monitor | Expires in < 14 days | | Model load time | HTTP monitor response time | > 10s on startup |

Conclusion

Axon unlocks production ML on the BEAM, but ML workloads need monitoring beyond process liveness: loss divergence, stalled training loops, and inference latency regressions are the real production risks. With a health endpoint for your inference service, a heartbeat inside your training loop, and Vigilmon watching both, you'll catch ML failures before they affect your users.

Get started free at vigilmon.online.