How to Monitor Muontrap with Vigilmon

Muontrap is the Elixir library for managing OS processes from Elixir and Nerves. It wraps subprocess execution with Linux cgroup support — meaning you can enforce CPU and memory limits on child processes, get clean process supervision, and guarantee that child processes are killed when the Elixir process exits (no orphaned children). It's used in embedded systems, data pipelines, FFmpeg wrappers, and any Elixir service that shells out to native binaries.

But OS process management has failure modes that HTTP uptime monitoring misses entirely: a cgroup memory limit is silently hit and the subprocess exits, a zombie process accumulates because the OS parent dies without cleanup, or a native binary crashes and Muontrap's supervisor quietly restarts it without alerting anyone. By the time you notice, your pipeline has been dropping data for hours.

This tutorial monitors Muontrap-managed processes end-to-end:

• A process health check that verifies a managed subprocess is alive and responsive
• HTTP uptime monitoring with Vigilmon
• Heartbeat monitoring for long-running subprocess health
• cgroup resource utilization alerting
• Restart-count monitoring via telemetry

Why Monitor Muontrap?

| Signal | What it catches | |---|---| | Process liveness | Subprocess exited without supervision catching it | | cgroup memory usage | Approaching memory limit before OOM kill | | cgroup CPU throttling | Subprocess being starved, causing latency spikes | | Restart frequency | Frequent crashes indicating a deeper bug | | Output pipe health | Subprocess producing no output (stuck/deadlocked) |

Standard HTTP uptime monitoring can't see any of these. You need OS-level probes.

Step 1: Wrap Your Subprocess in a Supervised GenServer

Structure your Muontrap usage so the subprocess is tracked by a GenServer that can report health:

# lib/my_app/pipeline/worker.ex
defmodule MyApp.Pipeline.Worker do
  use GenServer
  require Logger

  defstruct [:port, :pid, :restart_count, :last_output_at, :cgroup_path]

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  def health do
    GenServer.call(__MODULE__, :health)
  end

  def init(opts) do
    cgroup_path = Keyword.get(opts, :cgroup_path, "/sys/fs/cgroup/my_app/worker")
    cmd = Keyword.fetch!(opts, :cmd)
    args = Keyword.get(opts, :args, [])

    state = start_process(%__MODULE__{
      restart_count: 0,
      cgroup_path: cgroup_path
    }, cmd, args)

    {:ok, state}
  end

  def handle_call(:health, _from, state) do
    health = %{
      process_alive: port_alive?(state.port),
      restart_count: state.restart_count,
      last_output_seconds_ago: seconds_since(state.last_output_at),
      cgroup_memory_bytes: read_cgroup_stat(state.cgroup_path, "memory.current"),
      cgroup_cpu_throttle_us: read_cgroup_stat(state.cgroup_path, "cpu.stat", "throttled_usec")
    }
    {:reply, health, state}
  end

  def handle_info({port, {:data, data}}, %{port: port} = state) do
    Logger.debug("Worker output: #{inspect(data)}")
    {:noreply, %{state | last_output_at: System.monotonic_time(:second)}}
  end

  def handle_info({port, {:exit_status, code}}, %{port: port} = state) do
    Logger.error("Worker exited with code #{code}, restarting")
    new_state = start_process(%{state | restart_count: state.restart_count + 1}, nil, nil)
    {:noreply, new_state}
  end

  defp start_process(state, cmd, args) do
    cmd = cmd || state[:cmd]
    args = args || state[:args]

    muontrap_args = [
      cmd,
      args,
      [
        stderr_to_stdout: true,
        cgroup_path: state.cgroup_path
      ]
    ]

    port = Port.open({:spawn_executable, System.find_executable("muontrap")}, [
      :binary,
      :exit_status,
      :use_stdio,
      args: List.flatten(["--cgroup-path", state.cgroup_path, "--", cmd | args])
    ])

    %{state | port: port, last_output_at: System.monotonic_time(:second)}
  end

  defp port_alive?(nil), do: false
  defp port_alive?(port) do
    port in Port.list()
  end

  defp seconds_since(time), do: System.monotonic_time(:second) - time

  defp read_cgroup_stat(cgroup_path, file, key \\ nil) do
    path = Path.join(cgroup_path, file)
    case File.read(path) do
      {:ok, contents} when is_nil(key) ->
        String.trim(contents) |> Integer.parse() |> elem(0)
      {:ok, contents} ->
        contents
        |> String.split("\n")
        |> Enum.find_value(fn line ->
          case String.split(line, " ") do
            [^key, value] -> Integer.parse(value) |> elem(0)
            _ -> nil
          end
        end) || 0
      {:error, _} ->
        nil
    end
  end
end

Step 2: Add an HTTP Health Endpoint

Expose the worker health check via a Phoenix controller:

# lib/my_app_web/controllers/process_health_controller.ex
defmodule MyAppWeb.ProcessHealthController do
  use MyAppWeb, :controller

  @max_restart_rate 10
  @max_output_silence_seconds 300

  def worker(conn, _params) do
    health = MyApp.Pipeline.Worker.health()

    {status_code, status} = cond do
      not health.process_alive ->
        {503, "error: process not alive"}
      health.restart_count > @max_restart_rate ->
        {503, "error: restart rate too high"}
      health.last_output_seconds_ago > @max_output_silence_seconds ->
        {503, "error: process silent too long"}
      true ->
        {200, "ok"}
    end

    conn
    |> put_status(status_code)
    |> json(Map.merge(health, %{status: status}))
  end
end

# In your router
scope "/health", MyAppWeb do
  pipe_through :api
  get "/worker", ProcessHealthController, :worker
end

Step 3: Add Vigilmon HTTP Monitoring

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your worker health URL: https://yourapp.com/health/worker.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Advanced, add a Response body contains check: "status":"ok".
7. Click Save.

Vigilmon will alert you when your managed process crashes or stops producing output.

Step 4: Monitor Long-Running Processes with a Heartbeat

For pipelines that run indefinitely (FFmpeg transcoding, data ingestion, etc.), use a heartbeat that the process itself periodically pings — if the process stalls, the heartbeat stops:

# lib/my_app/pipeline/heartbeat_worker.ex
defmodule MyApp.Pipeline.HeartbeatWorker do
  use GenServer
  require Logger

  @check_interval :timer.seconds(30)
  @vigilmon_heartbeat_url "https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_ID"

  def start_link(_), do: GenServer.start_link(__MODULE__, nil, name: __MODULE__)

  def init(_), do: schedule() |> then(fn _ -> {:ok, nil} end)

  def handle_info(:check, state) do
    health = MyApp.Pipeline.Worker.health()

    if health.process_alive and health.last_output_seconds_ago < 60 do
      HTTPoison.get!(@vigilmon_heartbeat_url)
    else
      Logger.warning(
        "Muontrap worker unhealthy — skipping heartbeat " <>
        "(alive=#{health.process_alive}, silent=#{health.last_output_seconds_ago}s)"
      )
    end

    schedule()
    {:noreply, state}
  end

  defp schedule, do: Process.send_after(self(), :check, @check_interval)
end

To create the heartbeat monitor in Vigilmon:

1. Click Add Monitor → Cron / Heartbeat.
2. Set the expected interval to 1 minute (the probe runs every 30 seconds so Vigilmon gets two chances per minute).
3. Copy the generated URL into @vigilmon_heartbeat_url.

Step 5: Alert on cgroup Resource Exhaustion

cgroup memory limits cause silent OOM kills with no Elixir exception. Poll the cgroup stats and log warnings before the limit is hit:

# lib/my_app/pipeline/cgroup_monitor.ex
defmodule MyApp.Pipeline.CgroupMonitor do
  use GenServer
  require Logger

  @check_interval :timer.minutes(1)
  @memory_warning_pct 0.80

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)

  def init(opts) do
    cgroup_path = Keyword.fetch!(opts, :cgroup_path)
    memory_limit = read_memory_limit(cgroup_path)
    schedule()
    {:ok, %{cgroup_path: cgroup_path, memory_limit: memory_limit}}
  end

  def handle_info(:check, %{cgroup_path: path, memory_limit: limit} = state) do
    current = read_stat(path, "memory.current")

    if limit && current do
      usage_pct = current / limit
      if usage_pct >= @memory_warning_pct do
        Logger.error(
          "[Muontrap cgroup] Memory at #{Float.round(usage_pct * 100, 1)}% " <>
          "(#{format_bytes(current)} / #{format_bytes(limit)}). " <>
          "OOM kill imminent — reduce batch size or increase cgroup limit."
        )
      end
    end

    schedule()
    {:noreply, state}
  end

  defp read_memory_limit(path) do
    case File.read(Path.join(path, "memory.max")) do
      {:ok, "max\n"} -> nil
      {:ok, value} -> Integer.parse(String.trim(value)) |> elem(0)
      _ -> nil
    end
  end

  defp read_stat(path, file) do
    case File.read(Path.join(path, file)) do
      {:ok, value} -> Integer.parse(String.trim(value)) |> elem(0)
      _ -> nil
    end
  end

  defp format_bytes(bytes) when bytes >= 1_048_576,
    do: "#{Float.round(bytes / 1_048_576, 1)}MB"
  defp format_bytes(bytes) when bytes >= 1024,
    do: "#{Float.round(bytes / 1024, 1)}KB"
  defp format_bytes(bytes), do: "#{bytes}B"

  defp schedule, do: Process.send_after(self(), :check, @check_interval)
end

Add it to your supervision tree:

{MyApp.Pipeline.CgroupMonitor, cgroup_path: "/sys/fs/cgroup/my_app/worker"}

Step 6: Set Up Alerts

Configure Vigilmon alert channels for your Muontrap monitors:

1. Go to Alert Channels → Add Channel.
2. Add a Slack webhook for team alerts or PagerDuty for on-call.
3. Set Alert after 2 consecutive failures on the HTTP monitor (avoids noise from brief process restarts).
4. Set Alert after 1 failure on the heartbeat monitor — a stopped heartbeat means the pipeline is down.

For embedded/Nerves deployments without public endpoints, use heartbeat-only monitoring: have the device ping Vigilmon directly via HTTPoison from inside the supervised process.

Key Metrics to Watch

| Metric | Vigilmon feature | Alert threshold | |---|---|---| | Process liveness | HTTP monitor on /health/worker | Any non-200 | | Pipeline output activity | Heartbeat (stops when silent) | Missing for > 2 min | | SSL certificate | Cert expiry monitor | Expires in < 14 days | | cgroup memory usage | Custom telemetry + logs | > 80% of limit | | Restart count | HTTP response field | > 10 restarts |

Conclusion

Muontrap gives Elixir robust OS process management with cgroup-level resource control, but robust doesn't mean invisible — process crashes, memory exhaustion, and silent pipeline stalls all need monitoring. Combining an HTTP health endpoint that reflects process liveness with a heartbeat that the pipeline itself controls gives you real end-to-end visibility: if either the Elixir supervisor or the native subprocess is failing, Vigilmon will know.

Get started free at vigilmon.online.