How to Monitor FLAME with Vigilmon

title: How to Monitor FLAME with Vigilmon published: true description: Monitor Elixir FLAME elastic compute pools — track node provisioning latency, job queue depth, runner health, and BEAM memory with external uptime checks and heartbeat monitoring. tags: elixir, flame, monitoring, devops

FLAME (Function-Level Auto-scaling for Elixir) lets your Phoenix application elastically burst work onto remote BEAM nodes — spinning up fresh compute on Fly.io (or any cloud) only when needed, then tearing it down the moment the work is done. Chris McCord built it to solve a real production pain: your web dyno should stay snappy while CPU-heavy AI inference, video transcoding, or ML batch jobs run elsewhere.

The elastic nature of FLAME is exactly what makes it a monitoring challenge. Runners appear and disappear in seconds; queue depth can spike before the first runner node is even alive; a misconfigured pool silently refuses to scale. You need external eyes on both the coordination layer and the ephemeral compute itself.

This tutorial covers:

• A health check endpoint that exposes FLAME pool state
• HTTP uptime monitoring with Vigilmon
• Heartbeat monitoring for long-running FLAME jobs
• Alerts when runner provisioning fails or queues back up

Why Monitor FLAME?

FLAME abstractions hide failure gracefully — but silence is not health:

| Failure mode | Symptom without monitoring | |---|---| | Fly.io runner launch timeout | Jobs queue silently; users wait | | Pool at max_concurrency | Caller blocks with no feedback | | Runner OOM killed | Task crashes; caller gets {:exit, :killed} | | Network partition to runner | Pool stalls; no new runners are accepted | | Bad deploy image for runners | Every launch fails; pool stays empty |

Step 1: Expose FLAME pool health

Add a telemetry-backed status endpoint your health check can query.

# lib/my_app/flame_monitor.ex
defmodule MyApp.FlameMonitor do
  use GenServer

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def pool_status do
    GenServer.call(__MODULE__, :pool_status)
  end

  @impl true
  def init(state) do
    :ok = :telemetry.attach_many(
      "flame-monitor",
      [
        [:flame, :runner, :start],
        [:flame, :runner, :stop],
        [:flame, :runner, :exception],
      ],
      &__MODULE__.handle_event/4,
      nil
    )
    {:ok, Map.put(state, :counters, initial_counters())}
  end

  defp initial_counters do
    %{launches: 0, completions: 0, failures: 0, active: 0}
  end

  def handle_event([:flame, :runner, :start], _measurements, _meta, _config) do
    GenServer.cast(__MODULE__, :runner_started)
  end

  def handle_event([:flame, :runner, :stop], _measurements, _meta, _config) do
    GenServer.cast(__MODULE__, :runner_stopped)
  end

  def handle_event([:flame, :runner, :exception], _measurements, _meta, _config) do
    GenServer.cast(__MODULE__, :runner_failed)
  end

  @impl true
  def handle_cast(:runner_started, %{counters: c} = state) do
    {:noreply, %{state | counters: %{c | launches: c.launches + 1, active: c.active + 1}}}
  end

  def handle_cast(:runner_stopped, %{counters: c} = state) do
    {:noreply, %{state | counters: %{c | completions: c.completions + 1, active: max(0, c.active - 1)}}}
  end

  def handle_cast(:runner_failed, %{counters: c} = state) do
    {:noreply, %{state | counters: %{c | failures: c.failures + 1, active: max(0, c.active - 1)}}}
  end

  @impl true
  def handle_call(:pool_status, _from, %{counters: c} = state) do
    {:reply, c, state}
  end
end

Add to your supervision tree:

# lib/my_app/application.ex
children = [
  # ... existing children
  MyApp.FlameMonitor,
]

Step 2: Health check plug

# lib/my_app_web/plugs/health_check.ex
defmodule MyAppWeb.Plugs.HealthCheck do
  import Plug.Conn

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health"} = conn, _opts) do
    flame_status = MyApp.FlameMonitor.pool_status()
    db_ok = check_database()

    flame_ok =
      flame_status.failures < 5 or
        (flame_status.launches > 0 and
           flame_status.failures / flame_status.launches < 0.2)

    overall = if db_ok and flame_ok, do: 200, else: 503

    body =
      Jason.encode!(%{
        status: if(overall == 200, do: "ok", else: "degraded"),
        checks: %{
          database: if(db_ok, do: "ok", else: "error"),
          flame_runners: if(flame_ok, do: "ok", else: "degraded"),
          flame_active: flame_status.active,
          flame_failures: flame_status.failures
        }
      })

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(overall, body)
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp check_database do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> true
      _ -> false
    end
  end
end

Wire it into your endpoint before the router:

# lib/my_app_web/endpoint.ex
plug MyAppWeb.Plugs.HealthCheck
plug MyAppWeb.Router

Step 3: Add Vigilmon HTTP monitoring

1. Log in at vigilmon.online and click New Monitor.
2. Set URL to https://your-app.fly.dev/health.
3. Set check interval to 60 seconds (FLAME pools can cold-start in 10-30 s, so 60 s avoids false positives during a burst).
4. Set HTTP keyword check to "ok" so the monitor fails if the response body contains "degraded".
5. Set alert threshold to 2 consecutive failures before paging.

Step 4: Heartbeat monitoring for long FLAME jobs

Wrap long-running FLAME calls with a heartbeat ping so Vigilmon knows they are alive:

defmodule MyApp.FlameTasks do
  @heartbeat_url "https://vigilmon.online/api/heartbeats/YOUR_HEARTBEAT_ID/ping"

  def run_batch_job(items) do
    FLAME.call(:cpu_pool, fn ->
      ping_heartbeat()

      items
      |> Enum.chunk_every(100)
      |> Enum.each(fn chunk ->
        process_chunk(chunk)
        ping_heartbeat()
      end)
    end)
  end

  defp ping_heartbeat do
    Task.start(fn ->
      :httpc.request(:get, {String.to_charlist(@heartbeat_url), []}, [], [])
    end)
  end

  defp process_chunk(chunk), do: Enum.map(chunk, &transform/1)

  defp transform(item), do: item
end

Create a Heartbeat monitor in Vigilmon:

1. Go to Monitors -> New -> Heartbeat.
2. Set grace period to 10 minutes (enough time for runner cold-start + batch processing).
3. Copy the ping URL into @heartbeat_url above.

Step 5: Key metrics to alert on

| Metric | Alert threshold | Why | |---|---|---| | /health HTTP status | Non-200 for 2 checks | Core availability | | flame_failures in response body | > 5 in payload | Runner launch failures accumulating | | Heartbeat silence | Grace period exceeded | Batch job hung or runner evicted | | Response time on /health | > 5 s | Pool coordination delay |

Step 6: Status page

Add your FLAME-backed services to a public status page:

1. In Vigilmon, go to Status Pages -> New.
2. Add your HTTP monitor and heartbeat monitor as components.
3. Name them descriptively: "API (web)" and "Background Processing (FLAME)".
4. Share the URL with your team so on-call engineers have instant context.

Conclusion

FLAME elastic compute is powerful but invisible by default — jobs queue, runners fail, and OOM kills happen silently if you are not watching. With Vigilmon HTTP and heartbeat monitors you get:

• Uptime checks that detect when the health endpoint reports pool degradation
• Heartbeat monitors that catch hung or evicted batch jobs
• Alerts before your queue backlog becomes a user-facing outage

Start with the /health endpoint and a single heartbeat, then add metrics as your FLAME usage grows.