tutorial

How to Monitor Benchee with Vigilmon

Monitor your Elixir Benchee benchmarking setup — detect performance regressions, benchmark execution failures, and statistical anomalies before they reach production.

How to Monitor Benchee with Vigilmon

Benchee is the standard benchmarking library for Elixir. It measures and compares the performance of Elixir functions with statistical rigor — mean, median, standard deviation, percentiles, and IPS (iterations per second) — and supports multiple output formatters including console, HTML, CSV, and JSON. Teams use Benchee to catch performance regressions before they reach production and to validate that optimizations actually improve throughput.

When Benchee runs cleanly, you get reliable performance baselines. When a benchmark suite silently breaks — a dependency changes behavior, a warmup phase is skipped, or the benchmark harness crashes — you lose your baseline without knowing it. Vigilmon heartbeats paired with Benchee CI runs catch these gaps automatically.

Why Monitor Benchee?

Performance benchmarks are only as valuable as the infrastructure running them. Common failure modes include:

  • Benchmark suite crashes — a function under test raises an exception during warmup or measurement; Benchee exits non-zero but no one is watching
  • Regression without alert — a dependency upgrade changes a hot-path algorithm; IPS drops 40% but no comparison baseline exists to surface the delta
  • Warmup misconfiguration — warmup: 0 is set for speed in CI, causing the JIT to skew early measurements; benchmarks look faster than they are
  • Memory bloat — memory_time is not measured; a change that improves IPS while tripling heap allocations goes undetected
  • CI skipped under load — benchmark jobs are the first to be cancelled when CI queues fill; no heartbeat means the omission is invisible
  • JSON output drift — the Benchee JSON formatter output schema changes across versions; downstream tools parsing the JSON silently get wrong data

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | IPS (iterations per second) | Raw throughput for each scenario | | Mean execution time | Average latency per call | | Median execution time | Latency without outlier distortion | | Standard deviation | Measurement stability — high stddev means noisy environment | | p99 execution time | Worst-case latency for tail-sensitive paths | | Memory usage | Heap allocations per iteration (requires memory_time) | | CI heartbeat | Whether benchmarks ran recently | | Benchmark suite exit code | Whether all scenarios completed without error |

Step 1: Add Benchee to Your Project

# mix.exs
defp deps do
  [
    {:benchee, "~> 1.3", only: :dev},
    {:benchee_json, "~> 1.0", only: :dev},
    {:benchee_html, "~> 1.0", only: :dev},
    # rest of your deps
  ]
end

Step 2: Write a Benchmark Suite

# bench/parser_bench.exs
alias MyApp.Parser

inputs = %{
  "small (100 chars)"  => String.duplicate("x", 100),
  "medium (10 KB)"     => String.duplicate("x", 10_000),
  "large (1 MB)"       => String.duplicate("x", 1_000_000),
}

Benchee.run(
  %{
    "parse_v1" => fn input -> Parser.parse_v1(input) end,
    "parse_v2" => fn input -> Parser.parse_v2(input) end,
  },
  inputs: inputs,
  warmup: 2,
  time: 10,
  memory_time: 2,
  reduction_time: 2,
  formatters: [
    Benchee.Formatters.Console,
    {Benchee.Formatters.JSON, file: "bench/output/parser_bench.json"},
    {Benchee.Formatters.HTML, file: "bench/output/parser_bench.html"},
  ],
  print: [fast_warning: true]
)

Run it:

mkdir -p bench/output
mix run bench/parser_bench.exs

Step 3: Write a Regression Comparison Script

# bench/compare_results.exs
# Usage: mix run bench/compare_results.exs baseline.json current.json 0.10
[baseline_file, current_file, threshold_str] = System.argv()
threshold = String.to_float(threshold_str)

baseline = baseline_file |> File.read!() |> Jason.decode!()
current  = current_file  |> File.read!() |> Jason.decode!()

regressions =
  for {scenario, current_stats} <- current["scenarios"],
      baseline_stats = get_in(baseline, ["scenarios", scenario]),
      baseline_stats != nil do
    baseline_ips = get_in(baseline_stats, ["run_time_data", "statistics", "ips"])
    current_ips  = get_in(current_stats,  ["run_time_data", "statistics", "ips"])
    delta = (current_ips - baseline_ips) / baseline_ips

    if delta < -threshold do
      IO.puts("REGRESSION #{scenario}: IPS dropped #{Float.round(delta * 100, 1)}% " <>
              "(#{Float.round(baseline_ips, 0)} → #{Float.round(current_ips, 0)})")
      scenario
    else
      IO.puts("OK #{scenario}: IPS delta #{Float.round(delta * 100, 1)}%")
      nil
    end
  end
  |> Enum.filter(& &1)

if regressions != [] do
  IO.puts("\n#{length(regressions)} regression(s) exceed #{threshold * 100}% threshold")
  System.halt(1)
else
  IO.puts("\nNo regressions detected (threshold: #{threshold * 100}%)")
end

Step 4: Write a Benchmark Smoke Test

Add a fast ExUnit test that validates the benchmark suite itself executes without error:

# test/bench/benchee_smoke_test.exs
defmodule MyApp.BencheeSmokeTest do
  use ExUnit.Case, async: false

  @tag :benchmark_smoke
  test "parser benchmark scenarios execute without error" do
    results =
      Benchee.run(
        %{
          "parse_v1" => fn input -> MyApp.Parser.parse_v1(input) end,
          "parse_v2" => fn input -> MyApp.Parser.parse_v2(input) end,
        },
        inputs: %{"smoke" => "test input"},
        warmup: 0,
        time: 0.1,
        memory_time: 0,
        print: [benchmarking: false, configuration: false, fast_warning: false],
        formatters: []
      )

    assert map_size(results.scenarios) == 2, "Expected 2 scenarios, got #{map_size(results.scenarios)}"

    for {_name, scenario} <- results.scenarios do
      assert scenario.run_time_data.statistics.ips > 0,
        "#{scenario.name}: IPS is zero — benchmark may have crashed"
    end
  end

  @tag :benchmark_smoke
  test "all benchmarked functions return non-nil for all input sizes" do
    inputs = ["short", String.duplicate("x", 1_000), String.duplicate("x", 100_000)]

    for input <- inputs do
      assert is_binary(MyApp.Parser.parse_v1(input))
      assert is_binary(MyApp.Parser.parse_v2(input))
    end
  end
end

Step 5: Run Benchmarks in CI with a Heartbeat

# .github/workflows/benchmarks.yml
name: Benchee Benchmarks

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 4 * * 1'   # every Monday at 04:00 UTC

jobs:
  benchmark:
    name: Run Benchee Suite
    runs-on: ubuntu-latest
    env:
      VIGILMON_BENCHEE_HEARTBEAT_URL: ${{ secrets.VIGILMON_BENCHEE_HEARTBEAT_URL }}
    steps:
      - uses: actions/checkout@v4

      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.17'
          otp-version: '27'

      - name: Cache deps
        uses: actions/cache@v3
        with:
          path: deps
          key: ${{ runner.os }}-deps-${{ hashFiles('mix.lock') }}

      - run: mix deps.get

      - name: Benchmark smoke test
        run: mix test test/bench/benchee_smoke_test.exs --color --include benchmark_smoke

      - name: Run benchmark suite
        run: |
          mkdir -p bench/output
          mix run bench/parser_bench.exs

      - name: Upload benchmark results
        uses: actions/upload-artifact@v4
        with:
          name: benchmarks-${{ github.sha }}
          path: bench/output/

      - name: Compare against baseline
        run: |
          if [ -f bench/baseline.json ]; then
            mix run bench/compare_results.exs bench/baseline.json bench/output/parser_bench.json 0.10
          else
            echo "No baseline found — skipping regression check"
          fi

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -fsS "$VIGILMON_BENCHEE_HEARTBEAT_URL"

Step 6: Create a Heartbeat Monitor in Vigilmon

  1. Sign in at vigilmon.online
  2. Click New Monitor → Heartbeat
  3. Name it Benchee Benchmark Suite — main branch
  4. Set the expected interval to 8 days (matches the weekly Monday schedule with buffer)
  5. Copy the URL and store it as VIGILMON_BENCHEE_HEARTBEAT_URL in CI secrets

If the benchmark suite crashes — because a function under test was refactored to raise, a formatter was misconfigured, or CI cancelled the job — no heartbeat fires. Vigilmon alerts you within the missed-interval window.

Step 7: Track IPS Over Time

Emit benchmark results as structured logs so your APM tool or a simple dashboard can track IPS trends:

# bench/formatters/structured_log_formatter.ex
defmodule MyApp.Bench.StructuredLogFormatter do
  @behaviour Benchee.Formatter

  @impl true
  def format(suite, _opts) do
    Enum.each(suite.scenarios, fn scenario ->
      stats = scenario.run_time_data.statistics
      Jason.encode!(%{
        scenario: scenario.name,
        input: scenario.input_name,
        ips: stats.ips,
        mean_ns: stats.average,
        median_ns: stats.median,
        p99_ns: stats.percentiles[99],
        stddev_ns: stats.std_dev,
        timestamp: DateTime.utc_now() |> DateTime.to_iso8601()
      })
      |> IO.puts()
    end)
  end

  @impl true
  def write(data, _opts), do: :ok
end

Step 8: Alerting

In Vigilmon, go to Notifications → New Channel → Slack:

Heartbeat missed:

🔴 MISSED: Benchee Benchmark Suite — main branch
Last successful benchmark run: 10 days ago
Action: Check CI — benchmark suite crash or job cancellation

What You Built

| What | How | |------|-----| | Benchmark suite | Benchee with warmup, memory_time, multiple formatters | | Regression comparison | Elixir script comparing IPS delta against a threshold | | Smoke test | ExUnit test — validates all scenarios execute without error | | CI integration | GitHub Actions weekly schedule | | Artifact upload | Benchmark JSON and HTML results stored per commit | | CI heartbeat | Vigilmon heartbeat — alerts when benchmarks miss | | Structured logging | JSON formatter emitting per-scenario stats for trending | | Slack alerting | Vigilmon Slack notification channel |

Benchee gives you the measurement. Vigilmon ensures the measurement keeps running.

Next Steps

  • Store baseline JSON in a branch or object storage and update it via PR review to formalize regressions as accepted or rejected
  • Add reduction_time measurement to track reduction count regressions alongside time
  • Use Benchee.Formatters.HTML to publish interactive charts to GitHub Pages on each benchmark run
  • Set up a Vigilmon status page for your engineering team showing benchmark health over time

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles