tutorial

How to Monitor ExMachina with Vigilmon

Monitor your Elixir ExMachina factory setup — detect factory crashes, broken associations, and test data quality regressions before they corrupt your CI or staging databases.

How to Monitor ExMachina with Vigilmon

ExMachina is the standard factory library for Elixir and Ecto. It simplifies test data creation by defining reusable factory modules — with traits, sequences, and associations — so tests can say insert!(:user) or build(:post, published: true) instead of manually constructing structs or calling Repo.insert!/1 with hand-crafted changesets.

When ExMachina works, tests are clean and isolated. When factories break — an Ecto schema changes without updating the factory, an association factory is removed, or a sequence overflows — tests fail with cryptic database errors rather than clear fixture errors, and CI becomes hard to diagnose. Vigilmon heartbeats paired with factory health checks catch these regressions before they slow down your team.

Why Monitor ExMachina?

Factory regressions surface as test failures that look like application bugs:

  • Schema drift — a field is added to an Ecto schema with a NOT NULL constraint; the factory still uses the old fields; every test that inserts that schema fails with a database constraint error
  • Broken associations — a factory's belongs_to association references another factory that was renamed or deleted; build_list(:post) raises a KeyError or FunctionClauseError
  • Sequence overflow — a factory uses a sequence for a unique integer field; after enough test runs, the sequence counter hits database INT overflow and inserts fail
  • Trait conflicts — two traits set conflicting field values; a test using both traits silently gets unpredictable data
  • Missing required fields — a changeset adds a new required field; the factory does not include it; tests pass because build/2 skips changeset validation, but insert!/1 fails
  • Cascade delete gaps — a factory creates associations that leave orphan records when the primary record is deleted; Ecto raises a foreign-key constraint error in teardown

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Factory build error rate | Whether build/2 raises for any registered factory | | Factory insert error rate | Whether insert!/1 succeeds for all factories | | Association traversal success | Whether nested associations build without error | | Trait smoke test pass rate | Whether each trait produces valid changesets | | CI heartbeat | Whether factory health checks ran recently | | Orphan record count | Whether teardown leaves dangling association rows | | Sequence headroom | How close sequence counters are to overflow |

Step 1: Add ExMachina to Your Project

# mix.exs
defp deps do
  [
    {:ex_machina, "~> 2.8", only: :test},
    {:ecto_sql, "~> 3.11"},
    # rest of your deps
  ]
end

Step 2: Define Your Factory Module

# test/support/factory.ex
defmodule MyApp.Factory do
  use ExMachina.Ecto, repo: MyApp.Repo

  def user_factory do
    %MyApp.Accounts.User{
      first_name:     sequence(:first_name, &"User#{&1}"),
      last_name:      "Smith",
      email:          sequence(:email, &"user#{&1}@example.com"),
      role:           :member,
      confirmed_at:   DateTime.utc_now() |> DateTime.truncate(:second),
    }
  end

  def post_factory do
    %MyApp.Blog.Post{
      title:       sequence(:title, &"Post Title #{&1}"),
      body:        "Default post body for testing.",
      published:   false,
      author:      build(:user),
    }
  end

  def comment_factory do
    %MyApp.Blog.Comment{
      body:   "Test comment.",
      post:   build(:post),
      author: build(:user),
    }
  end

  # Traits
  def published_post_factory do
    struct!(
      post_factory(),
      published: true,
      published_at: DateTime.utc_now() |> DateTime.truncate(:second)
    )
  end

  def admin_user_factory do
    struct!(user_factory(), role: :admin)
  end
end

Step 3: Write a Factory Health Check Mix Task

# lib/mix/tasks/check_factories.ex
defmodule Mix.Tasks.CheckFactories do
  use Mix.Task

  @shortdoc "Validate all ExMachina factories build without error"

  @factories [
    :user,
    :post,
    :comment,
    :published_post,
    :admin_user,
  ]

  def run(_args) do
    Mix.Task.run("app.start")

    build_results =
      Enum.map(@factories, fn factory ->
        try do
          MyApp.Factory.build(factory)
          {factory, :build_ok}
        rescue
          e -> {factory, {:build_error, Exception.message(e)}}
        end
      end)

    failed_builds =
      Enum.filter(build_results, fn {_f, status} -> status != :build_ok end)

    if failed_builds == [] do
      Mix.shell().info("All #{length(@factories)} factories built successfully")
    else
      Enum.each(failed_builds, fn {factory, {:build_error, msg}} ->
        Mix.shell().error("BUILD FAIL :#{factory} — #{msg}")
      end)
      exit({:shutdown, 1})
    end
  end
end

Step 4: Write Factory Tests

# test/support/factory_test.exs
defmodule MyApp.FactoryTest do
  use MyApp.DataCase, async: false
  import MyApp.Factory

  describe "build/1 — all factories" do
    test "user factory builds a valid struct" do
      user = build(:user)
      assert %MyApp.Accounts.User{} = user
      assert is_binary(user.email)
      assert String.contains?(user.email, "@")
    end

    test "post factory builds with association" do
      post = build(:post)
      assert %MyApp.Blog.Post{} = post
      assert %MyApp.Accounts.User{} = post.author
    end

    test "comment factory builds with nested associations" do
      comment = build(:comment)
      assert %MyApp.Blog.Comment{} = comment
      assert %MyApp.Blog.Post{} = comment.post
      assert %MyApp.Accounts.User{} = comment.author
    end
  end

  describe "insert!/1 — all factories" do
    test "user inserts without database error" do
      user = insert!(:user)
      assert user.id != nil
    end

    test "post inserts with associated user" do
      post = insert!(:post)
      assert post.id != nil
      assert post.author_id != nil
    end

    test "comment inserts with associated post and user" do
      comment = insert!(:comment)
      assert comment.id != nil
      assert comment.post_id != nil
      assert comment.author_id != nil
    end
  end

  describe "traits" do
    test "published_post trait sets published: true" do
      post = build(:published_post)
      assert post.published == true
      assert post.published_at != nil
    end

    test "admin_user trait sets role: :admin" do
      user = build(:admin_user)
      assert user.role == :admin
    end

    test "published_post inserts successfully" do
      post = insert!(:published_post)
      assert post.id != nil
      assert post.published == true
    end
  end

  describe "sequences" do
    test "email sequences generate unique values" do
      emails = Enum.map(1..20, fn _ -> build(:user).email end)
      assert length(Enum.uniq(emails)) == 20, "Email sequence produced duplicates"
    end

    test "title sequences generate unique values" do
      titles = Enum.map(1..20, fn _ -> build(:post).title end)
      assert length(Enum.uniq(titles)) == 20, "Title sequence produced duplicates"
    end
  end

  describe "params_for/1" do
    test "user params_for produces a valid changeset" do
      params = params_for(:user)
      changeset = MyApp.Accounts.User.changeset(%MyApp.Accounts.User{}, params)
      assert changeset.valid?
    end
  end
end

Step 5: Run Factory Tests in CI with a Heartbeat

# .github/workflows/factory-health.yml
name: ExMachina Factory Health

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 7 * * *'   # daily at 07:00 UTC

jobs:
  factory-health:
    name: Factory Smoke Test
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: my_app_test
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    env:
      MIX_ENV: test
      DATABASE_URL: postgres://postgres:postgres@localhost:5432/my_app_test
      VIGILMON_EXMACHINA_HEARTBEAT_URL: ${{ secrets.VIGILMON_EXMACHINA_HEARTBEAT_URL }}
    steps:
      - uses: actions/checkout@v4

      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.17'
          otp-version: '27'

      - name: Cache deps
        uses: actions/cache@v3
        with:
          path: deps
          key: ${{ runner.os }}-deps-${{ hashFiles('mix.lock') }}

      - run: mix deps.get

      - name: Setup database
        run: mix ecto.create && mix ecto.migrate

      - name: Factory build smoke test
        run: mix check_factories

      - name: Factory ExUnit tests
        run: mix test test/support/factory_test.exs --color

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -fsS "$VIGILMON_EXMACHINA_HEARTBEAT_URL"

Step 6: Create a Heartbeat Monitor in Vigilmon

  1. Sign in at vigilmon.online
  2. Click New Monitor → Heartbeat
  3. Name it ExMachina Factory Health — main branch
  4. Set the expected interval to 25 hours (matches the daily CI schedule)
  5. Copy the URL and store it as VIGILMON_EXMACHINA_HEARTBEAT_URL in CI secrets

If a factory breaks — because an Ecto schema changed, an association was removed, or a sequence-based field overflowed — CI exits non-zero and no heartbeat fires. Vigilmon alerts you within the missed-interval window.

Step 7: Track Schema Drift

Add a schema-drift check to catch field mismatches between your Ecto schemas and factory definitions before they cause test failures:

# test/support/factory_schema_test.exs
defmodule MyApp.FactorySchemaTest do
  use ExUnit.Case, async: true
  import MyApp.Factory

  @schemas [
    {MyApp.Accounts.User, :user},
    {MyApp.Blog.Post,     :post},
    {MyApp.Blog.Comment,  :comment},
  ]

  describe "factory fields match schema fields" do
    for {schema, factory} <- @schemas do
      @schema schema
      @factory factory

      test "#{factory} factory covers all non-virtual, non-association fields" do
        schema_fields =
          @schema.__schema__(:fields)
          |> Enum.reject(fn f -> f in [:id, :inserted_at, :updated_at] end)

        built = build(@factory)
        built_map = Map.from_struct(built)

        missing =
          Enum.filter(schema_fields, fn field ->
            !Map.has_key?(built_map, field) or Map.get(built_map, field) == nil
          end)

        assert missing == [],
          "#{@factory} factory missing or nil for schema fields: #{inspect(missing)}"
      end
    end
  end
end

Step 8: Alerting

In Vigilmon, go to Notifications → New Channel → Slack:

Heartbeat missed:

🔴 MISSED: ExMachina Factory Health — main branch
Last healthy factory run: 30 hours ago
Action: Check CI — schema drift, broken association, or factory crash

What You Built

| What | How | |------|-----| | Factory module | ExMachina factories with traits, sequences, associations | | Build smoke test | Mix task: validates all factories build without error | | Insert tests | ExUnit: validates database inserts for all factories | | Trait tests | ExUnit: validates each trait produces the right fields | | Sequence uniqueness | ExUnit: checks sequences generate unique values | | Schema drift check | ExUnit: compares factory fields against Ecto schema | | CI integration | GitHub Actions daily schedule with Postgres service | | CI heartbeat | Vigilmon heartbeat — alerts when factory health misses | | Slack alerting | Vigilmon Slack notification channel |

ExMachina keeps your test data clean and consistent. Vigilmon ensures ExMachina keeps working.

Next Steps

  • Add params_for/2 changeset tests for all factories to catch required-field drift early
  • Use build_pair/2 and build_list/3 to test pagination, sorting, and list endpoints
  • Combine ExMachina with StreamData generators for property-based factory testing
  • Set up a Vigilmon status page for your QA team showing factory health over time

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles