tutorial

How to Monitor Faker (Elixir) with Vigilmon

Monitor your Elixir Faker data-generation setup — detect seed failures, locale misconfigurations, and test data quality regressions before they corrupt your staging or production databases.

How to Monitor Faker (Elixir) with Vigilmon

Faker is the go-to Elixir library for generating realistic fake data in tests and database seeds. It produces names, addresses, email addresses, phone numbers, URLs, and dozens of domain-specific values across multiple locales. Developers use it in ExUnit factories, seeds.exs scripts, and load-testing pipelines to populate databases without manually crafting fixtures.

When Faker works, test databases are rich and representative. When it breaks — a locale module is misconfigured, a custom provider crashes, or a seed script silently generates malformed data — tests produce misleading results, staging diverges from production, and data-quality bugs slip through. Vigilmon heartbeats paired with Faker health checks catch these regressions automatically.

Why Monitor Faker?

Faker failures are rarely loud. The library generates data lazily; a bad locale configuration or a missing provider module can return nil or raise only on specific code paths:

  • Locale misconfiguration — Faker.set_locale(:de) followed by Faker.Address.city/0 raises if the German locale module is not loaded; other locales continue working, masking the gap
  • Custom provider crashes — teams add custom Faker.Provider modules for domain data (insurance IDs, product SKUs); a refactor breaks the module without test coverage
  • Seed script failures — mix run priv/repo/seeds.exs runs inside a transaction; a Faker crash mid-seed leaves the database partially populated and subsequent seeds non-idempotent
  • Determinism drift — seeded random state (Faker.start/0 with a fixed seed) drifts when Faker updates its generator order; snapshot tests start failing for unrelated reasons
  • Nil data infiltrating CI — Faker.Internet.email/0 returns a valid string on every call, but a provider misconfiguration can silently return nil; assertions that only check is_binary/1 pass while downstream validations fail

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Seed script exit code | Whether mix run priv/repo/seeds.exs completes without error | | Faker provider smoke tests | Whether each configured locale and provider returns non-nil values | | CI heartbeat | Whether seed and test-data pipelines ran recently | | Seed duration | Unusual growth may indicate N+1 generation loops | | Duplicate key rate | Faker generating non-unique values when unique sequences are required | | Custom provider count | Whether all registered providers are still active |

Step 1: Add Faker to Your Project

# mix.exs
defp deps do
  [
    {:faker, "~> 0.18", only: [:dev, :test]},
    # rest of your deps
  ]
end

Initialize Faker in test/test_helper.exs:

# test/test_helper.exs
Faker.start()
ExUnit.start()

For a fixed seed (deterministic runs):

# test/test_helper.exs
Faker.start()
:rand.seed(:exsss, {1, 2, 3})
ExUnit.start()

Step 2: Write a Faker Provider Smoke-Test

# test/support/faker_smoke_test.exs
defmodule MyApp.FakerSmokeTest do
  use ExUnit.Case, async: false

  @locales [:en, :de, :fr, :es, :pt]

  describe "core providers return non-nil values for all locales" do
    for locale <- @locales do
      @locale locale

      test "Person provider — #{locale}" do
        Faker.locale(@locale)
        assert is_binary(Faker.Person.name()), "Person.name/0 returned nil for #{@locale}"
        assert is_binary(Faker.Person.first_name()), "Person.first_name/0 returned nil for #{@locale}"
        assert is_binary(Faker.Person.last_name()), "Person.last_name/0 returned nil for #{@locale}"
      end

      test "Address provider — #{locale}" do
        Faker.locale(@locale)
        assert is_binary(Faker.Address.city()), "Address.city/0 returned nil for #{@locale}"
        assert is_binary(Faker.Address.country()), "Address.country/0 returned nil for #{@locale}"
        assert is_binary(Faker.Address.zip_code()), "Address.zip_code/0 returned nil for #{@locale}"
      end

      test "Internet provider — #{locale}" do
        Faker.locale(@locale)
        email = Faker.Internet.email()
        assert is_binary(email), "Internet.email/0 returned nil for #{@locale}"
        assert String.contains?(email, "@"), "Internet.email/0 returned invalid email for #{@locale}"
      end
    end
  end

  describe "custom providers" do
    test "all registered custom providers return valid data" do
      Faker.locale(:en)
      # Replace with your own custom providers
      assert is_binary(Faker.Company.name())
      assert is_binary(Faker.Phone.EnUs.phone())
    end
  end
end

Step 3: Write a Seed Health Check Mix Task

# lib/mix/tasks/check_seeds.ex
defmodule Mix.Tasks.CheckSeeds do
  use Mix.Task

  @shortdoc "Validate Faker-based seed scripts without writing to the database"

  def run(_args) do
    Faker.start()

    checks = [
      {"Person.name", fn -> Faker.Person.name() end},
      {"Person.first_name", fn -> Faker.Person.first_name() end},
      {"Address.city", fn -> Faker.Address.city() end},
      {"Address.country", fn -> Faker.Address.country() end},
      {"Internet.email", fn -> Faker.Internet.email() end},
      {"Internet.url", fn -> Faker.Internet.url() end},
      {"UUID.v4", fn -> Faker.UUID.v4() end},
      {"Lorem.paragraph", fn -> Faker.Lorem.paragraph() end},
    ]

    results =
      Enum.map(checks, fn {name, generator} ->
        try do
          value = generator.()
          if is_nil(value) do
            {name, :nil_value}
          else
            {name, :ok}
          end
        rescue
          e -> {name, {:error, Exception.message(e)}}
        end
      end)

    failed = Enum.filter(results, fn {_name, status} -> status != :ok end)

    if failed == [] do
      Mix.shell().info("All Faker providers OK (#{length(checks)} checks passed)")
    else
      Enum.each(failed, fn {name, status} ->
        Mix.shell().error("FAIL #{name}: #{inspect(status)}")
      end)
      exit({:shutdown, 1})
    end
  end
end

Run it:

mix check_seeds
# All Faker providers OK (8 checks passed)

Step 4: Add Uniqueness Validation

When Faker is used for seeding and unique constraints exist on the database, check for duplicate generation:

# test/support/faker_uniqueness_test.exs
defmodule MyApp.FakerUniquenessTest do
  use ExUnit.Case, async: false

  @sample_size 1_000

  test "UUID.v4 generates unique values" do
    uuids = Enum.map(1..@sample_size, fn _ -> Faker.UUID.v4() end)
    unique_count = uuids |> Enum.uniq() |> length()
    assert unique_count == @sample_size,
      "UUID.v4 produced #{@sample_size - unique_count} duplicates in #{@sample_size} samples"
  end

  test "Internet.email generates sufficiently unique values" do
    emails = Enum.map(1..100, fn _ -> Faker.Internet.email() end)
    unique_count = emails |> Enum.uniq() |> length()
    # Allow some collision for email — still expect >80% unique
    assert unique_count >= 80,
      "Internet.email collision rate too high: #{unique_count}/100 unique"
  end
end

Step 5: Run Seed Checks in CI with a Heartbeat

# .github/workflows/seed-health.yml
name: Faker Seed Health

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 6 * * *'   # daily at 06:00 UTC

jobs:
  faker-smoke:
    name: Faker Provider Smoke Test
    runs-on: ubuntu-latest
    env:
      VIGILMON_FAKER_HEARTBEAT_URL: ${{ secrets.VIGILMON_FAKER_HEARTBEAT_URL }}
    steps:
      - uses: actions/checkout@v4

      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.17'
          otp-version: '27'

      - name: Cache deps
        uses: actions/cache@v3
        with:
          path: deps
          key: ${{ runner.os }}-deps-${{ hashFiles('mix.lock') }}

      - run: mix deps.get

      - name: Faker provider smoke test
        run: MIX_ENV=test mix check_seeds

      - name: ExUnit faker smoke tests
        run: mix test test/support/faker_smoke_test.exs test/support/faker_uniqueness_test.exs --color

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -fsS "$VIGILMON_FAKER_HEARTBEAT_URL"

Step 6: Create a Heartbeat Monitor in Vigilmon

  1. Sign in at vigilmon.online
  2. Click New Monitor → Heartbeat
  3. Name it Faker Seed Health — main branch
  4. Set the expected interval to 25 hours (matches the daily CI schedule)
  5. Copy the URL and store it as VIGILMON_FAKER_HEARTBEAT_URL in CI secrets

If a Faker provider crashes — because a locale module was removed, a custom provider was refactored, or an Elixir upgrade changed generator behavior — CI exits non-zero and no heartbeat fires. Vigilmon alerts you within the missed-interval window.

Step 7: Alerting

In Vigilmon, go to Notifications → New Channel → Slack:

Heartbeat missed:

🔴 MISSED: Faker Seed Health — main branch
Last healthy seed check: 30 hours ago
Action: Check CI — Faker provider crash or locale misconfiguration

Configure a 1-hour grace period so transient CI failures don't produce false pages on every push. A 25-hour heartbeat with a 1-hour grace catches any full day where seeds didn't pass.

What You Built

| What | How | |------|-----| | Provider smoke tests | Mix task: validates all configured locales and providers | | Locale coverage | ExUnit tests for Person, Address, Internet across all locales | | Uniqueness validation | ExUnit test for UUIDs and email collision rates | | CI heartbeat | Vigilmon heartbeat — alerts when seed health checks miss | | Daily schedule | GitHub Actions cron at 06:00 UTC | | Slack alerting | Vigilmon Slack notification channel |

Faker is a dev/test dependency, but broken test data corrupts CI results, staging environments, and load-test baselines. Vigilmon ensures Faker stays healthy across every push and every day.

Next Steps

  • Add locale-specific format assertions (e.g., German postal codes match \d{5}, US phone numbers match \d{3}-\d{3}-\d{4})
  • Integrate Faker health checks into your ExMachina factory module to catch bad data at factory-definition time
  • Track seed duration in CI to catch performance regressions in large seed scripts
  • Set up a Vigilmon status page for your QA team showing seed health over time

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles