tutorial

How to Monitor PropCheck with Vigilmon

Monitor your Elixir PropCheck property-based testing setup — detect generator failures, shrinking regressions, and property suite drift before they let subtle invariant bugs reach production.

How to Monitor PropCheck with Vigilmon

PropCheck is a property-based testing library for Elixir built on PropEr. Instead of writing fixed example inputs, you define generators that produce random values and properties that must hold for every generated value. PropCheck runs hundreds or thousands of inputs against each property — and when it finds a failure, it shrinks the counterexample to the smallest possible input that still breaks the property.

Teams use PropCheck to verify stateless functions (encoders, parsers, formatters), stateful systems (GenServers, caches, queues), and protocol implementations. When PropCheck works correctly, subtle edge-case bugs are caught automatically. When it's misconfigured — generators crash, properties are overly permissive, or the property suite runs too few iterations in CI — real bugs slip through as if they were never tested. Vigilmon heartbeats paired with PropCheck health checks catch these regressions automatically.

Why Monitor PropCheck?

PropCheck failures are often intermittent and configuration-sensitive:

  • Too few iterations in CI — @tag numtests: 10 speeds up CI but reduces coverage; a property that needs 500 runs to hit the edge case never fails but never catches the bug either
  • Generator crashes — a custom generator raises on certain seeds; PropCheck reports an error rather than a property failure, and the message is easy to miss in verbose CI output
  • Overly permissive properties — a property asserts result != nil when it should assert the full round-trip invariant; the property always passes but tests nothing meaningful
  • Shrinking hangs — a poorly written generator produces shrink trees that take minutes to traverse; CI times out and the property is skipped rather than failed
  • PropCheck not integrated with ExUnit — teams add PropCheck but forget to include it in their ExUnit run; properties pass in isolation but never run in the main CI pipeline

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | CI test suite exit code | Whether all properties pass across the configured number of runs | | Property suite run count | Whether the nightly deep run (higher iteration count) completed | | Generator smoke test results | Whether all custom generators produce valid values | | Shrink duration | Whether shrinking completes in reasonable time | | Failed property rate | Proportion of properties that found a counterexample | | Daily CI heartbeat | Whether property-based tests ran and passed |

Step 1: Add PropCheck to Your Project

# mix.exs
defp deps do
  [
    {:propcheck, "~> 1.4", only: [:test, :dev]},
    # rest of your deps
  ]
end

mix deps.get

Step 2: Write Basic Properties

# test/my_app/encoder_property_test.exs
defmodule MyApp.EncoderPropertyTest do
  use ExUnit.Case, async: true
  use PropCheck

  # Round-trip property: encode then decode returns original value
  property "JSON round-trip preserves integers" do
    forall n <- integer() do
      {:ok, decoded} = Jason.decode(Jason.encode!(n))
      decoded == n
    end
  end

  property "JSON round-trip preserves strings" do
    forall s <- utf8() do
      {:ok, decoded} = Jason.decode(Jason.encode!(s))
      decoded == s
    end
  end

  # Boundary property: output is always within valid range
  property "clamp/3 always returns value in [min, max]" do
    forall {value, min_val, max_val} <- {integer(), integer(), integer()} do
      lo = min(min_val, max_val)
      hi = max(min_val, max_val)
      result = MyApp.Math.clamp(value, lo, hi)
      result >= lo and result <= hi
    end
  end

  # Commutativity property
  property "addition is commutative" do
    forall {a, b} <- {integer(), integer()} do
      a + b == b + a
    end
  end
end

Step 3: Write Custom Generators

# test/support/generators.ex
defmodule MyApp.Generators do
  use PropCheck

  def valid_email do
    let {local, domain, tld} <- {
      non_empty(utf8(10, :printable)),
      non_empty(utf8(10, :printable)),
      oneof(["com", "org", "net", "io"])
    } do
      local_clean = String.replace(local, ~r/[^a-zA-Z0-9]/, "a")
      domain_clean = String.replace(domain, ~r/[^a-zA-Z0-9]/, "b")
      "#{local_clean}@#{domain_clean}.#{tld}"
    end
  end

  def positive_money_amount do
    let cents <- pos_integer() do
      cents
    end
  end

  def currency_code do
    oneof(["USD", "EUR", "GBP", "JPY", "AUD"])
  end

  def payment_request do
    let {amount, currency} <- {positive_money_amount(), currency_code()} do
      %{amount: amount, currency: currency}
    end
  end
end

Use in a property test:

# test/my_app/payment_property_test.exs
defmodule MyApp.PaymentPropertyTest do
  use ExUnit.Case, async: true
  use PropCheck

  import MyApp.Generators

  property "payment amounts are always positive after processing" do
    forall request <- payment_request() do
      {:ok, processed} = MyApp.Payments.process(request)
      processed.amount > 0
    end
  end

  property "email normalisation preserves the @ symbol" do
    forall email <- valid_email() do
      normalized = MyApp.User.normalize_email(email)
      String.contains?(normalized, "@")
    end
  end
end

Step 4: Add a Generator Smoke-Test Mix Task

# lib/mix/tasks/check_propcheck.ex
defmodule Mix.Tasks.CheckPropcheck do
  use Mix.Task

  @shortdoc "Smoke-test PropCheck generators to verify they produce valid values"

  @numtests 50

  def run(_args) do
    Code.require_file("test/support/generators.ex")
    Application.ensure_all_started(:propcheck)

    generators = [
      {"valid_email", &MyApp.Generators.valid_email/0,
       fn v -> is_binary(v) and String.contains?(v, "@") end},
      {"positive_money_amount", &MyApp.Generators.positive_money_amount/0,
       fn v -> is_integer(v) and v > 0 end},
      {"currency_code", &MyApp.Generators.currency_code/0,
       fn v -> v in ["USD", "EUR", "GBP", "JPY", "AUD"] end},
    ]

    results =
      Enum.map(generators, fn {name, gen_fn, validator} ->
        samples =
          Enum.map(1..@numtests, fn _ ->
            try do
              value = :proper_gen.pick(gen_fn.())
              {value, validator.(value)}
            rescue
              e -> {nil, {:error, Exception.message(e)}}
            end
          end)

        failures = Enum.filter(samples, fn {_v, ok} -> ok != true end)
        {name, length(failures), @numtests}
      end)

    failed = Enum.filter(results, fn {_name, failures, _total} -> failures > 0 end)

    if failed == [] do
      Mix.shell().info(
        "PropCheck generators OK — all #{length(generators)} generators produced valid values in #{@numtests} samples"
      )
    else
      Enum.each(failed, fn {name, failures, total} ->
        Mix.shell().error("FAIL #{name}: #{failures}/#{total} samples invalid")
      end)
      exit({:shutdown, 1})
    end
  end
end

Run it:

MIX_ENV=test mix check_propcheck
# PropCheck generators OK — all 3 generators produced valid values in 50 samples

Step 5: Run PropCheck in CI with a Heartbeat

# .github/workflows/propcheck-health.yml
name: PropCheck Property Test Health

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 3 * * *'   # daily at 03:00 UTC — full run with higher numtests

jobs:
  propcheck-smoke:
    name: PropCheck Generator Smoke Test
    runs-on: ubuntu-latest
    env:
      VIGILMON_PROPCHECK_HEARTBEAT_URL: ${{ secrets.VIGILMON_PROPCHECK_HEARTBEAT_URL }}
    steps:
      - uses: actions/checkout@v4

      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.17'
          otp-version: '27'

      - name: Cache deps
        uses: actions/cache@v3
        with:
          path: deps
          key: ${{ runner.os }}-deps-${{ hashFiles('mix.lock') }}

      - run: mix deps.get

      - name: Generator smoke test
        run: MIX_ENV=test mix check_propcheck

      - name: Run property tests (nightly deep run)
        run: |
          PROPCHECK_NUMTESTS=500 mix test test/my_app/encoder_property_test.exs \
            test/my_app/payment_property_test.exs --color
        env:
          PROPCHECK_NUMTESTS: 500

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -fsS "$VIGILMON_PROPCHECK_HEARTBEAT_URL"

Step 6: Create a Heartbeat Monitor in Vigilmon

  1. Sign in at vigilmon.online
  2. Click New Monitor → Heartbeat
  3. Name it PropCheck Property Test Health — main branch
  4. Set the expected interval to 25 hours (matches the daily CI schedule)
  5. Copy the URL and store it as VIGILMON_PROPCHECK_HEARTBEAT_URL in CI secrets

If a generator crashes, a property finds a counterexample on the nightly deep run, or the CI step times out from a shrinking hang, CI exits non-zero and no heartbeat fires. Vigilmon alerts you within the missed-interval window.

Step 7: Alerting

In Vigilmon, go to Notifications → New Channel → Slack:

Heartbeat missed:

🔴 MISSED: PropCheck Property Test Health — main branch
Last healthy run: 30 hours ago
Action: Check CI — property found a counterexample, generator crashed, or shrinking timed out

Configure a 1-hour grace period. A 25-hour heartbeat with a 1-hour grace catches any full day where the property test suite didn't complete cleanly.

What You Built

| What | How | |------|-----| | Basic properties | Round-trip, boundary, and commutativity properties with forall | | Custom generators | Domain-specific generators for emails, money, and currency | | Generator smoke test | Mix task: runs generators 50 times and validates output | | Nightly deep run | 500 iterations in CI — catches rare edge cases missed by quick runs | | CI heartbeat | Vigilmon heartbeat — alerts when property tests miss | | Daily schedule | GitHub Actions cron at 03:00 UTC | | Slack alerting | Vigilmon Slack notification channel |

PropCheck is most effective when run with enough iterations to explore edge cases. Without monitoring, teams reduce iteration counts under CI time pressure and property tests become useless. Vigilmon ensures property-based tests run fully and pass across every push and every day.

Next Steps

  • Use PROPCHECK_NUMTESTS environment variable to parameterise run count — lower in PR CI, higher in nightly runs
  • Write stateful properties with PropCheck.StateM for GenServer or cache behaviour verification
  • Add a second Vigilmon heartbeat for the nightly deep run with a 49 hour interval so missed nightly runs alert separately from missed PR runs
  • Use collect/2 in properties to inspect the distribution of generated inputs and ensure generators aren't biased toward trivial cases

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles