How to Monitor Scrivener with Vigilmon

Scrivener is the standard offset-based pagination library for Ecto queries in Elixir. It wraps any Ecto query with paginate/3, automatically running both the page data query and a COUNT(*) query to compute total pages. Phoenix controllers and API endpoints use Scrivener to return sliced result sets with metadata like total_entries, total_pages, page_number, and page_size.

When Scrivener-based endpoints become slow or break, users get blank lists, stuck loading spinners, or API consumers receive malformed pagination responses. The most common culprit is the COUNT(*) query — on large tables without proper indexing, the total count query can take seconds while the page data query is fast. This asymmetry is invisible to endpoint-level monitoring unless you're watching query timing at the Ecto level.

Vigilmon HTTP monitors let you verify that your paginated endpoints are responding correctly and within acceptable latency budgets.

Why Monitor Scrivener?

Scrivener issues often appear as latency spikes or incorrect pagination metadata rather than outright failures:

• Slow COUNT query — Scrivener.Ecto runs a COUNT(*) on the full filtered dataset; on a 50M-row table with a complex WHERE clause and no covering index, this query can take 10+ seconds
• Incorrect total pages — if the count query is cached or race-conditioned against concurrent writes, total_pages can be stale, causing frontend pagination to show the wrong number of pages
• Page overflow — when a user requests page 500 of a 10-page result set, Scrivener returns an empty list rather than an error; API consumers may interpret this as an empty dataset
• N+1 on page data — paginated list queries that load associations without preload trigger N database queries per page, multiplying latency with page size
• Memory pressure from large page sizes — unrestricted page_size parameters (without a max) allow API consumers to request 10,000 results per page, blowing Ecto's result buffer
• Missing index on sort column — ORDER BY inserted_at DESC on a column without an index causes a full sequential scan before the LIMIT can be applied

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Paginated endpoint response time | End-to-end latency including both data and count queries | | COUNT query duration | Whether the total count query is the bottleneck | | Data query duration | Whether the page data query is fast | | Total entries count | Dataset size (growing datasets may need cursor pagination) | | Error rate on paginated endpoints | 500s from bad query parameters or Ecto errors | | Requested page vs. total pages ratio | Detect out-of-range page requests |

Step 1: Add Scrivener to Your Application

# mix.exs
defp deps do
  [
    {:scrivener_ecto, "~> 2.7"}
  ]
end

# lib/my_app/repo.ex
defmodule MyApp.Repo do
  use Ecto.Repo,
    otp_app: :my_app,
    adapter: Ecto.Adapters.Postgres

  use Scrivener, page_size: 20, max_page_size: 100
end

Step 2: Instrument Paginated Queries with Telemetry

Wrap Scrivener's paginate/3 to capture timing for both the data and count queries:

# lib/my_app/pagination.ex
defmodule MyApp.Pagination do
  require Logger

  @max_page_size 100

  def paginate(query, params, opts \\ []) do
    page = parse_page(params["page"])
    page_size = min(parse_page_size(params["page_size"]), @max_page_size)

    start = System.monotonic_time()

    result = MyApp.Repo.paginate(query, page: page, page_size: page_size)

    duration = System.monotonic_time() - start

    :telemetry.execute(
      [:my_app, :pagination, :query],
      %{duration: duration, total_entries: result.total_entries},
      %{
        resource: Keyword.get(opts, :resource, "unknown"),
        page: page,
        page_size: page_size
      }
    )

    if result.total_entries > 1_000_000 do
      Logger.warning(
        "Large paginated dataset: #{result.total_entries} entries for #{opts[:resource]}. " <>
          "Consider cursor-based pagination."
      )
    end

    result
  end

  defp parse_page(nil), do: 1
  defp parse_page(p) when is_binary(p), do: max(1, String.to_integer(p))
  defp parse_page(p) when is_integer(p), do: max(1, p)
  defp parse_page(_), do: 1

  defp parse_page_size(nil), do: 20
  defp parse_page_size(ps) when is_binary(ps), do: max(1, String.to_integer(ps))
  defp parse_page_size(ps) when is_integer(ps), do: max(1, ps)
  defp parse_page_size(_), do: 20
end

Define Telemetry.Metrics:

# lib/my_app/telemetry.ex
[
  distribution("my_app.pagination.query.duration",
    unit: {:native, :millisecond},
    tags: [:resource],
    reporter_options: [buckets: [10, 50, 100, 250, 500, 1000, 5000]]
  ),
  last_value("my_app.pagination.query.total_entries",
    tags: [:resource]
  )
]

Step 3: Add a Paginated Health Endpoint

Create a health check that exercises a real Scrivener pagination call:

# lib/my_app/pagination_health.ex
defmodule MyApp.PaginationHealth do
  import Ecto.Query

  @timeout_ms 2_000

  def check do
    start = System.monotonic_time(:millisecond)

    result =
      Task.async(fn ->
        MyApp.Repo.paginate(from(u in MyApp.User, order_by: [desc: u.id]), page: 1, page_size: 1)
      end)
      |> Task.yield(@timeout_ms)

    duration = System.monotonic_time(:millisecond) - start

    case result do
      {:ok, page} ->
        %{
          status: :ok,
          duration_ms: duration,
          total_entries: page.total_entries,
          total_pages: page.total_pages
        }

      nil ->
        %{status: :error, reason: "pagination timeout after #{@timeout_ms}ms"}

      {:exit, reason} ->
        %{status: :error, reason: inspect(reason)}
    end
  rescue
    e -> %{status: :error, reason: inspect(e)}
  end
end

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    pagination_check = MyApp.PaginationHealth.check()

    status = if pagination_check.status == :ok, do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      pagination: pagination_check
    })
  end
end

# lib/my_app_web/router.ex
scope "/", MyAppWeb do
  get "/health", HealthController, :index
end

Step 4: Use Scrivener in Phoenix Controllers

# lib/my_app_web/controllers/post_controller.ex
defmodule MyAppWeb.PostController do
  use MyAppWeb, :controller

  import Ecto.Query

  def index(conn, params) do
    query =
      from p in MyApp.Post,
        where: p.published == true,
        order_by: [desc: p.inserted_at],
        preload: [:author]

    page = MyApp.Pagination.paginate(query, params, resource: "post")

    conn
    |> put_status(200)
    |> json(%{
      data: render_posts(page.entries),
      meta: %{
        page: page.page_number,
        page_size: page.page_size,
        total_pages: page.total_pages,
        total_entries: page.total_entries
      }
    })
  end

  defp render_posts(posts) do
    Enum.map(posts, fn post ->
      %{
        id: post.id,
        title: post.title,
        author: post.author.name,
        published_at: post.inserted_at
      }
    end)
  end
end

Step 5: Optimize Scrivener Query Performance

Add indexes to support efficient pagination:

# priv/repo/migrations/20260101000000_add_pagination_indexes.exs
defmodule MyApp.Repo.Migrations.AddPaginationIndexes do
  use Ecto.Migration

  def change do
    # Index on sort column to avoid full sequential scans
    create index(:posts, [:inserted_at])

    # Composite index for filtered + sorted pagination
    create index(:posts, [:published, :inserted_at])

    # If you filter by user_id, add a partial index
    create index(:posts, [:user_id, :inserted_at])
  end
end

For expensive COUNT queries, add an estimate option to Scrivener:

# lib/my_app/pagination.ex — add count estimation for large tables
defmodule MyApp.Pagination do
  @count_estimate_threshold 100_000

  def paginate_with_estimate(query, params, opts \\ []) do
    page = parse_page(params["page"])
    page_size = parse_page_size(params["page_size"])

    # Use PostgreSQL's fast estimate for large tables
    estimated_count = get_estimated_count(opts[:table])

    if estimated_count > @count_estimate_threshold do
      # Skip exact count for performance — use estimate
      entries =
        query
        |> limit(^page_size)
        |> offset(^((page - 1) * page_size))
        |> MyApp.Repo.all()

      total_pages = ceil(estimated_count / page_size)

      %Scrivener.Page{
        entries: entries,
        page_number: page,
        page_size: page_size,
        total_entries: estimated_count,
        total_pages: total_pages
      }
    else
      MyApp.Repo.paginate(query, page: page, page_size: page_size)
    end
  end

  defp get_estimated_count(nil), do: 0

  defp get_estimated_count(table) do
    sql = "SELECT reltuples::bigint FROM pg_class WHERE relname = $1"

    case MyApp.Repo.query(sql, [table]) do
      {:ok, %{rows: [[count]]}} -> count
      _ -> 0
    end
  end
end

Step 6: Create Monitors in Vigilmon

HTTP monitor for the health endpoint:

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. Set URL to https://your-app.example.com/health
4. Set interval to 60 seconds
5. Alert condition: status 200 and body contains "pagination":{"status":"ok"}

HTTP monitor for a real paginated endpoint:

1. Click New Monitor → HTTP
2. Set URL to https://your-app.example.com/api/posts?page=1&page_size=10
3. Set interval to 120 seconds
4. Alert condition: status 200 and response time under 2000ms
5. Body must contain "total_pages"

This directly validates that Scrivener pagination is working end-to-end, including the count query.

Heartbeat monitor for pagination liveness:

# lib/my_app/pagination_heartbeat.ex
defmodule MyApp.PaginationHeartbeat do
  use GenServer

  @heartbeat_url System.get_env("VIGILMON_PAGINATION_HEARTBEAT_URL")
  @interval :timer.minutes(2)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    check = MyApp.PaginationHealth.check()

    if check.status == :ok do
      send_heartbeat()
    end

    schedule()
    {:noreply, state}
  end

  defp send_heartbeat do
    if @heartbeat_url do
      :httpc.request(:get, {String.to_charlist(@heartbeat_url), []}, [{:timeout, 3000}], [])
    end
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

Create a Heartbeat monitor in Vigilmon with a 5-minute expected interval.

Step 7: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for pagination degradation:

🔴 DOWN: Scrivener Pagination — MyApp
Monitor: /health returning pagination.status = "error"
Action: Check Ecto.Repo connectivity, large table COUNT query performance, and missing indexes

Slack for slow paginated API:

⚠️ SLOW: /api/posts response time > 2s
Action: Check EXPLAIN ANALYZE on the posts pagination query; look for sequential scans on posts.inserted_at

Common Scrivener Issues and Fixes

Slow COUNT query on large table:

-- Run EXPLAIN ANALYZE on the count query Scrivener generates
EXPLAIN ANALYZE
SELECT COUNT(*) FROM posts WHERE published = true;

-- Add a partial index
CREATE INDEX idx_posts_published ON posts (inserted_at) WHERE published = true;

Out-of-range page request returning empty list:

# Clamp page number to valid range before calling paginate
defp safe_page(page, total_pages) when page > total_pages, do: total_pages
defp safe_page(page, _), do: max(1, page)

# Or return 404 for out-of-range pages
if page.page_number > page.total_pages do
  conn |> put_status(404) |> json(%{error: "page not found"})
end

N+1 queries on paginated results:

# Always preload associations in the base query
query =
  from p in MyApp.Post,
    where: p.published == true,
    order_by: [desc: p.inserted_at],
    preload: [:author, :tags]  # preload here, not after paginate

What You've Built

| What | How | |------|-----| | Paginated query instrumentation | Telemetry on duration and total_entries per resource | | Health endpoint with real pagination | Live Scrivener call verifying the count query works | | Paginated API monitor | Vigilmon HTTP check on real endpoint with latency budget | | Page size guard | Max page size enforcement to prevent memory pressure | | Count estimation for large tables | PostgreSQL reltuples estimate to bypass slow COUNT | | Liveness heartbeat | GenServer pinging Vigilmon when pagination is healthy |

Scrivener makes Ecto pagination simple. Vigilmon makes sure your paginated APIs stay fast and available.

Next Steps

• Add Grafana panels for pagination query duration by resource to catch slow queries before users notice
• Set per-resource max_page_size limits to prevent abuse of expensive pagination endpoints
• Consider migrating high-traffic paginated endpoints to cursor-based pagination (Paginator) when datasets exceed 1M rows
• Use Vigilmon's response time history to measure pagination performance trends over time

Get started free at vigilmon.online.