How to Monitor Paginator with Vigilmon

Paginator is a cursor-based pagination library for Ecto that implements keyset pagination as an alternative to offset pagination. Instead of LIMIT n OFFSET m, Paginator uses the values of the last row's sort columns as a cursor, enabling the database to use indexes efficiently regardless of how deep into the result set you are. This makes Paginator ideal for large datasets, infinite scroll UIs, and any scenario where offset pagination becomes prohibitively slow as page numbers grow.

When Paginator's cursor-based queries break down, you get symptoms that are harder to diagnose than simple 500 errors: infinite scroll feeds that loop back to previously seen items, feeds that skip records between cursor refreshes, or API responses that return the wrong total counts. These failures often only surface under load or after schema changes that affect the cursor columns.

Vigilmon HTTP monitors let you verify that your cursor-based pagination endpoints are responding correctly and that cursors decode and advance as expected.

Why Monitor Paginator?

Paginator failures often appear as data correctness issues rather than outright errors:

• Cursor column index missing — keyset pagination requires an index on the sort column; without it, WHERE id > $1 ORDER BY id degrades to a sequential scan, getting slower the further into the dataset you paginate
• Cursor deserialization failure — if the cursor is stored client-side (in a URL or cookie) and the schema changes (column renamed, type changed), the cursor will fail to decode on the next request
• Non-unique sort column — if you paginate by inserted_at alone without a secondary sort by id, records inserted in the same millisecond can appear on multiple pages or be skipped entirely
• Inconsistent page size — the last page of a keyset result may have fewer items; if your API consumer treats a partial page as end-of-feed, it stops paginating too early
• Backward cursor errors — if you use before: cursors for reverse pagination and the cursor is stale (the row was deleted), Paginator may return unexpected results
• Missing total count — Paginator intentionally omits total_count (it would require a full COUNT scan); if your API consumer expects it, you need to add the count separately, which adds a query

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Cursor endpoint response time | End-to-end latency for each paginated page | | Cursor decode error rate | Invalid or expired cursors from API consumers | | Page size consistency | Whether each page returns the expected number of items | | Index usage on sort columns | Whether keyset queries are using indexes | | Feed freshness | Whether new records appear in the feed within expected latency | | Cursor advance depth | How many pages deep API consumers are paginating |

Step 1: Add Paginator to Your Application

# mix.exs
defp deps do
  [
    {:paginator, "~> 1.2"}
  ]
end

# lib/my_app/repo.ex
defmodule MyApp.Repo do
  use Ecto.Repo,
    otp_app: :my_app,
    adapter: Ecto.Adapters.Postgres

  use Paginator
end

Step 2: Build a Cursor-Paginated Query

# lib/my_app/posts.ex
defmodule MyApp.Posts do
  import Ecto.Query

  @page_size 20
  @max_page_size 100

  def list_paginated(cursor_params \\ %{}, opts \\ []) do
    page_size = min(opts[:page_size] || @page_size, @max_page_size)

    query =
      from p in MyApp.Post,
        where: p.published == true,
        order_by: [desc: p.inserted_at, desc: p.id],
        preload: [:author]

    cursor_fields = [{:inserted_at, :desc}, {:id, :desc}]

    start = System.monotonic_time()

    result =
      MyApp.Repo.paginate(query,
        cursor_fields: cursor_fields,
        after: cursor_params[:after],
        before: cursor_params[:before],
        limit: page_size
      )

    duration = System.monotonic_time() - start

    :telemetry.execute(
      [:my_app, :paginator, :query],
      %{duration: duration, entry_count: length(result.entries)},
      %{
        resource: opts[:resource] || "unknown",
        has_after: !is_nil(cursor_params[:after]),
        has_before: !is_nil(cursor_params[:before])
      }
    )

    result
  end
end

Step 3: Add a Health Endpoint

# lib/my_app/paginator_health.ex
defmodule MyApp.PaginatorHealth do
  import Ecto.Query

  @timeout_ms 2_000

  def check do
    start = System.monotonic_time(:millisecond)

    result =
      Task.async(fn ->
        query =
          from p in MyApp.Post,
            where: p.published == true,
            order_by: [desc: p.id]

        MyApp.Repo.paginate(query,
          cursor_fields: [{:id, :desc}],
          limit: 1
        )
      end)
      |> Task.yield(@timeout_ms)

    duration = System.monotonic_time(:millisecond) - start

    case result do
      {:ok, page} ->
        cursor_ok = validate_cursors(page)

        %{
          status: if(cursor_ok, do: :ok, else: :degraded),
          duration_ms: duration,
          entry_count: length(page.entries),
          has_next_page: !is_nil(page.metadata.after),
          cursor_valid: cursor_ok
        }

      nil ->
        %{status: :error, reason: "pagination timeout after #{@timeout_ms}ms"}

      {:exit, reason} ->
        %{status: :error, reason: inspect(reason)}
    end
  rescue
    e -> %{status: :error, reason: inspect(e)}
  end

  defp validate_cursors(%{metadata: meta}) do
    case meta.after do
      nil ->
        true

      cursor ->
        # Verify the cursor is a non-empty string
        is_binary(cursor) and byte_size(cursor) > 0
    end
  end
end

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    paginator_check = MyApp.PaginatorHealth.check()

    status = if paginator_check.status == :ok, do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      paginator: paginator_check
    })
  end
end

# lib/my_app_web/router.ex
scope "/", MyAppWeb do
  get "/health", HealthController, :index
end

Step 4: Build a Cursor-Paginated API Controller

# lib/my_app_web/controllers/api/post_controller.ex
defmodule MyAppWeb.API.PostController do
  use MyAppWeb, :controller

  def index(conn, params) do
    cursor_params = %{
      after: params["after"],
      before: params["before"]
    }

    page_size =
      case Integer.parse(params["page_size"] || "20") do
        {n, ""} -> n
        _ -> 20
      end

    page = MyApp.Posts.list_paginated(cursor_params, page_size: page_size, resource: "post")

    conn
    |> put_status(200)
    |> json(%{
      data: render_posts(page.entries),
      pagination: %{
        after: page.metadata.after,
        before: page.metadata.before,
        limit: page_size
      }
    })
  rescue
    Paginator.InvalidCursorError ->
      conn
      |> put_status(400)
      |> json(%{error: "invalid cursor"})
  end

  defp render_posts(posts) do
    Enum.map(posts, fn post ->
      %{
        id: post.id,
        title: post.title,
        author: post.author.name,
        published_at: post.inserted_at
      }
    end)
  end
end

Step 5: Add Database Indexes for Keyset Pagination

Keyset pagination only outperforms offset pagination when the sort columns are indexed:

# priv/repo/migrations/20260101000001_add_keyset_indexes.exs
defmodule MyApp.Repo.Migrations.AddKeysetIndexes do
  use Ecto.Migration

  def change do
    # Composite index for (inserted_at DESC, id DESC) — the most common keyset pattern
    create index(:posts, [:inserted_at, :id],
             name: :posts_keyset_idx,
             options: "USING btree"
           )

    # Filtered keyset index for the common published = true filter
    create index(:posts, [:inserted_at, :id],
             name: :posts_published_keyset_idx,
             where: "published = true"
           )

    # If paginating by a score or rank column
    create index(:posts, [:score, :id], name: :posts_score_keyset_idx)
  end
end

Verify index usage with EXPLAIN:

-- This should show "Index Scan" not "Seq Scan"
EXPLAIN ANALYZE
SELECT * FROM posts
WHERE published = true
  AND (inserted_at, id) < ('2026-01-01', 12345)
ORDER BY inserted_at DESC, id DESC
LIMIT 20;

Step 6: Instrument Cursor Depth and Feed Freshness

# lib/my_app/paginator_poller.ex
defmodule MyApp.PaginatorPoller do
  use GenServer
  require Logger

  @poll_interval :timer.minutes(5)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:poll, state) do
    report_metrics()
    schedule()
    {:noreply, state}
  end

  defp report_metrics do
    check_feed_freshness()
    check_cursor_depth()
  end

  defp check_feed_freshness do
    import Ecto.Query

    newest =
      from(p in MyApp.Post, where: p.published == true, order_by: [desc: p.inserted_at], limit: 1)
      |> MyApp.Repo.one()

    if newest do
      age_seconds =
        DateTime.diff(DateTime.utc_now(), newest.inserted_at)

      :telemetry.execute(
        [:my_app, :paginator, :feed_freshness],
        %{newest_entry_age_seconds: age_seconds},
        %{resource: "post"}
      )

      if age_seconds > 3600 do
        Logger.warning("Post feed is stale: newest entry is #{div(age_seconds, 60)} minutes old")
      end
    end
  rescue
    e -> Logger.warning("Feed freshness check failed: #{inspect(e)}")
  end

  defp check_cursor_depth do
    import Ecto.Query

    # Simulate deep pagination to verify keyset performance
    start = System.monotonic_time(:millisecond)

    _result =
      from(p in MyApp.Post, where: p.published == true, order_by: [desc: p.id], limit: 1)
      |> MyApp.Repo.paginate(cursor_fields: [{:id, :desc}], limit: 20)

    duration = System.monotonic_time(:millisecond) - start

    :telemetry.execute(
      [:my_app, :paginator, :depth_check],
      %{duration_ms: duration},
      %{}
    )
  rescue
    e -> Logger.warning("Cursor depth check failed: #{inspect(e)}")
  end

  defp schedule, do: Process.send_after(self(), :poll, @poll_interval)
end

Define Telemetry.Metrics:

[
  distribution("my_app.paginator.query.duration",
    unit: {:native, :millisecond},
    tags: [:resource],
    reporter_options: [buckets: [5, 25, 50, 100, 250, 500, 1000]]
  ),
  last_value("my_app.paginator.feed_freshness.newest_entry_age_seconds",
    tags: [:resource]
  ),
  last_value("my_app.paginator.depth_check.duration_ms")
]

Step 7: Create Monitors in Vigilmon

HTTP monitor for the health endpoint:

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. Set URL to https://your-app.example.com/health
4. Set interval to 60 seconds
5. Alert condition: status 200 and body contains "paginator":{"status":"ok"}

HTTP monitor for the paginated API:

1. Click New Monitor → HTTP
2. Set URL to https://your-app.example.com/api/posts
3. Set interval to 120 seconds
4. Alert condition: status 200, response time under 500ms, and body contains "pagination"

Heartbeat monitor for pagination liveness:

# lib/my_app/paginator_heartbeat.ex
defmodule MyApp.PaginatorHeartbeat do
  use GenServer

  @heartbeat_url System.get_env("VIGILMON_PAGINATOR_HEARTBEAT_URL")
  @interval :timer.minutes(2)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    check = MyApp.PaginatorHealth.check()

    if check.status == :ok do
      send_heartbeat()
    end

    schedule()
    {:noreply, state}
  end

  defp send_heartbeat do
    if @heartbeat_url do
      :httpc.request(:get, {String.to_charlist(@heartbeat_url), []}, [{:timeout, 3000}], [])
    end
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

Create a Heartbeat monitor in Vigilmon with a 5-minute expected interval.

Step 8: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for pagination degradation:

🔴 DOWN: Paginator — MyApp
Monitor: /health returning paginator.status = "error"
Action: Check Ecto.Repo connectivity, keyset index health on sort columns, cursor serialization

Slack for slow paginated API:

⚠️ SLOW: /api/posts response > 500ms
Action: Run EXPLAIN ANALYZE on the keyset query; check for missing index on (inserted_at, id)

Slack for stale feed:

Set a Vigilmon HTTP monitor alert on the newest_entry_age_seconds metric exported via /metrics if you're using PromEx or a similar exporter. Alert when the value exceeds 3600 seconds.

Common Paginator Issues and Fixes

Sequential scan instead of index scan:

-- Check index usage
EXPLAIN ANALYZE
SELECT * FROM posts
WHERE published = true AND (inserted_at, id) < ($1, $2)
ORDER BY inserted_at DESC, id DESC
LIMIT 20;

-- If you see Seq Scan, add the composite index
CREATE INDEX CONCURRENTLY posts_keyset_idx ON posts (inserted_at DESC, id DESC) WHERE published = true;

Invalid cursor from schema change:

# Handle cursor errors gracefully in the controller
def index(conn, params) do
  # ...
rescue
  Paginator.InvalidCursorError ->
    # Return first page with a 200 (cursor expired) or 400 (invalid)
    conn
    |> put_status(400)
    |> json(%{error: "cursor_expired", message: "Please restart from the first page"})
end

Duplicate records across pages with non-unique sort:

# Always include a unique column (id) as a tiebreaker in cursor_fields
cursor_fields = [
  {:inserted_at, :desc},
  {:id, :desc}  # <-- required for deterministic ordering
]

Consumer interprets partial last page as end-of-feed:

// Include explicit has_next_page in the response
{
  "data": [...],
  "pagination": {
    "after": null,
    "before": "cursor_abc123",
    "has_next_page": false,
    "limit": 20
  }
}

What You've Built

| What | How | |------|-----| | Keyset query instrumentation | Telemetry on duration and entry count per resource | | Health endpoint with real cursor pagination | Live Paginator call verifying cursor generation | | Cursor-based API controller | Handles after/before params with error handling | | Index verification | EXPLAIN-guided composite index on sort columns | | Feed freshness monitoring | Alerts when newest entries are too old | | Liveness heartbeat | GenServer pinging Vigilmon when pagination is healthy |

Paginator makes deep pagination fast. Vigilmon makes sure your infinite scroll APIs stay reliable.

Next Steps

• Add a Grafana panel for keyset query duration by resource to track index efficiency over time
• Implement cursor expiry on the server side to avoid confusion when clients replay stale cursors
• Consider adding an optional total count endpoint (backed by PostgreSQL reltuples estimate) for UIs that need approximate totals
• Use Vigilmon's response time history to measure pagination performance at different times of day

Get started free at vigilmon.online.