How to Monitor Floki with Vigilmon

Floki is a fast, CSS-selector-powered HTML parser for Elixir. It's the standard choice for scraping web pages, extracting structured data from HTML responses, testing rendered output in Phoenix controllers, and transforming markup in pipelines.

When Floki is embedded in a data pipeline — scraping product prices, extracting content from feeds, parsing API responses that return HTML — failures are often silent. The process doesn't crash; it just returns an empty list. This tutorial shows you how to add monitoring to Floki-based workflows with Vigilmon so you catch empty results, source changes, and pipeline staleness before they affect your data.

Why Monitor Floki?

Floki itself is a library, not a service — it doesn't crash in ways that surface to an ops dashboard. The risk is silent data degradation:

• Target HTML structure changes — your CSS selectors stop matching; Floki returns [] while the process succeeds
• Source pages return errors or redirect to login walls — HTTP layer succeeds but Floki gets no meaningful content
• Scheduled scraping jobs stop running — CI misconfiguration or infrastructure failure means data goes stale
• Encoding or malformed HTML — Floki handles most of it gracefully, but edge cases can return partial results

Monitoring strategy: validate parse results and send heartbeats. If the expected data isn't found, treat it as a failure.

Key Metrics to Track

| Metric | Why It Matters | |--------|---------------| | Parse job heartbeat | Confirms the pipeline ran and found data | | Result count | Zero results after a successful parse = selector drift | | Source fetch success rate | HTTP failures upstream of Floki | | Pipeline run duration | Sudden slowdowns indicate new anti-scraping measures | | Data freshness | Time since last successful extraction |

Step 1: Instrument Your Floki Pipeline

Here's a typical Floki scraping function with result validation:

defmodule MyApp.Scraper do
  require Logger

  @expected_minimum 1

  def scrape_products(url) do
    with {:ok, %{body: html, status: 200}} <- fetch_page(url),
         products <- parse_products(html),
         :ok <- validate_results(products) do
      {:ok, products}
    else
      {:ok, %{status: status}} ->
        Logger.error("Unexpected HTTP status #{status} from #{url}")
        {:error, :bad_status}
      {:error, :empty_results} ->
        Logger.error("Floki returned no products — selector may have drifted")
        {:error, :empty_results}
      {:error, reason} ->
        Logger.error("Scrape failed: #{inspect(reason)}")
        {:error, reason}
    end
  end

  defp fetch_page(url) do
    Req.get(url, receive_timeout: 10_000)
  end

  defp parse_products(html) do
    html
    |> Floki.parse_document!()
    |> Floki.find(".product-card")
    |> Enum.map(fn card ->
      %{
        name: card |> Floki.find(".product-name") |> Floki.text() |> String.trim(),
        price: card |> Floki.find(".price") |> Floki.text() |> String.trim()
      }
    end)
  end

  defp validate_results(products) when length(products) >= @expected_minimum, do: :ok
  defp validate_results(_), do: {:error, :empty_results}
end

Step 2: Add Heartbeat Pings to Your Pipeline

After a successful scrape-and-validate cycle, ping Vigilmon:

defmodule MyApp.ScraperWorker do
  use GenServer
  require Logger

  @interval :timer.minutes(30)
  @heartbeat_url_key :scraper_heartbeat_url

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  @impl true
  def init(state) do
    schedule_run()
    {:ok, state}
  end

  @impl true
  def handle_info(:run, state) do
    case MyApp.Scraper.scrape_products(target_url()) do
      {:ok, products} ->
        Logger.info("Scraped #{length(products)} products")
        persist_products(products)
        ping_heartbeat()

      {:error, reason} ->
        Logger.error("Scraper cycle failed: #{inspect(reason)}")
        # No ping → Vigilmon will alert after the grace window
    end

    schedule_run()
    {:noreply, state}
  end

  defp schedule_run, do: Process.send_after(self(), :run, @interval)

  defp ping_heartbeat do
    url = Application.get_env(:my_app, :vigilmon)[@heartbeat_url_key]
    if url do
      case Req.get(url, receive_timeout: 5_000) do
        {:ok, _} -> :ok
        {:error, r} -> Logger.warning("Heartbeat ping failed: #{inspect(r)}")
      end
    end
  end

  defp target_url, do: Application.get_env(:my_app, :scraper)[:target_url]
  defp persist_products(_products), do: :ok  # your persistence logic
end

Configure in runtime.exs:

config :my_app, :vigilmon,
  scraper_heartbeat_url: System.get_env("VIGILMON_SCRAPER_HEARTBEAT_URL")

Step 3: Create a Heartbeat Monitor in Vigilmon

1. Sign in at vigilmon.online
2. Click New Monitor → Heartbeat
3. Name it Floki Product Scraper
4. Set the expected interval to match your scraper cycle (e.g. 30 minutes)
5. Set a grace period: 5 minutes is usually right for a 30-minute scraper
6. Copy the ping URL, set it as VIGILMON_SCRAPER_HEARTBEAT_URL

If your GenServer crashes, the supervisor fails to restart it, or Floki returns empty results, no ping arrives and Vigilmon alerts you.

Step 4: Monitor the Target Source

The source site is an external dependency. Add an HTTP check so you know when it goes down independently of your scraper:

1. Click New Monitor → HTTP
2. URL: the target site's root or a known stable page
3. Interval: 5 minutes
4. Optional keyword check: a string that must appear in the HTML (e.g. your selector target's parent element)

When you receive a "source site is down" alert alongside "scraper heartbeat missed," you know the problem is upstream — not in your Floki code.

Step 5: Validate Results with a Custom Metric Endpoint

For richer observability, expose a metrics endpoint from your app that reports the last scrape status:

defmodule MyApp.ScraperState do
  use Agent

  def start_link(_), do: Agent.start_link(fn -> %{last_run: nil, last_count: 0} end, name: __MODULE__)

  def record_success(count) do
    Agent.update(__MODULE__, fn _ -> %{last_run: DateTime.utc_now(), last_count: count} end)
  end

  def get, do: Agent.get(__MODULE__, & &1)
end

# In your router or health plug
get "/scraper/status" do
  state = MyApp.ScraperState.get()
  age_seconds = if state.last_run, do: DateTime.diff(DateTime.utc_now(), state.last_run), else: nil
  status = if age_seconds && age_seconds < 3600, do: 200, else: 503

  conn
  |> put_resp_content_type("application/json")
  |> send_resp(status, Jason.encode!(%{
    last_run: state.last_run,
    last_count: state.last_count,
    age_seconds: age_seconds
  }))
end

Add an HTTP monitor in Vigilmon pointing at /scraper/status. A 503 means data is stale.

Step 6: Set Up Alerts

In Vigilmon, go to Notifications → New Channel and configure Slack or email:

Recommended alert rules:

• Heartbeat missed → alert data team immediately (pipeline stopped)
• Source HTTP down → alert with lower urgency (external dependency, not your code)
• Scraper status endpoint returns 503 → alert data team (data is stale beyond threshold)

Example Slack alert:

🔴 MISSED: Floki Product Scraper heartbeat
Expected every 30 minutes — last ping was 47 minutes ago
Check: scraper GenServer, ChromeDriver, source site availability

What You've Built

| What | How | |------|-----| | Result validation | validate_results/1 guard in scraping pipeline | | Pipeline liveness | Heartbeat ping on successful scrape cycle | | Missed run alerts | Vigilmon Heartbeat monitor | | Source site health | Vigilmon HTTP monitor on target URL | | Data freshness check | HTTP monitor on /scraper/status with 503 on stale data | | Slack alerts | Vigilmon notification channel |

Floki is silent when selectors drift. Vigilmon gives it a voice.

Next Steps

• Add per-selector result count tracking to distinguish "site restructured" from "site down"
• Use Vigilmon's response time history to detect when the target site starts serving slower (anti-scraping throttles)
• Add heartbeat monitors for each independent data source if your pipeline scrapes multiple sites
• Consider a status page for internal stakeholders who depend on your scraped data

Get started free at vigilmon.online.