How to Monitor PromEx with Vigilmon

PromEx is a community-maintained Elixir metrics library that ships pre-built Prometheus metric plugins for Phoenix, Ecto, Oban, Broadway, Absinthe, and more — plus matching Grafana dashboard templates. It hooks into :telemetry events emitted by these libraries and exposes them on a /metrics endpoint for Prometheus to scrape. When the PromEx pipeline itself stops exporting — the scrape endpoint returns an error, the BEAM emits no events, or a plugin crashes — Prometheus silently collects stale data and your dashboards look healthy when they aren't. Vigilmon adds external uptime monitoring, heartbeat checks, and alerting so you know the moment your metrics pipeline breaks.

What You'll Set Up

• HTTP monitor on the PromEx /metrics scrape endpoint with a keyword check
• Heartbeat monitor for periodic metric export jobs (e.g. Oban workers)
• Slack alerts when the scrape endpoint goes dark
• SSL monitoring for your Grafana and Prometheus servers

Prerequisites

• Elixir 1.14+ with prom_ex in mix.exs
• A running PromEx-instrumented Phoenix or Plug application
• A free Vigilmon account

Step 1: Verify Your PromEx Metrics Endpoint

PromEx registers a /metrics route automatically when you configure the PromEx.Plug or the cowboy exporter. Confirm it works before pointing Vigilmon at it:

# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    children = [
      MyApp.Repo,
      MyAppWeb.Endpoint,
      MyApp.PromEx,  # ← PromEx supervisor must be in the tree
    ]

    Supervisor.start_link(children, strategy: :one_for_one)
  end
end

# lib/my_app/prom_ex.ex
defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      PromEx.Plugins.Application,
      PromEx.Plugins.Beam,
      PromEx.Plugins.Phoenix,
      PromEx.Plugins.Ecto,
      {PromEx.Plugins.Oban, otp_app: :my_app},
    ]
  end

  @impl true
  def dashboard_assigns do
    [
      datasource_id: "prometheus",
    ]
  end

  @impl true
  def dashboards do
    [
      {:prom_ex, "application.json"},
      {:prom_ex, "beam.json"},
      {:prom_ex, "phoenix.json"},
      {:prom_ex, "ecto.json"},
    ]
  end
end

Configure the exporter in your endpoint or router:

# lib/my_app_web/router.ex
defmodule MyAppWeb.Router do
  use MyAppWeb, :router

  # Expose /metrics without authentication (restrict by IP at your proxy layer)
  forward "/metrics", PromEx.Plug, prom_ex_module: MyApp.PromEx

  # ... your normal pipelines and scopes
end

Test the endpoint locally:

mix phx.server
curl http://localhost:4000/metrics | head -20
# HELP beam_vm_memory_atom_bytes ...
# TYPE beam_vm_memory_atom_bytes gauge
# beam_vm_memory_atom_bytes 1234567

If the response contains # HELP or # TYPE lines, PromEx is exporting correctly.

Step 2: Add a Lightweight Health Endpoint Alongside /metrics

The /metrics endpoint exposes all telemetry but returns 200 even if most plugins are silently broken. Add a dedicated health endpoint that actively checks PromEx plugin state:

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    checks = %{
      prom_ex: check_prom_ex(),
      database: check_database(),
      metrics_endpoint: check_metrics_endpoint()
    }

    status = if Enum.all?(checks, fn {_, v} -> v == :ok end), do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{status: if(status == 200, do: "ok", else: "degraded"), checks: checks})
  end

  defp check_prom_ex do
    # Check the PromEx supervisor is alive
    case Process.whereis(MyApp.PromEx) do
      nil -> :error
      _pid -> :ok
    end
  end

  defp check_database do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> :ok
      _ -> :error
    end
  end

  defp check_metrics_endpoint do
    # Self-check: ensure /metrics returns a non-empty body
    case Req.get("http://localhost:#{Application.get_env(:my_app, MyAppWeb.Endpoint)[:http][:port]}/metrics",
           receive_timeout: 3_000) do
      {:ok, %{status: 200, body: body}} when byte_size(body) > 100 -> :ok
      _ -> :error
    end
  end
end

Add the route:

# lib/my_app_web/router.ex
scope "/", MyAppWeb do
  get "/health", HealthController, :index
end

Step 3: Set Up HTTP Monitoring in Vigilmon

Point Vigilmon at both endpoints:

Primary health check:

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. URL: https://yourdomain.com/health
4. Expected status: 200
5. Interval: 1 minute
6. Save

Metrics scrape endpoint (keyword check):

1. Click New Monitor → HTTP
2. URL: https://yourdomain.com/metrics
3. Add a keyword check: beam_vm_memory
4. This fails if PromEx stops emitting BEAM metrics entirely
5. Interval: 5 minutes

The keyword check is the key piece — a 200 response with an empty or truncated body means PromEx has silently stopped, and without this check Prometheus scrapes stale data indefinitely.

Step 4: Heartbeat Monitoring for Oban-Based Metric Exporters

If you use an Oban worker to push aggregated metrics to an external endpoint (e.g. a StatsD gateway or a custom dashboard), add a heartbeat so Vigilmon alerts when the job stops running:

# lib/my_app/workers/metrics_export_worker.ex
defmodule MyApp.Workers.MetricsExportWorker do
  use Oban.Worker, queue: :metrics, max_attempts: 3

  require Logger

  @impl Oban.Worker
  def perform(%Oban.Job{}) do
    with :ok <- collect_custom_metrics(),
         :ok <- push_to_gateway() do
      ping_heartbeat()
      :ok
    else
      {:error, reason} ->
        Logger.error("Metrics export failed: #{inspect(reason)}")
        {:error, reason}
    end
  end

  defp collect_custom_metrics do
    # Gather business metrics not covered by PromEx plugins
    :ok
  end

  defp push_to_gateway do
    # Push to Prometheus Pushgateway or custom endpoint
    :ok
  end

  defp ping_heartbeat do
    url = Application.get_env(:my_app, :vigilmon)[:metrics_heartbeat_url]

    if url do
      case Req.get(url, receive_timeout: 5_000) do
        {:ok, _} -> :ok
        {:error, reason} ->
          Logger.warning("Vigilmon heartbeat ping failed: #{inspect(reason)}")
      end
    end
  end
end

Configure the heartbeat URL:

# config/runtime.exs
config :my_app, :vigilmon,
  metrics_heartbeat_url: System.get_env("VIGILMON_METRICS_HEARTBEAT_URL")

In Vigilmon:

1. Click New Monitor → Heartbeat
2. Name: metrics-export-worker
3. Expected interval: match your Oban schedule (e.g. 5 minutes)
4. Copy the ping URL and set it as VIGILMON_METRICS_HEARTBEAT_URL

Now if Oban stops processing the metrics queue — due to a misconfiguration, database connection loss, or queue saturation — Vigilmon alerts you within one missed interval.

Step 5: Alert on BEAM Anomalies via Slack

PromEx exposes rich BEAM metrics: process count, memory allocations, scheduler utilization, GC pressure. These don't alert on their own — Prometheus Alert Manager handles that. But your PromEx infrastructure itself can go silent. Add a Slack channel to catch that:

1. In Vigilmon: Notifications → New Channel → Slack
2. Paste your Slack incoming webhook URL
3. Enable it on both your /health monitor and the metrics heartbeat

Alert messages look like:

🔴 DOWN: yourdomain.com/health
Status: 503 (prom_ex: error)
Detected: EU-West, US-East

And on recovery:

✅ RECOVERED: yourdomain.com/health
Downtime: 4 minutes

Step 6: SSL Monitoring for Prometheus and Grafana

Your metrics pipeline depends on TLS: Prometheus scrapes over HTTPS, Grafana serves dashboards over HTTPS. Add SSL certificate monitors:

1. New Monitor → SSL Certificate → https://prometheus.yourdomain.com
2. New Monitor → SSL Certificate → https://grafana.yourdomain.com
3. Alert threshold: 14 days before expiry

A lapsed certificate on Prometheus breaks scraping silently — Prometheus reports a scrape error in logs, but your dashboards show the last scraped value, not a gap.

What You've Built

| What | How | |------|-----| | PromEx metrics pipeline | /metrics endpoint with keyword check for beam_vm_memory | | Active health check | /health verifying PromEx supervisor is alive | | Oban exporter monitoring | Heartbeat monitor with 5-minute window | | BEAM anomaly alerts | Slack notification on health or heartbeat failure | | SSL monitoring | Certificate expiry alerts for Prometheus and Grafana |

PromEx makes your BEAM visible to Prometheus. Vigilmon makes PromEx's own health visible to you.

Next Steps

• Add a second keyword check for phoenix_router_dispatched_duration to confirm Phoenix plugin is active
• Use Vigilmon's response time history to track how long your /metrics endpoint takes to scrape — growth indicates accumulating cardinality
• Add a heartbeat per Oban queue you instrument with the Oban PromEx plugin
• Point Vigilmon at your Prometheus /api/v1/query endpoint to confirm Prometheus itself is ingesting data

Get started free at vigilmon.online.