How to Monitor Telemetry.Metrics with Vigilmon

Telemetry.Metrics is the Elixir library that turns raw :telemetry events into structured metrics — counters, gauges, distributions, and summaries — and routes them to reporting backends like Prometheus, StatsD, or LiveDashboard. It is the standard metrics layer in every Phoenix application; if you use Plug.Telemetry or Ecto.Repo, you are already emitting events that Telemetry.Metrics can capture.

The problem is that metrics pipelines fail silently. A crashed reporter process stops sending data to Prometheus without raising an exception visible to the user. A missing measurement function drops an entire event class. A backend scrape endpoint that times out appears healthy in your Phoenix app while your dashboards show no data. If no one is watching the watchers, your observability stack becomes a liability — it gives you false confidence that everything is measured.

Vigilmon heartbeat monitors keep your Telemetry.Metrics pipeline honest.

Why Monitor Telemetry.Metrics?

Metrics pipeline failures hide behind healthy application logs:

• Reporter process crash — a TelemetryMetricsPrometheus or TelemetryMetricsStatsd process crashes and is not restarted by the supervisor, silently dropping all metrics
• Silent event drop — a measurement key mismatch (:duration vs :total_duration) means events are emitted but produce no metric values
• Scrape endpoint failure — the /metrics endpoint returns 500 or times out, causing Prometheus to mark the target as down and gap all time-series
• Missing business counters — a background job counter stops incrementing because the job itself silently stopped, but the metric is zero rather than absent, masking the outage
• Backend connectivity loss — StatsD UDP packets are dropped because the host or port changed, but the Elixir process continues emitting without error

Catching these requires treating your metrics pipeline as a monitored service, not passive infrastructure.

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Reporter process uptime | Whether the Prometheus/StatsD reporter is alive and restarting | | Scrape endpoint availability | Whether /metrics returns 200 with valid content | | Event emission rate | Whether :telemetry.execute/3 calls are still reaching reporters | | Counter increment frequency | Whether business-critical counters are moving as expected | | Distribution sample count | Whether histogram metrics are receiving measurements | | Backend flush success rate | Whether StatsD/Prometheus pushes complete without error |

Step 1: Add Telemetry.Metrics to Your Project

# mix.exs
defp deps do
  [
    {:telemetry_metrics, "~> 1.0"},
    {:telemetry_metrics_prometheus_core, "~> 1.1"},
    {:telemetry_poller, "~> 1.0"},
    # rest of your deps
  ]
end

mix deps.get

Define your metrics in a dedicated module:

# lib/my_app/telemetry.ex
defmodule MyApp.Telemetry do
  import Telemetry.Metrics

  def metrics do
    [
      # HTTP request metrics (from Plug.Telemetry)
      counter("phoenix.router_dispatch.stop.count",
        tags: [:route, :method, :status]
      ),
      distribution("phoenix.router_dispatch.stop.duration",
        unit: {:native, :millisecond},
        reporter_options: [buckets: [10, 50, 100, 250, 500, 1000, 2500]]
      ),

      # Database metrics (from Ecto)
      counter("my_app.repo.query.count",
        tags: [:source, :command]
      ),
      distribution("my_app.repo.query.total_time",
        unit: {:native, :millisecond},
        reporter_options: [buckets: [5, 10, 25, 50, 100, 250]]
      ),

      # Business metrics
      counter("my_app.orders.created.count"),
      counter("my_app.payments.processed.count", tags: [:status]),
      last_value("my_app.queue.length"),

      # VM metrics (from telemetry_poller)
      last_value("vm.memory.total", unit: :byte),
      last_value("vm.total_run_queue_lengths.total"),
      counter("vm.total_run_queue_lengths.cpu")
    ]
  end
end

Step 2: Start the Reporter in Your Supervision Tree

# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    children = [
      MyApp.Repo,
      MyAppWeb.Endpoint,

      # Periodic VM metrics
      {:telemetry_poller,
        measurements: [
          {:process_info, event: [:my_app, :worker], name: MyApp.Worker},
          :memory,
          :run_queue_lengths,
          :system_counts
        ],
        period: 10_000
      },

      # Prometheus reporter
      {TelemetryMetricsPrometheus,
        metrics: MyApp.Telemetry.metrics(),
        port: 4001,
        path: "/metrics"
      }
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Verify it works locally:

curl http://localhost:4001/metrics

You should see Prometheus-formatted metric lines for each defined metric.

Step 3: Emit Business Events

Manually emit Telemetry events for domain actions:

# lib/my_app/orders.ex
defmodule MyApp.Orders do
  def create_order(attrs) do
    start_time = System.monotonic_time()

    case do_create(attrs) do
      {:ok, order} ->
        :telemetry.execute(
          [:my_app, :orders, :created],
          %{count: 1, duration: System.monotonic_time() - start_time},
          %{payment_method: order.payment_method}
        )
        {:ok, order}

      {:error, changeset} ->
        :telemetry.execute(
          [:my_app, :orders, :failed],
          %{count: 1},
          %{reason: :validation_error}
        )
        {:error, changeset}
    end
  end

  defp do_create(attrs), do: MyApp.Repo.insert(MyApp.Order.changeset(%MyApp.Order{}, attrs))
end

Step 4: Add a Metrics Health Check

Expose a health check that verifies the Prometheus reporter is alive and the scrape endpoint is responsive:

# lib/my_app_web/plugs/metrics_health.ex
defmodule MyAppWeb.Plugs.MetricsHealth do
  import Plug.Conn

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health/metrics"} = conn, _opts) do
    checks = %{
      reporter_alive: check_reporter(),
      scrape_reachable: check_scrape()
    }

    status = if Enum.all?(checks, fn {_, v} -> v == :ok end), do: 200, else: 503

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, Jason.encode!(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      checks: checks
    }))
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp check_reporter do
    case Process.whereis(TelemetryMetricsPrometheus) do
      nil -> :down
      pid -> if Process.alive?(pid), do: :ok, else: :down
    end
  end

  defp check_scrape do
    case :httpc.request(:get, {'http://localhost:4001/metrics', []}, [{:timeout, 2000}], []) do
      {:ok, {{_, 200, _}, _, _}} -> :ok
      _ -> :unreachable
    end
  end
end

Register the plug in your router:

# lib/my_app_web/router.ex
pipeline :health do
  plug MyAppWeb.Plugs.MetricsHealth
end

scope "/health" do
  pipe_through :health
  get "/metrics", MyAppWeb.HealthController, :metrics
end

Step 5: Create a Heartbeat Monitor in Vigilmon

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. Set the URL to https://your-app.example.com/health/metrics
4. Set the check interval to 60 seconds
5. Under Alert Conditions, add: response status must be 200 and response body must contain "status":"ok"

For the Prometheus scrape endpoint itself, add a second monitor:

1. Click New Monitor → HTTP
2. Set URL to https://your-app.example.com:4001/metrics
3. Set interval to 2 minutes
4. Add keyword match: response body must contain phoenix_router_dispatch_stop_count

This ensures Vigilmon alerts you if the Prometheus endpoint goes dark or starts returning unexpected content.

Step 6: Monitor Business Counter Liveness

Beyond infrastructure health, you want to alert when business counters stop moving. Use a Vigilmon heartbeat triggered from within your application:

# lib/my_app/metrics_heartbeat.ex
defmodule MyApp.MetricsHeartbeat do
  use GenServer
  require Logger

  @heartbeat_url System.get_env("VIGILMON_METRICS_HEARTBEAT_URL")
  @check_interval :timer.minutes(5)
  @counter_key [:my_app, :orders, :created, :count]

  def start_link(_opts) do
    GenServer.start_link(__MODULE__, %{last_count: 0}, name: __MODULE__)
  end

  def init(state) do
    schedule_check()
    {:ok, state}
  end

  def handle_info(:check, %{last_count: last} = state) do
    current = read_counter()

    if current > last do
      ping_vigilmon()
    else
      Logger.warning("MetricsHeartbeat: order counter has not moved in 5 minutes (count=#{current})")
    end

    schedule_check()
    {:noreply, %{state | last_count: current}}
  end

  defp read_counter do
    :telemetry_metrics_prometheus_core.scrape()
    |> String.split("\n")
    |> Enum.find("0", &String.starts_with?(&1, "my_app_orders_created_count "))
    |> String.split(" ")
    |> List.last()
    |> String.to_integer()
  rescue
    _ -> 0
  end

  defp ping_vigilmon do
    if @heartbeat_url do
      :httpc.request(:get, {@heartbeat_url |> String.to_charlist(), []}, [], [])
    end
  end

  defp schedule_check, do: Process.send_after(self(), :check, @check_interval)
end

Add this GenServer to your supervision tree and create a matching Heartbeat monitor in Vigilmon with a 10-minute expected interval.

Step 7: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack:

🔴 ALERT: Telemetry.Metrics scrape endpoint down
Monitor: MyApp /metrics (Prometheus)
Last success: 8 minutes ago
Action: Check TelemetryMetricsPrometheus supervisor — process may have crashed

PagerDuty for the business counter heartbeat:

Configure a second alert policy for the order counter heartbeat with a 15-minute threshold. A stopped order counter in production is an incident, not a warning.

Common Telemetry.Metrics Issues and Fixes

Reporter not attached — metrics show no data:

# Verify the reporter is in your supervision tree
iex> Process.whereis(TelemetryMetricsPrometheus)
#PID<0.234.0>  # ok
nil            # not started

Event name mismatch — measurement key missing:

# Event emitted with :total_time
:telemetry.execute([:my_app, :repo, :query], %{total_time: 500}, %{})

# Metric defined for :duration — will always be empty
distribution("my_app.repo.query.duration", unit: {:native, :millisecond})

# Fix: match the actual measurement key
distribution("my_app.repo.query.total_time", unit: {:native, :millisecond})

Prometheus port already in use on startup:

# Each Prometheus reporter occupies a port — ensure no two reporters share the same port
{TelemetryMetricsPrometheus, metrics: MyApp.Telemetry.metrics(), port: 4001}

What You've Built

| What | How | |------|-----| | Structured metrics pipeline | Telemetry.Metrics definitions with counters, distributions, last_value | | Prometheus scrape endpoint | TelemetryMetricsPrometheus reporter on port 4001 | | Business counter liveness | GenServer heartbeat to Vigilmon when counters move | | Metrics health endpoint | HTTP check verifying reporter process and scrape reachability | | Infrastructure alerting | Vigilmon HTTP monitor on /health/metrics | | Business alerting | Vigilmon heartbeat for order counter with 10-minute deadline |

Telemetry.Metrics makes your Elixir observability stack consistent and composable. Vigilmon makes sure the pipeline itself never goes dark.

Next Steps

• Add a Grafana dashboard fed by Prometheus with panels for each key distribution metric
• Set up TelemetryMetricsStatsd as a secondary reporter for redundant metric delivery
• Define SLO-aligned alert thresholds: p99 request duration, error rate, queue depth
• Use Vigilmon's response time history to detect gradual latency regressions

Get started free at vigilmon.online.