How to Monitor OpenTelemetry Elixir with Vigilmon

OpenTelemetry Elixir is the official OpenTelemetry SDK for the BEAM, providing vendor-neutral distributed tracing, structured metrics, and logs that export to Jaeger, Zipkin, Honeycomb, and any OTLP-compatible backend. It auto-instruments Phoenix request spans, Ecto query traces, and Oban job lifecycles without code changes.

OpenTelemetry tells you what happened inside a request. It does not tell you whether your application is reachable from the internet, whether the OTLP exporter is delivering spans, or whether your periodic metric-flush jobs are running. That is the gap Vigilmon fills.

This tutorial covers:

• A health check endpoint that exposes OTEL pipeline status alongside standard checks
• HTTP uptime monitoring with Vigilmon
• Heartbeat monitoring for OTLP export jobs and span processors
• Alerts when your observability pipeline goes dark

Why Monitor OpenTelemetry Elixir?

| Failure mode | Symptom without monitoring | |---|---| | OTLP exporter drops connection | Spans silently lost; traces disappear from backend | | Batch span processor queue full | New spans dropped; no error surfaced to app | | Phoenix app unreachable | Auto-instrumentation moot — no traffic, no traces | | Periodic metric flush job crashes | Metric gaps in Prometheus / Honeycomb | | Bad SDK config at deploy | All instrumentation silently disabled |

Step 1: Add a health endpoint with OTEL pipeline status

Install the core packages:

# mix.exs
{:opentelemetry, "~> 1.3"},
{:opentelemetry_api, "~> 1.3"},
{:opentelemetry_exporter, "~> 1.6"},
{:opentelemetry_phoenix, "~> 1.2"},
{:opentelemetry_ecto, "~> 1.2"},

Configure the OTLP exporter:

# config/runtime.exs
config :opentelemetry,
  span_processor: :batch,
  traces_exporter: :otlp

config :opentelemetry_exporter,
  otlp_protocol: :http_protobuf,
  otlp_endpoint: System.get_env("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318")

Set up auto-instrumentation in your application start callback:

# lib/my_app/application.ex
def start(_type, _args) do
  OpentelemetryPhoenix.setup(adapter: :bandit)
  OpentelemetryEcto.setup([:my_app, :repo])

  children = [
    MyApp.Repo,
    MyAppWeb.Endpoint,
  ]

  Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end

Now add a health plug that surfaces OTEL pipeline health:

# lib/my_app_web/plugs/health_check.ex
defmodule MyAppWeb.Plugs.HealthCheck do
  import Plug.Conn
  require OpenTelemetry.Tracer

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health"} = conn, _opts) do
    checks = %{
      database:   check_database(),
      otel_sdk:   check_otel_sdk(),
      memory:     check_memory()
    }

    status = if Enum.all?(checks, fn {_, v} -> v == :ok end), do: 200, else: 503

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, Jason.encode!(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      checks: checks
    }))
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp check_database do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> :ok
      _ -> :error
    end
  end

  defp check_otel_sdk do
    # Verify the SDK tracer provider is running (not a no-op)
    case :otel_tracer_provider.get_tracer(:opentelemetry, "my_app", "1.0.0") do
      {:ok, _tracer} -> :ok
      _ -> :error
    end
  rescue
    _ -> :error
  end

  defp check_memory do
    total_mb = :erlang.memory()[:total] / 1_048_576
    if total_mb < 1_500, do: :ok, else: :error
  end
end

Register before the router:

# lib/my_app_web/endpoint.ex
plug MyAppWeb.Plugs.HealthCheck
plug MyAppWeb.Router

Test it:

mix phx.server
curl http://localhost:4000/health | jq .
# {
#   "status": "ok",
#   "checks": {"database": "ok", "otel_sdk": "ok", "memory": "ok"}
# }

Step 2: Set up HTTP monitoring in Vigilmon

1. Sign up at vigilmon.online.
2. Click New Monitor → HTTP.
3. Enter https://your-app.example.com/health.
4. Set check interval to 60 seconds.
5. Add a keyword check for "ok" to catch degraded responses that still return HTTP 200.
6. Set alert threshold to 2 consecutive failures.

Vigilmon checks from multiple geographic regions. This matters for distributed Elixir deployments: a node may be reachable from one region but not another, and you want independent alerts per region rather than a single aggregate probe.

Step 3: Heartbeat monitoring for OTLP export jobs

Auto-instrumentation creates spans, but an export failure means those spans never reach your backend. A periodic job that verifies round-trip delivery gives you a heartbeat that ties OTEL health to an external signal.

# lib/my_app/otel_export_verifier.ex
defmodule MyApp.OtelExportVerifier do
  use GenServer
  require Logger
  require OpenTelemetry.Tracer

  @interval :timer.minutes(5)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  @impl true
  def init(state) do
    schedule_check()
    {:ok, state}
  end

  @impl true
  def handle_info(:check, state) do
    ctx = OpenTelemetry.Tracer.start_span("vigilmon.health_check")

    case verify_export() do
      :ok ->
        OpenTelemetry.Tracer.set_status(ctx, :ok)
        ping_heartbeat()
      {:error, reason} ->
        OpenTelemetry.Tracer.set_status(ctx, {:error, inspect(reason)})
        Logger.error("OTEL export verification failed: #{inspect(reason)}")
    end

    OpenTelemetry.Tracer.end_span(ctx)
    schedule_check()
    {:noreply, state}
  end

  defp verify_export do
    # Simple check: SDK is initialized and can create spans
    case :otel_tracer_provider.get_tracer(:opentelemetry, "my_app", "1.0.0") do
      {:ok, _} -> :ok
      _ -> {:error, :no_tracer}
    end
  rescue
    e -> {:error, e}
  end

  defp ping_heartbeat do
    url = System.get_env("VIGILMON_OTEL_HEARTBEAT_URL")
    if url do
      Task.start(fn ->
        :httpc.request(:get, {String.to_charlist(url), []}, [], [])
      end)
    end
  end

  defp schedule_check, do: Process.send_after(self(), :check, @interval)
end

Add to your supervision tree:

children = [
  MyApp.Repo,
  MyAppWeb.Endpoint,
  MyApp.OtelExportVerifier,  # ← add
]

In Vigilmon, create a Heartbeat Monitor:

1. Click New Monitor → Heartbeat.
2. Set grace period to 10 minutes.
3. Copy the ping URL into VIGILMON_OTEL_HEARTBEAT_URL.

Step 4: Key metrics to alert on

| Metric | Alert threshold | Why | |---|---|---| | /health HTTP status | Non-200 for 2 checks | App unreachable; instrumentation moot | | otel_sdk check | error in body | Tracer provider not running | | Heartbeat silence | Grace period exceeded | Export pipeline stalled | | Response latency p95 | > 2 000 ms | Batch processor backpressure |

Use Vigilmon's keyword check with a body match on "degraded" to alert on SDK status without requiring a full 503.

Step 5: Status page

1. In Vigilmon, go to Status Pages → New.
2. Add your HTTP monitor and OTEL heartbeat monitor.
3. Label them "App health" and "OTEL export pipeline".
4. Include the status page URL in your on-call runbook so engineers can check observability pipeline health independently of the traces themselves.

Conclusion

OpenTelemetry Elixir gives you deep visibility into what requests are doing. Vigilmon ensures you know when the system those requests run on — and the pipeline that collects the traces — is healthy. Together they close the loop:

• Uptime checks catch node failures that would otherwise appear as trace gaps
• Heartbeat monitors alert when your OTEL export pipeline goes dark
• Keyword checks surface degraded SDK state before it causes data loss

Start with the /health endpoint and one heartbeat monitor, then expand to per-service monitors as your OpenTelemetry deployment grows.

Get started free at vigilmon.online.