How to Monitor Spandex with Vigilmon

Spandex is a platform-agnostic distributed tracing library for Elixir that integrates with Datadog APM and other backends, providing trace and span creation, context propagation, and sampling across Phoenix and Plug applications. Tracing tells you how requests flow through your system; uptime monitoring tells you whether the system is reachable at all. Vigilmon complements Spandex's request-level tracing with external health checks, heartbeat monitoring for background workers, and alerting for when your services go down entirely — before any traces can even start.

What You'll Set Up

• HTTP health endpoint confirming the Spandex tracer and its reporter are connected
• Uptime monitoring for your Phoenix or Plug application
• Heartbeat monitoring for background workers that emit traces
• SSL certificate monitoring to keep TLS connections to Datadog healthy

Prerequisites

• Elixir 1.14+ with spandex and spandex_datadog (or another Spandex adapter) configured
• A running Phoenix or Plug application with Spandex middleware installed
• A free Vigilmon account

Step 1: Add a Health Endpoint That Checks the Spandex Reporter

Spandex sends completed traces to a reporter process (e.g. SpandexDatadog.ApiServer). If that reporter crashes or loses connectivity to the Datadog agent, traces are silently dropped. Add a health endpoint that checks both the application and the reporter:

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    checks = %{
      database: check_database(),
      trace_reporter: check_trace_reporter()
    }

    all_ok = Enum.all?(checks, fn {_, v} -> v == :ok end)

    conn
    |> put_status(if(all_ok, do: :ok, else: :service_unavailable))
    |> json(%{
      status: if(all_ok, do: "ok", else: "degraded"),
      checks: checks
    })
  end

  defp check_database do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> :ok
      {:error, _} -> :error
    end
  end

  defp check_trace_reporter do
    # SpandexDatadog.ApiServer is a GenServer; check it's alive
    reporter = Application.get_env(:spandex_datadog, :api_adapter, SpandexDatadog.ApiServer)

    case Process.whereis(reporter) do
      nil -> :error
      pid -> if Process.alive?(pid), do: :ok, else: :error
    end
  end
end

Add the route:

# lib/my_app_web/router.ex
scope "/health" do
  get "/", HealthController, :index
end

Verify:

curl http://localhost:4000/health
# {"status":"ok","checks":{"database":"ok","trace_reporter":"ok"}}

A degraded trace reporter should be treated as a health issue — dropped traces mean blind spots in your distributed system.

Step 2: Add the Uptime Monitor in Vigilmon

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter https://myapp.example.com/health.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Response body, enable Contains keyword and enter "ok".
7. Click Save.

The keyword check catches the case where your app returns 200 but the health check reports "degraded" — important because a degraded trace reporter won't cause request failures but will blind your distributed tracing.

Step 3: Verify Spandex Middleware Is Active

Spandex integrates with Phoenix via Spandex.Plug.StartTrace and Spandex.Plug.EndTrace, and with Ecto via telemetry callbacks. Add a lightweight check to your health endpoint confirming the tracing configuration is valid:

defp check_tracing_config do
  tracer = Application.get_env(:my_app, :tracer)

  if tracer && function_exported?(tracer, :start_trace, 2) do
    :ok
  else
    :error
  end
end

Update the checks map:

checks = %{
  database: check_database(),
  trace_reporter: check_trace_reporter(),
  tracing_config: check_tracing_config()
}

This catches misconfigured deployments where the tracer module is missing or not started — a common issue after dependency upgrades that change the Spandex adapter module name.

Step 4: Heartbeat Monitoring for Background Workers That Emit Traces

Background workers — Oban jobs, GenServers, Task.Supervisor children — emit traces for their own operations. If a worker crashes and its supervisor gives up retrying, its traces disappear from Datadog without any alert. Add a Vigilmon cron heartbeat to catch silent worker failures:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to match your worker's execution cycle.
3. Copy the heartbeat URL.

Ping Vigilmon after each successful worker run:

defmodule MyApp.Workers.ReportWorker do
  use Oban.Worker, queue: :reports

  require Logger

  @impl Oban.Worker
  def perform(%Oban.Job{}) do
    Spandex.trace("report_worker.run", [service: :report_worker]) do
      with :ok <- generate_report(),
           :ok <- deliver_report() do
        ping_vigilmon()
        Spandex.finish_trace()
        :ok
      else
        {:error, reason} ->
          Spandex.span_error(reason)
          Spandex.finish_trace()
          {:error, reason}
      end
    end
  end

  defp generate_report do
    Spandex.span("generate_report") do
      # your report generation logic
      :ok
    end
  end

  defp deliver_report do
    Spandex.span("deliver_report") do
      # your delivery logic
      :ok
    end
  end

  defp ping_vigilmon do
    url = System.get_env("VIGILMON_REPORT_HEARTBEAT_URL")
    if url do
      :httpc.request(:get, {String.to_charlist(url), []}, [{:timeout, 5000}], [])
    end
  end
end

Now if the worker crashes before completing, Vigilmon alerts you — even if Datadog shows no recent errors (because no traces were completed to send).

Step 5: Monitor the Datadog Agent TLS Certificate

Spandex's Datadog adapter sends traces to a local Datadog agent (usually at localhost:8126) or a remote agent. If your setup routes through an HTTPS proxy or uses a remote Datadog endpoint, add SSL certificate monitoring:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter the Datadog agent or intake URL your Spandex adapter targets.
3. Enable Monitor SSL certificate.
4. Set Alert when certificate expires in less than 21 days.
5. Click Save.

For local agent deployments, monitor the agent's own health endpoint instead:

http://localhost:8126/info

Add this as an HTTP monitor so Vigilmon alerts when the local Datadog agent is unreachable — which would cause all Spandex traces to queue up indefinitely until the reporter's buffer overflows.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. Set Consecutive failures before alert to 2 for HTTP monitors — transient connectivity blips between the Spandex reporter and Datadog resolve quickly.
3. Set Consecutive failures before alert to 1 for cron heartbeats — a missed heartbeat from a report worker always indicates a real failure.
4. Use Maintenance windows during planned deployments to suppress false positives from process restarts:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 5}'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health check | /health endpoint | App down, trace reporter crash, config errors | | Cron heartbeat | Heartbeat URL | Silent background worker failure | | Datadog agent HTTP | localhost:8126/info | Local agent unreachable, trace loss | | SSL certificate | Remote Datadog endpoint | TLS expiry causing trace delivery failure |

Spandex gives you deep visibility into how requests propagate through your Elixir system — Vigilmon ensures that system is reachable and its background workers are running in the first place. Together they give you both the uptime signal ("is the service up?") and the distributed trace signal ("what is the service doing?") needed for complete production observability.