tutorial

How to Monitor Thousand Island with Vigilmon

Learn how to add health checks, uptime monitoring, and alerts to your Thousand Island TCP server with Vigilmon.

How to Monitor Thousand Island with Vigilmon

Thousand Island is a pure Elixir TCP server framework that forms the foundation of Bandit, the modern Elixir HTTP server. Where Bandit handles HTTP protocol specifics, Thousand Island handles everything beneath: socket acceptance, per-connection process spawning, backpressure, connection pooling, and graceful shutdown. If you're building a custom TCP protocol, a proxy, or any network service in Elixir, Thousand Island is the idiomatic choice.

Because Thousand Island manages the socket layer directly, failures here are silent and fundamental: the server process tree might look healthy from inside the BEAM while no new connections are being accepted from the network. You need external monitoring that actually opens TCP connections rather than relying on internal process checks.

This tutorial adds production observability to a Thousand Island application:

  • A TCP health endpoint with status reporting
  • External HTTP monitoring with Vigilmon
  • Telemetry-driven connection and listener tracking
  • Alerting on connection count anomalies
  • Heartbeat monitoring for application-level workers

Step 1: Build a Thousand Island server with health reporting

Thousand Island handlers implement the ThousandIsland.Handler behaviour. Start with a minimal echo-style server that also exposes a health HTTP endpoint:

# mix.exs
defp deps do
  [
    {:thousand_island, "~> 1.0"},
    {:plug, "~> 1.14"},
    {:plug_cowboy, "~> 2.0"},  # or {:bandit, "~> 1.0"} for the health HTTP endpoint
    {:jason, "~> 1.4"}
  ]
end

Define your protocol handler:

# lib/my_app/protocol_handler.ex
defmodule MyApp.ProtocolHandler do
  use ThousandIsland.Handler

  require Logger

  @impl ThousandIsland.Handler
  def handle_connection(socket, state) do
    Logger.debug("New connection from #{inspect(ThousandIsland.Socket.peer_info(socket))}")
    {:continue, state}
  end

  @impl ThousandIsland.Handler
  def handle_data(data, socket, state) do
    # Echo the data back (replace with your protocol logic)
    ThousandIsland.Socket.send(socket, data)
    {:continue, state}
  end

  @impl ThousandIsland.Handler
  def handle_close(socket, _state) do
    Logger.debug("Connection closed: #{inspect(ThousandIsland.Socket.peer_info(socket))}")
    :ok
  end
end

Add a separate HTTP health endpoint using Plug + Bandit (or Cowboy):

# lib/my_app/health_router.ex
defmodule MyApp.HealthRouter do
  use Plug.Router

  plug :match
  plug :dispatch

  get "/health" do
    checks = %{
      thousand_island: check_server(),
      memory: check_memory()
    }

    status = if Enum.all?(checks, fn {_, v} -> v == :ok end), do: 200, else: 503
    label = if status == 200, do: "ok", else: "degraded"

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, Jason.encode!(%{status: label, checks: checks}))
  end

  match _ do
    send_resp(conn, 404, "Not found")
  end

  defp check_server do
    case Process.whereis(MyApp.Server) do
      nil -> :error
      pid ->
        if Process.alive?(pid), do: :ok, else: :error
    end
  end

  defp check_memory do
    case :memsup.get_system_memory_data() do
      [] -> :ok
      data ->
        total = Keyword.get(data, :total_memory, 1)
        free = Keyword.get(data, :free_memory, total)
        used_pct = (total - free) / total * 100
        if used_pct < 90, do: :ok, else: :error
    end
  end
end

Wire everything up in your application supervisor:

# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    port = String.to_integer(System.get_env("PORT") || "4000")
    health_port = String.to_integer(System.get_env("HEALTH_PORT") || "4001")

    children = [
      # Your Thousand Island TCP server
      {ThousandIsland, [
        handler_module: MyApp.ProtocolHandler,
        port: port,
        num_acceptors: 100,
        name: MyApp.Server
      ]},

      # Separate HTTP endpoint for health checks
      {Bandit, plug: MyApp.HealthRouter, port: health_port}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

Test your health endpoint:

mix run --no-halt
curl http://localhost:4001/health
# {"status":"ok","checks":{"thousand_island":"ok","memory":"ok"}}

# Test TCP connectivity separately
nc -zv localhost 4000
# Connection to localhost 4000 port [tcp/*] succeeded!

Step 2: Attach to Thousand Island's telemetry

Thousand Island emits :telemetry events for the full connection lifecycle. Use these to track connection counts, per-connection latency, and error rates:

# lib/my_app/server_telemetry.ex
defmodule MyApp.ServerTelemetry do
  require Logger

  def attach do
    :telemetry.attach_many(
      "thousand-island-metrics",
      [
        [:thousand_island, :listener, :start],
        [:thousand_island, :connection, :start],
        [:thousand_island, :connection, :stop],
        [:thousand_island, :connection, :error]
      ],
      &handle_event/4,
      nil
    )
  end

  def handle_event([:thousand_island, :listener, :start], _measurements, metadata, _) do
    Logger.info("Thousand Island listener started", port: metadata[:port], num_acceptors: metadata[:num_acceptors])
  end

  def handle_event([:thousand_island, :connection, :start], _measurements, metadata, _) do
    Logger.debug("Connection established", remote_ip: inspect(metadata[:remote_ip]))
  end

  def handle_event([:thousand_island, :connection, :stop], measurements, _metadata, _) do
    duration_ms = System.convert_time_unit(measurements.duration, :native, :millisecond)
    Logger.debug("Connection closed", duration_ms: duration_ms)
  end

  def handle_event([:thousand_island, :connection, :error], _measurements, metadata, _) do
    Logger.warning("Connection error", error: inspect(metadata[:error]))
  end
end

Start telemetry in your application:

def start(_type, _args) do
  MyApp.ServerTelemetry.attach()
  # ... rest of your children
end

Step 3: Enrich health checks with connection metrics

Expose live connection count and acceptor stats in your health response:

defp check_server do
  case Process.whereis(MyApp.Server) do
    nil -> :error
    pid ->
      if Process.alive?(pid) do
        # ThousandIsland exposes connection info via its API
        case ThousandIsland.connection_info(MyApp.Server) do
          {:ok, %{num_connections: n}} when n >= 0 -> :ok
          _ -> :ok  # Connected but couldn't get stats — server still alive
        end
      else
        :error
      end
  end
rescue
  _ -> :error
end

Step 4: Set up HTTP monitoring in Vigilmon

Point Vigilmon at your health endpoint:

  1. Sign up at vigilmon.online
  2. Click New Monitor → HTTP
  3. Enter http://yourdomain.com:4001/health (or the public URL if behind a proxy)
  4. Set check interval: 1 minute (paid) or 5 minutes (free)
  5. Add a keyword check for "ok" to verify the health response
  6. Save

Why Vigilmon's external probe matters for TCP servers:

An internal process check tells you the server process is alive. It does not tell you whether:

  • The port is actually listening (firewall rule change, port conflict on restart)
  • The acceptor pool is processing connections (pool saturation)
  • The network path from clients to your server is intact

Vigilmon's probes open a real TCP connection through the network stack and verify the health endpoint, catching each of these failure modes independently.

Step 5: Add a TCP port monitor

For a second line of defense, add a raw TCP port monitor to Vigilmon:

  1. Click New Monitor → TCP
  2. Enter your server's hostname and TCP port (e.g. yourdomain.com:4000)
  3. Set check interval: 1 minute
  4. Save

This monitor verifies that Thousand Island is accepting TCP connections at the socket level, independently of your application-level health check. If the HTTP health check passes but the TCP monitor fails, you have a protocol-level issue rather than an application issue.

Step 6: Alerts via Slack

In Vigilmon, go to Notifications → New Channel → Slack and paste your incoming webhook URL.

Enable the notification channel on both monitors (HTTP health and TCP port). Example alert:

🔴 DOWN: yourdomain.com:4000 (TCP)
Status: Connection refused
Detected from: EU-West, US-East
1 minute ago

Configure separate alert channels for your HTTP health endpoint and TCP port monitor so you can distinguish application-level failures from network-level failures at a glance.

Step 7: Heartbeat monitoring for protocol workers

If Thousand Island handlers drive business processes (ingesting data, processing messages), add heartbeat monitoring:

# lib/my_app/protocol_handler.ex
defmodule MyApp.ProtocolHandler do
  use ThousandIsland.Handler

  require Logger

  @impl ThousandIsland.Handler
  def handle_data(data, socket, state) do
    case process_message(data) do
      {:ok, response} ->
        ThousandIsland.Socket.send(socket, response)
        ping_heartbeat()
        {:continue, state}

      {:error, reason} ->
        Logger.error("Message processing failed: #{inspect(reason)}")
        {:continue, state}
    end
  end

  defp process_message(data), do: {:ok, data}  # your logic

  defp ping_heartbeat do
    url = Application.get_env(:my_app, :vigilmon)[:protocol_heartbeat_url]
    if url, do: Task.start(fn -> Req.get(url, receive_timeout: 5_000) end)
  end
end

Add config:

# config/runtime.exs
config :my_app, :vigilmon,
  protocol_heartbeat_url: System.get_env("VIGILMON_PROTOCOL_HEARTBEAT_URL")

In Vigilmon:

  1. New Monitor → Heartbeat
  2. Set the expected ping interval to slightly above your message rate
  3. Copy the ping URL → set as VIGILMON_PROTOCOL_HEARTBEAT_URL

Step 8: Status page

  1. Status Pages → New Status Page in Vigilmon
  2. Add your HTTP health monitor, TCP port monitor, and heartbeat monitors
  3. Share the public URL in your README and internal runbooks

README badge:

![Uptime](https://vigilmon.online/badge/your-monitor-slug)

What you've built

| What | How | |------|-----| | Health check endpoint | HealthRouter Plug with process liveness check | | Telemetry instrumentation | [:thousand_island, :connection, :*] handlers | | HTTP health monitoring | Vigilmon HTTP monitor → :4001/health | | TCP port monitoring | Vigilmon TCP monitor → :4000 | | Multi-region probes | Vigilmon multi-probe checks from global locations | | Slack downtime alerts | Vigilmon Slack notification channel | | Protocol worker heartbeats | Heartbeat ping on successful message processing | | Status page | Vigilmon public status page |

Thousand Island manages your sockets. Vigilmon verifies those sockets are reachable from the network.

Next steps

  • Export [:thousand_island, :connection, :stop] duration metrics to Prometheus or StatsD for per-connection latency histograms
  • Use Vigilmon's response time history to detect acceptor pool saturation before it causes client-visible timeouts
  • Add TCP monitors for every port your Thousand Island server listens on, not just the primary one
  • Set up escalation policies in Vigilmon so TCP failures page on-call immediately while HTTP health degradations alert the team channel

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles