tutorial

Monitoring Highlight.run with Vigilmon

Highlight.run is an open source session replay and error monitoring platform — here's how to monitor its web UI, API server, ClickHouse storage, Redis, session ingestion rate, and TLS certificates with Vigilmon.

Highlight.run is a fully self-hosted, open source platform for session replay, JavaScript error tracking, and frontend log collection — giving teams the capabilities of FullStory or LogRocket without sending user session data to a third party. When you run Highlight.run on your own infrastructure you operate a multi-service stack: a Go API server, a React web frontend, ClickHouse for compressed session storage, and Redis for job queuing and caching. Any of these services failing silently means engineers lose access to debug data exactly when they need it most. Vigilmon gives you end-to-end visibility across every service in the Highlight.run stack.

What You'll Set Up

  • HTTP uptime monitor for the Highlight.run web UI (port 3000)
  • API server availability monitor (port 8082)
  • ClickHouse TCP connectivity monitor
  • Redis connectivity monitor
  • Cron heartbeat for session ingestion pipeline health
  • SSL certificate expiry alerts for the web UI endpoint
  • Alert channels with appropriate thresholds

Prerequisites

  • Highlight.run self-hosted instance deployed via Docker Compose or Kubernetes
  • Web UI accessible on port 3000 (or reverse-proxied to HTTPS)
  • API server accessible on port 8082 (or reverse-proxied)
  • ClickHouse running (default port 9000 native, 8123 HTTP)
  • Redis running (default port 6379)
  • A free Vigilmon account

Step 1: Monitor the Web UI (Port 3000)

The Highlight.run dashboard is how your engineering team browses session replays, reviews JavaScript errors, and searches frontend logs. If the React frontend is down, the entire debugging workflow stops — even if the API is still ingesting events in the background.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. URL: https://highlight.yourdomain.com (or http://your-server-ip:3000).
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Keyword check, enter Highlight to verify the page actually loaded and isn't an nginx error page.
  7. Enable Monitor SSL certificate and set Alert when certificate expires in less than 21 days.
  8. Click Save.

Downtime on this monitor means engineers cannot access any session replays, error details, or logs — even if the underlying data is still being collected.

Step 2: Monitor the API Server (Port 8082)

The Go API server is the ingestion endpoint for the Highlight.run SDKs embedded in your web applications. Every session event, JavaScript error, and frontend log line is sent here. If the API server goes down, all that observability data is lost for the duration of the outage — there is no replay or recovery from the SDK client side.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-server-ip:8082/healthz (or the health endpoint for your Highlight.run version; fall back to http://your-server-ip:8082/ if no dedicated health route exists).
  3. Check interval: 1 minute
  4. Expected HTTP status: 200
  5. Click Save.

Alternatively, monitor via TCP if the API server does not expose an HTTP health route:

  1. Click Add Monitor → TCP Port.
  2. Host: your-server-ip
  3. Port: 8082
  4. Check interval: 1 minute
  5. Click Save.

Step 3: Monitor ClickHouse Connectivity

Highlight.run stores compressed session replay data in ClickHouse. If ClickHouse becomes unavailable, the API server can still accept incoming events but cannot persist them — sessions are silently dropped. Monitor both the TCP port and the HTTP interface:

TCP monitor (native protocol):

  1. Click Add Monitor → TCP Port.
  2. Host: localhost (or your ClickHouse host).
  3. Port: 9000
  4. Check interval: 1 minute
  5. Click Save.

HTTP interface monitor:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://localhost:8123/ping
  3. Expected HTTP status: 200
  4. Under Keyword check, enter Ok. — the ClickHouse ping endpoint returns exactly this string when healthy.
  5. Check interval: 1 minute
  6. Click Save.

The HTTP ping monitor is more reliable than TCP alone: ClickHouse can accept TCP connections while being in a degraded state (e.g., during a disk-full condition) that the HTTP handler will surface correctly.

Step 4: Monitor Redis Connectivity

Redis is Highlight.run's job queue and caching layer. Session processing tasks — transcoding replay segments, enriching error events with source maps — are dispatched through Redis queues. If Redis goes down, the job queue freezes and unprocessed sessions accumulate.

  1. Click Add Monitor → TCP Port.
  2. Host: localhost (or your Redis host).
  3. Port: 6379
  4. Check interval: 1 minute
  5. Click Save.

Pair this TCP check with the session ingestion heartbeat in Step 5. Redis TCP availability confirms the port is open; the heartbeat confirms sessions are actually flowing end-to-end through the pipeline.

Step 5: Heartbeat Monitor for Session Ingestion Pipeline

TCP port checks confirm individual services are running but do not verify that the full ingestion pipeline — SDK → API → ClickHouse via Redis queues — is functioning. A cron heartbeat catches silent failures like a stuck worker, a full Redis queue, or a ClickHouse write rejection that leaves the port up but the data flow broken.

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 10 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
  4. Add a lightweight script to your server that queries ClickHouse for recent session activity and pings the heartbeat if data is flowing:
#!/bin/bash
# /opt/highlight/check-ingestion.sh
RECENT=$(clickhouse-client --query "
  SELECT count() FROM sessions
  WHERE created_at > now() - INTERVAL 15 MINUTE
" 2>/dev/null)

if [ "$RECENT" -gt "0" ] 2>/dev/null; then
  curl -s https://vigilmon.online/heartbeat/abc123
fi

Schedule it via cron:

*/10 * * * * /opt/highlight/check-ingestion.sh

If no sessions have arrived in ClickHouse in the last 15 minutes and no heartbeat reaches Vigilmon, you get an alert indicating the ingestion pipeline is broken — before your team notices that session replays have stopped appearing in the dashboard.

Step 6: Monitor Error Ingestion Health

JavaScript error events follow a separate processing path from session replays. A dedicated monitor on the error ingestion endpoint confirms that error capture is functioning independently of session recording:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-server-ip:8082/public (the Highlight.run GraphQL ingestion endpoint).
  3. Method: POST
  4. Expected HTTP status: 200 or 400 — a 400 on an empty POST body confirms the server is processing requests; a 502 or timeout indicates the API server is unreachable.
  5. Check interval: 2 minutes
  6. Click Save.

Step 7: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
  2. For the web UI and API server monitors, set Consecutive failures before alert to 2 — container restarts during deployments cause brief probe failures.
  3. For ClickHouse and Redis TCP monitors, set Consecutive failures to 1 — storage or queue loss is immediately critical and warrants instant alerting.
  4. For the session ingestion heartbeat, Vigilmon alerts automatically after the expected interval passes with no ping received.
  5. For the SSL certificate monitor, set alerts to 21 days before expiry to leave time for renewal troubleshooting.

Docker Compose Health Check Integration

If you deployed Highlight.run via Docker Compose, align your container health checks with the Vigilmon monitors:

services:
  backend:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8082/healthz"]
      interval: 30s
      timeout: 10s
      retries: 3

  frontend:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000"]
      interval: 30s
      timeout: 10s
      retries: 3

  clickhouse:
    healthcheck:
      test: ["CMD", "clickhouse-client", "--query", "SELECT 1"]
      interval: 30s
      timeout: 10s
      retries: 3

Docker's internal health checks drive container restarts. Vigilmon provides the external view — confirming services are reachable from outside the container network and that the full user-facing flow is working.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web UI | https://highlight.domain.com | Dashboard down, frontend crash | | API server | :8082/healthz | Ingestion endpoint unavailable | | ClickHouse HTTP | :8123/ping | Storage degraded or full | | ClickHouse TCP | :9000 | Database unreachable | | Redis TCP | :6379 | Job queue and cache unavailable | | Ingestion heartbeat | Heartbeat URL | Session pipeline silently broken | | Error endpoint | :8082/public | Error capture path broken | | SSL certificate | Web UI domain | TLS expiry on dashboard |

Highlight.run gives you full control over session replay and error monitoring data — but self-hosting means you also own the reliability of the platform your debugging workflow depends on. With Vigilmon watching the web UI, API server, ClickHouse, Redis, and the end-to-end ingestion pipeline, you get alerts before a silent service failure leaves your engineers blind to production JavaScript errors.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles