How to Monitor HyperDX with Vigilmon

HyperDX is an open-source observability platform that unifies logs, traces, and metrics into a single searchable interface with session replay, alert management, and AI-powered root cause analysis. Designed as a self-hosted alternative to Datadog for teams that want full data ownership, HyperDX uses ClickHouse as its storage backend and OpenTelemetry for ingestion — the same components as SigNoz, but with a different focus on developer experience and session context. Like any complex distributed system, HyperDX can fail silently: the ingestion API can accept requests while the ClickHouse writer is backlogged, the UI can load while the query backend is returning errors, or ClickHouse can run out of disk while still responding to pings. Vigilmon monitors HyperDX from the outside so you know the moment your observability stack stops observing.

In this guide you'll set up external health monitoring for HyperDX's core components, build an ingestion pipeline heartbeat, and wire in Vigilmon alerting.

What You'll Build

• Vigilmon HTTP monitors for the HyperDX API and frontend
• An ingestion endpoint health check
• A ClickHouse storage check
• A Vigilmon heartbeat for the log/trace ingestion pipeline
• Email and Slack alert channels

Prerequisites

• HyperDX deployed via Docker Compose (see github.com/hyperdxio/hyperdx)
• HyperDX UI accessible at http://your-server:8080 (default)
• A free Vigilmon account

Step 1: Check HyperDX Component Ports

A default HyperDX Docker Compose deployment exposes:

| Component | Port | Purpose | |---|---|---| | Frontend (UI) | 8080 | Web dashboard | | API server | 8000 | REST API and ingestion | | ClickHouse | 8123 | HTTP interface | | ClickHouse | 9000 | Native TCP interface | | OTel Collector | 4317 | gRPC OTLP ingestion | | OTel Collector | 4318 | HTTP OTLP ingestion |

Verify connectivity:

curl -s http://localhost:8000/health   # API health
curl -s http://localhost:8080/         # Frontend HTML
curl -s http://localhost:8123/ping     # ClickHouse

Step 2: Monitor the HyperDX API

The HyperDX API server handles authentication, ingestion routing, alert evaluation, and query serving. If it crashes, the UI becomes a blank screen and your agents stop shipping logs.

Check the health endpoint:

curl -s http://localhost:8000/health
# → {"status":"ok"}

Set up a Vigilmon HTTP monitor:

1. Log in to Vigilmon and click New Monitor → HTTP.
2. Enter http://your-hyperdx-host:8000/health.
3. Set check interval to 60 seconds.
4. Add assertions:
   • Status code equals 200
   • Response body contains "status":"ok"
5. Save the monitor.

Step 3: Monitor the HyperDX Frontend

The HyperDX frontend is a Next.js application. Monitor it separately from the API to detect frontend-specific failures (missing static assets, crashed Next.js server, OOM kill).

curl -sf http://localhost:8080/ | grep -q "HyperDX" && echo "ok" || echo "fail"

Add a second Vigilmon monitor:

1. New Monitor → HTTP.
2. URL: http://your-hyperdx-host:8080/.
3. Check interval: 60 seconds.
4. Assertions:
   • Status code equals 200
   • Response body contains HyperDX
5. Save.

Step 4: Monitor ClickHouse

HyperDX stores all logs, traces, and metrics in ClickHouse. A ClickHouse failure means HyperDX silently drops all incoming data while appearing to accept it. Monitor ClickHouse independently:

# HTTP ping — fastest check
curl -s http://localhost:8123/ping
# → Ok.

# Query check — exercises the query engine
curl -s "http://localhost:8123/?query=SELECT+1"
# → 1

Set up the Vigilmon monitor:

1. New Monitor → HTTP.
2. URL: http://your-hyperdx-host:8123/ping.
3. Check interval: 60 seconds.
4. Assertions:
   • Status code equals 200
   • Response body contains Ok.
5. Save.

For a more thorough check that also exercises ClickHouse's query engine, use the SELECT 1 URL:

http://your-hyperdx-host:8123/?query=SELECT%201

with body assertion 1.

Step 5: Monitor the Ingestion Endpoint

HyperDX accepts logs and events via its own ingestion API (distinct from the OTel collector). Check that the ingestion endpoint is accepting requests:

# Check if the ingestion endpoint is reachable
curl -s -o /dev/null -w "%{http_code}" \
  -X POST http://localhost:8000/v1/logs/ingest \
  -H "Content-Type: application/json" \
  -d '[]'
# → 401 (unauthorized — but the endpoint is reachable)

A 401 response confirms the endpoint is alive and processing requests. Add a Vigilmon monitor with status code assertion 401:

1. New Monitor → HTTP.
2. URL: http://your-hyperdx-host:8000/v1/logs/ingest with method POST and body [].
3. Check interval: 60 seconds.
4. Assertions:
   • Status code equals 401
5. Save.

This works because a 401 response proves the API server received, parsed, and authenticated the request — it only fails if the server is down, timing out, or returning a 5xx error.

Step 6: Set Up a Heartbeat for the Ingestion Pipeline

An HTTP check confirms the ingestion endpoint is reachable, but not that data is actually flowing through to ClickHouse. Set up a heartbeat that sends a real log event and verifies end-to-end delivery.

1. In Vigilmon, go to New Monitor → Heartbeat.
2. Set expected interval to 10 minutes.
3. Copy the ping URL.

Get a HyperDX API key from Settings → API Keys. Then create the heartbeat script:

#!/usr/bin/env bash
# /usr/local/bin/hyperdx-heartbeat.sh
HEARTBEAT_URL="https://vigilmon.online/ping/YOUR_HEARTBEAT_ID"
HYPERDX_API="http://localhost:8000"
HYPERDX_API_KEY="YOUR_HYPERDX_API_KEY"

# Send a test log event
response=$(curl -sf -w "%{http_code}" -o /dev/null \
  -X POST "$HYPERDX_API/v1/logs/ingest" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $HYPERDX_API_KEY" \
  -d "[{
    \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"level\": \"info\",
    \"body\": \"vigilmon-heartbeat-check\",
    \"service\": \"vigilmon\"
  }]")

if [[ "$response" == "200" || "$response" == "204" ]]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
  echo "$(date -u) Heartbeat sent (ingestion accepted)"
else
  echo "$(date -u) Heartbeat withheld — ingestion returned $response"
fi

Schedule it:

chmod +x /usr/local/bin/hyperdx-heartbeat.sh
echo "*/10 * * * * root /usr/local/bin/hyperdx-heartbeat.sh >> /var/log/hyperdx-heartbeat.log 2>&1" \
  >> /etc/cron.d/hyperdx-heartbeat

Step 7: Monitor ClickHouse Disk Usage

ClickHouse is HyperDX's most common operational failure mode — disk exhaustion. HyperDX stores all logs, traces, and session replays in ClickHouse tables that can grow quickly in high-traffic environments. Set up a disk usage check:

#!/usr/bin/env bash
# /usr/local/bin/hyperdx-disk-check.sh
CLICKHOUSE="http://localhost:8123"
MAX_GB=50  # Alert if HyperDX data exceeds 50GB

disk_gb=$(curl -sf "$CLICKHOUSE" \
  --data "SELECT toUInt64(sum(bytes_on_disk) / pow(1024,3)) FROM system.parts WHERE database = 'default'" \
  2>/dev/null || echo "error")

if [[ "$disk_gb" == "error" ]]; then
  echo "503 ClickHouse unreachable"
  exit 1
fi

if [[ "$disk_gb" -gt "$MAX_GB" ]]; then
  echo "503 ClickHouse data exceeds ${MAX_GB}GB (current: ${disk_gb}GB)"
  exit 1
fi

echo "200 ClickHouse OK (${disk_gb}GB used)"

Wrap this in a simple HTTP server (see the Litestream tutorial for the Python wrapper pattern) and add a Vigilmon monitor.

Step 8: Alerting for Session Replay Storage

HyperDX's session replay feature stores browser session recordings in ClickHouse. These recordings are large and can exhaust storage faster than logs. If session replay storage fills up, HyperDX silently drops new recordings. Check the session replay table size:

curl -sf "http://localhost:8123/?query=SELECT+formatReadableSize(sum(bytes_on_disk))+FROM+system.parts+WHERE+table+LIKE+'%session%'"

Add this as a scheduled check or include it in your disk usage monitoring script above.

Step 9: Configure Alert Channels

In Vigilmon under Alerts → Channels:

Email — add your on-call address or SRE distribution list.

Slack — click Add Channel → Slack, paste your webhook URL. Route CRITICAL monitors to both channels.

Recommended alert policy:

| Monitor | Consecutive failures before alert | |---|---| | API health | 1 (instant alert — all users affected) | | Frontend health | 2 (brief restarts are normal) | | ClickHouse ping | 1 (storage failure = data loss) | | Ingestion endpoint | 1 (data gap starts immediately) | | Ingestion heartbeat | 1 (missing heartbeat = confirmed data gap) |

Step 10: Verify End-to-End

1. Confirm all Vigilmon monitors show UP.
2. Stop the HyperDX API container: docker compose stop hyperdx-api — verify Vigilmon fires an alert within two intervals.
3. Restart: docker compose start hyperdx-api — verify recovery notification.
4. Run the heartbeat script manually and confirm the heartbeat appears in Vigilmon.
5. Stop ClickHouse: docker compose stop clickhouse — verify the ClickHouse monitor fires an alert.

Production Checklist

• [ ] HyperDX API /health returning 200 with "status":"ok"
• [ ] Frontend returning 200 with HyperDX content
• [ ] ClickHouse /ping returning Ok.
• [ ] Ingestion endpoint check (expecting 401 to confirm liveness)
• [ ] Vigilmon HTTP monitors for all four components
• [ ] Heartbeat monitor for end-to-end ingestion pipeline (10-minute interval)
• [ ] ClickHouse disk usage monitored
• [ ] Email and Slack alert channels configured

Summary

You now have HyperDX monitored end-to-end with Vigilmon:

• HTTP monitors covering the API, frontend, ClickHouse, and ingestion endpoint
• A heartbeat confirming logs are actually flowing through the ingestion pipeline to storage
• Email and Slack alerts firing within seconds of any component failure

HyperDX gives your team unified visibility into logs, traces, and sessions. Vigilmon makes sure HyperDX itself is always visible.