How to Monitor SigNoz with Vigilmon

SigNoz is an open-source full-stack APM and observability platform that combines distributed tracing, metrics, and structured log management in a single self-hosted deployment. It uses OpenTelemetry as its native ingestion protocol and ClickHouse as its storage backend — making it a popular open-source alternative to Datadog and New Relic. But SigNoz is itself a complex distributed system: if the OpenTelemetry collector drops spans, if ClickHouse runs out of disk, or if the query service crashes, your observability stack goes dark. Vigilmon monitors SigNoz from the outside so you know when the tool that watches everything else stops working.

In this guide you'll set up external health monitoring for SigNoz's core components — the UI, the query service, the OpenTelemetry collector, and ClickHouse — and wire in Vigilmon alerts so your team is notified before anyone opens a ticket asking "why are there no traces?"

What You'll Build

• A Vigilmon HTTP monitor for the SigNoz frontend and query service
• An OTel collector health check
• A ClickHouse readiness check
• A Vigilmon heartbeat for the trace ingestion pipeline
• Email and Slack alert channels

Prerequisites

• SigNoz deployed via Docker Compose or Kubernetes (see github.com/SigNoz/signoz)
• SigNoz UI accessible at http://your-server:3301 (default)
• A free Vigilmon account

Step 1: Check SigNoz Component Ports

A default Docker Compose SigNoz deployment exposes these ports:

| Component | Port | Purpose | |---|---|---| | Frontend (UI) | 3301 | Web dashboard | | Query Service | 8080 | REST API for dashboards and alerts | | OTel Collector | 4317 | gRPC OTLP ingestion | | OTel Collector | 4318 | HTTP OTLP ingestion | | ClickHouse | 8123 | HTTP interface | | ClickHouse | 9000 | Native TCP interface |

Verify each is reachable:

curl -s http://localhost:3301/          # SigNoz UI — returns HTML
curl -s http://localhost:8080/api/v1/health  # Query service health
curl -s http://localhost:8123/ping      # ClickHouse HTTP ping

Step 2: Monitor the SigNoz Query Service

The query service is the API backend for the SigNoz UI. It handles all dashboard queries, alert evaluation, and trace lookups. A failed query service makes the entire UI nonfunctional even if ClickHouse and the OTel collector are healthy.

The query service exposes a health endpoint:

curl -s http://localhost:8080/api/v1/health
# → {"status":"ok"}

Set up a Vigilmon HTTP monitor:

1. Log in to Vigilmon and click New Monitor → HTTP.
2. Enter http://your-signoz-host:8080/api/v1/health.
3. Set check interval to 60 seconds.
4. Add assertions:
   • Status code equals 200
   • Response body contains "status":"ok"
5. Save the monitor.

If you expose SigNoz behind a reverse proxy (nginx, Caddy, Traefik), use the public URL instead and enable SSL monitoring as well.

Step 3: Monitor the SigNoz Frontend

The SigNoz frontend is a React SPA served by a Node.js server. Monitoring it separately from the query service catches cases where the frontend crashes but the API is still responding.

curl -sf http://localhost:3301/ | grep -q "SigNoz" && echo "ok" || echo "fail"

Add a second Vigilmon HTTP monitor:

1. New Monitor → HTTP.
2. URL: http://your-signoz-host:3301/.
3. Set check interval to 60 seconds.
4. Add assertions:
   • Status code equals 200
   • Response body contains SigNoz
5. Save.

Step 4: Monitor ClickHouse

ClickHouse is SigNoz's storage backend for all traces, metrics, and logs. If ClickHouse becomes unavailable or runs out of disk, SigNoz stops ingesting data and querying history. ClickHouse exposes a simple HTTP ping:

curl -s http://localhost:8123/ping
# → Ok.

Set up a Vigilmon HTTP monitor:

1. New Monitor → HTTP.
2. URL: http://your-signoz-host:8123/ping.
3. Set check interval to 60 seconds.
4. Add assertions:
   • Status code equals 200
   • Response body contains Ok.
5. Save.

For a more thorough ClickHouse check, query a lightweight system table:

curl -s "http://localhost:8123/?query=SELECT+1" 
# → 1

Use this URL as your Vigilmon monitor endpoint if you want to exercise the query engine, not just the HTTP listener.

Step 5: Monitor the OpenTelemetry Collector

SigNoz ships with an OpenTelemetry collector that receives spans, metrics, and logs from your instrumented applications and writes them to ClickHouse. If the collector crashes or its pipeline backs up, you lose observability data with no warning to users.

The OTel collector health check endpoint:

curl -s http://localhost:13133/
# → {"status":"Server available","upSince":"...","uptime":"..."}

Port 13133 is the default health check extension port in the OTel collector. Verify it is enabled in your collector config:

# otel-collector-config.yaml
extensions:
  health_check:
    endpoint: "0.0.0.0:13133"

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]

Set up the Vigilmon monitor:

1. New Monitor → HTTP.
2. URL: http://your-signoz-host:13133/.
3. Set check interval to 60 seconds.
4. Add assertions:
   • Status code equals 200
   • Response body contains Server available
5. Save.

Step 6: Set Up a Heartbeat for Trace Ingestion

An external HTTP check confirms the collector is running, but doesn't confirm that traces are actually flowing through the pipeline end-to-end. Set up a heartbeat that sends a test span and verifies it appears in SigNoz.

1. In Vigilmon, go to New Monitor → Heartbeat.
2. Set expected interval to 10 minutes.
3. Copy the ping URL.

Create a script that sends a synthetic trace and pings the heartbeat:

#!/usr/bin/env bash
# /usr/local/bin/signoz-pipeline-heartbeat.sh
HEARTBEAT_URL="https://vigilmon.online/ping/YOUR_HEARTBEAT_ID"
OTEL_ENDPOINT="http://localhost:4318/v1/traces"

# Send a minimal OTLP/HTTP span
SPAN_ID=$(openssl rand -hex 8)
TRACE_ID=$(openssl rand -hex 16)
NOW_NS=$(($(date +%s) * 1000000000))

payload=$(cat <<JSON
{
  "resourceSpans": [{
    "resource": {
      "attributes": [{"key": "service.name", "value": {"stringValue": "vigilmon-heartbeat"}}]
    },
    "scopeSpans": [{
      "scope": {"name": "vigilmon"},
      "spans": [{
        "traceId": "${TRACE_ID}",
        "spanId": "${SPAN_ID}",
        "name": "heartbeat-check",
        "kind": 1,
        "startTimeUnixNano": "${NOW_NS}",
        "endTimeUnixNano": "$((NOW_NS + 1000000))",
        "status": {"code": 1}
      }]
    }]
  }]
}
JSON
)

response=$(curl -sf -w "%{http_code}" -o /dev/null \
  -X POST "$OTEL_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d "$payload")

if [[ "$response" == "200" ]]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
  echo "$(date -u) Heartbeat sent (span $SPAN_ID accepted)"
else
  echo "$(date -u) Heartbeat withheld — OTLP endpoint returned $response"
fi

Add to cron:

chmod +x /usr/local/bin/signoz-pipeline-heartbeat.sh
echo "*/10 * * * * root /usr/local/bin/signoz-pipeline-heartbeat.sh >> /var/log/signoz-heartbeat.log 2>&1" \
  >> /etc/cron.d/signoz-heartbeat

Step 7: Monitor ClickHouse Disk Usage

ClickHouse is SigNoz's most common failure mode — not crashes, but running out of disk. SigNoz stores all traces and logs in ClickHouse tables that grow without bound unless you configure retention. Monitor disk usage before it becomes a problem:

#!/usr/bin/env bash
# /usr/local/bin/clickhouse-disk-check.sh
CLICKHOUSE="http://localhost:8123"
HEALTH_PORT=9102
WARNING_PERCENT=80
CRITICAL_PERCENT=90

disk_used=$(curl -sf "$CLICKHOUSE" \
  --data "SELECT toUInt64(sum(bytes_on_disk) / (1024*1024*1024)) FROM system.parts" \
  2>/dev/null || echo "error")

if [[ "$disk_used" == "error" ]]; then
  echo "503 ClickHouse unreachable"
  exit 1
fi

echo "ClickHouse data: ${disk_used}GB"

Expose this as a simple HTTP health endpoint (similar to Step 2 of the Litestream tutorial) and add a Vigilmon monitor with an assertion on the response body.

Step 8: Configure Alert Channels

In Vigilmon under Alerts → Channels:

Email — add your on-call or SRE distribution list.

Slack — click Add Channel → Slack, paste your webhook URL. Route CRITICAL monitors to both channels.

Recommended alert policy per monitor:

| Monitor | Consecutive failures before alert | |---|---| | Query service health | 1 (instant alert — UI is broken) | | Frontend health | 2 (brief restarts are normal) | | ClickHouse ping | 1 (storage failure = data loss) | | OTel collector | 2 (brief restarts OK) | | Ingestion heartbeat | 1 (missing heartbeat = data gap) |

Step 9: Verify End-to-End

1. Confirm all Vigilmon monitors show UP.
2. Stop the SigNoz query service container: docker compose stop query-service — verify Vigilmon fires an alert within two check intervals.
3. Restart: docker compose start query-service — verify recovery notification.
4. Run the heartbeat script manually and confirm the heartbeat is received in Vigilmon.

Production Checklist

• [ ] Query service /api/v1/health returning 200 with "status":"ok"
• [ ] Frontend returning 200 with SigNoz body content
• [ ] ClickHouse /ping returning Ok.
• [ ] OTel collector health check at port 13133 returning Server available
• [ ] Vigilmon HTTP monitors for all four components
• [ ] Heartbeat monitor for end-to-end trace ingestion (10-minute interval)
• [ ] ClickHouse disk usage monitored
• [ ] Email and Slack alert channels configured

Summary

You now have SigNoz monitored end-to-end with Vigilmon:

• HTTP monitors covering the query service, frontend, ClickHouse, and OTel collector
• A heartbeat confirming traces are actually flowing through the ingestion pipeline
• Email and Slack alerts firing within seconds of any component failure

SigNoz watches your application. Vigilmon makes sure SigNoz itself never goes dark unnoticed.