tutorial

How to Monitor OpenObserve with Vigilmon (Cloud-Native Observability Platform Health)

OpenObserve (formerly ZincObserve) is a cloud-native observability platform designed to ingest, store, and query logs, metrics, and traces at scale. Built in...

OpenObserve (formerly ZincObserve) is a cloud-native observability platform designed to ingest, store, and query logs, metrics, and traces at scale. Built in Rust for performance, it offers a unified interface for your entire observability stack — and can run as a single binary, a Docker container, or a highly available Kubernetes cluster.

Like all critical infrastructure, OpenObserve must itself be monitored. If the ingestion API goes down, logs stop arriving. If the query endpoint becomes unreachable, dashboards go stale. If the web UI breaks, your team loses visibility. These failures often happen silently, with no notification unless you have external monitoring in place.

This tutorial walks through setting up Vigilmon to monitor every critical component of your OpenObserve deployment.

Why OpenObserve needs external monitoring

OpenObserve is your observability platform — but observability platforms aren't self-monitoring by nature. Common failure modes that go undetected without external checks:

  • Ingestion API crash — if the ingestion endpoint stops responding, all log shippers (Fluent Bit, Vector, OpenTelemetry Collector) keep trying to send but their data is lost or queued forever
  • Query service slowdown — storage backend pressure can cause the query API to time out, leaving dashboards loading indefinitely with no error
  • Web UI failures — the OpenObserve frontend is served separately from the API; a static asset serving failure breaks the UI while the API continues to work
  • Cluster node failure in HA mode — in a multi-node OpenObserve cluster, a node leaving the ring can cause query failures for data on that shard
  • Object storage connectivity — OpenObserve uses S3, GCS, or Azure Blob for cold storage; if this connection breaks, ingested data may be lost after the write-ahead log fills

External monitoring from Vigilmon catches all of these before your engineers notice stale dashboards.

What you'll need

  • A running OpenObserve instance (single node or cluster)
  • OpenObserve's HTTP API accessible externally (or via reverse proxy)
  • A free Vigilmon account — sign up takes 30 seconds

Step 1: Understand OpenObserve's endpoint structure

OpenObserve exposes all functionality through a single HTTP API. Default port is 5080 (HTTP) or 5443 (HTTPS). Key endpoints:

| Endpoint | Purpose | |----------|---------| | GET /healthz | Liveness check — returns {"status":"ok"} | | POST /api/{org}/logs | Log ingestion | | GET /api/{org}/logs/_search | Log query | | POST /api/{org}/metrics | Metrics ingestion | | GET /api/{org}/metrics/labels | Metrics query | | GET /api/{org}/streams | List available streams | | GET / | Web UI |

Test your OpenObserve health endpoint:

curl https://openobserve.yourdomain.com/healthz
# Expected: {"status":"ok"}

Test with authentication (OpenObserve uses basic auth or token auth):

curl -u admin:your-password https://openobserve.yourdomain.com/api/default/streams
# Expected: {"list":[...]} with your stream names

Step 2: Monitor the health endpoint

Log in to vigilmon.online and go to Monitors → New Monitor.

| Field | Value | |-------|-------| | Type | HTTP / HTTPS | | Name | OpenObserve Health | | URL | https://openobserve.yourdomain.com/healthz | | Check interval | 1 minute | | Expected status code | 200 | | Response body contains | ok |

This is your primary liveness check. If OpenObserve's HTTP server stops responding, this fires immediately.

Click Save.

Step 3: Monitor the query endpoint

The health endpoint confirms OpenObserve is running, but it doesn't confirm the query layer is functional. A crashed storage connection or corrupted index can leave the health check green while queries fail.

Add a query endpoint monitor using a low-cost API call — listing streams is lightweight and exercises the full query pipeline:

| Field | Value | |-------|-------| | Type | HTTP / HTTPS | | Name | OpenObserve Query API | | URL | https://openobserve.yourdomain.com/api/default/streams | | Check interval | 2 minutes | | Expected status code | 200 | | Response body contains | list | | HTTP headers | Authorization: Basic <base64(admin:password)> |

To generate the base64 header value:

echo -n "admin:your-password" | base64
# YWRtaW46eW91ci1wYXNzd29yZA==

Set the header in Vigilmon's advanced options: Authorization: Basic YWRtaW46eW91ci1wYXNzd29yZA==

Step 4: Monitor the web UI

OpenObserve's web UI is what your team uses day-to-day for log queries and dashboards. Monitor it separately from the API:

| Field | Value | |-------|-------| | Type | HTTP / HTTPS | | Name | OpenObserve Web UI | | URL | https://openobserve.yourdomain.com/ | | Check interval | 2 minutes | | Expected status code | 200 | | Response body contains | OpenObserve |

Step 5: Monitor ingestion API health with a heartbeat

The ingestion API is the most critical path — if it fails, logs stop arriving and you lose operational visibility. Set up a heartbeat to verify the ingestion pipeline end-to-end.

Create a Vigilmon heartbeat

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name it: OpenObserve Ingestion Pipeline
  3. Expected interval: 5 minutes
  4. Save — copy the ping URL (e.g., https://hb.vigilmon.online/ping/abc123xyz)

Create an ingestion health check script

Set up a cron job or Kubernetes CronJob that sends a test log entry to OpenObserve, verifies it was ingested, and pings Vigilmon:

#!/bin/bash
# openobserve-ingestion-check.sh

OO_URL="https://openobserve.yourdomain.com"
OO_ORG="default"
OO_STREAM="health-checks"
OO_AUTH="admin:your-password"
VIGILMON_HB="https://hb.vigilmon.online/ping/abc123xyz"

TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
TEST_PAYLOAD='[{"_timestamp":"'"$TIMESTAMP"'","level":"info","message":"vigilmon-health-check","check_id":"'"$(uuidgen)"'"}]'

# Ingest a test log
HTTP_STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
  -X POST \
  -H "Content-Type: application/json" \
  -u "$OO_AUTH" \
  "$OO_URL/api/$OO_ORG/$OO_STREAM/_json" \
  -d "$TEST_PAYLOAD")

if [ "$HTTP_STATUS" = "200" ] || [ "$HTTP_STATUS" = "204" ]; then
  curl -sf "$VIGILMON_HB"
  echo "Heartbeat sent. Ingestion healthy."
else
  echo "Ingestion check failed. HTTP $HTTP_STATUS"
  exit 1
fi

Run this as a Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: openobserve-ingestion-heartbeat
  namespace: observability
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: heartbeat
              image: curlimages/curl:latest
              command: ["/bin/sh", "/scripts/openobserve-ingestion-check.sh"]
              envFrom:
                - secretRef:
                    name: openobserve-credentials
              volumeMounts:
                - name: scripts
                  mountPath: /scripts
          volumes:
            - name: scripts
              configMap:
                name: openobserve-health-scripts
          restartPolicy: OnFailure

Step 6: Monitor cluster node status (HA deployments)

If you run OpenObserve in cluster mode (with multiple nodes), each node exposes its own health endpoint. Monitor each node individually to detect shard failures:

# Get OpenObserve node addresses in Kubernetes
kubectl get pods -n observability -l app=openobserve -o wide

For each node, add a Vigilmon monitor:

| Monitor Name | URL | Interval | |-------------|-----|----------| | OpenObserve Node 1 | https://oo-node-1.internal:5080/healthz | 2 min | | OpenObserve Node 2 | https://oo-node-2.internal:5080/healthz | 2 min | | OpenObserve Node 3 | https://oo-node-3.internal:5080/healthz | 2 min |

If you're behind an Ingress, also monitor the load balancer endpoint separately to catch LB-level failures vs. individual node failures.

Step 7: Configure alert channels

Connect Vigilmon to your notification stack.

Email

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your SRE or platform team email
  3. Assign to all OpenObserve monitors

PagerDuty / OpsGenie

For production observability platforms, connect Vigilmon to your on-call system:

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your PagerDuty Events API URL or OpsGenie webhook URL
  3. Assign to all monitors — especially the ingestion pipeline heartbeat

Vigilmon alert payload:

{
  "monitor_name": "OpenObserve Ingestion Pipeline",
  "status": "down",
  "url": "https://hb.vigilmon.online/ping/abc123xyz",
  "started_at": "2026-07-01T08:45:00Z",
  "duration_seconds": 310
}

Step 8: Create a status page

A public or internal status page for your observability platform helps your team quickly assess whether OpenObserve is the issue or the workloads being observed.

  1. Go to Status Pages → New Status Page
  2. Name it: Observability Platform
  3. Add monitors: OpenObserve Health, OpenObserve Query API, OpenObserve Web UI, OpenObserve Ingestion Pipeline
  4. Group them: UI, API, Ingestion
  5. Publish

Share the URL in your runbook's "Is OpenObserve down?" section.

Monitor summary

| Monitor | Type | Catches | |---------|------|---------| | /healthz | HTTP | Server crash, process exit | | /api/default/streams | HTTP | Query pipeline failure, storage unreachable | | Web UI / | HTTP | Frontend serving failure | | Ingestion pipeline | Heartbeat | Log ingestion broken, write path down | | TCP port 5080 | TCP | Port-level connectivity failure | | Per-node health (HA) | HTTP | Individual cluster node failure |

Useful OpenObserve diagnostics when alerts fire

# Check OpenObserve process status
systemctl status openobserve  # bare metal
kubectl get pods -n observability -l app=openobserve  # Kubernetes

# Check OpenObserve logs for errors
journalctl -u openobserve --since "10 minutes ago"  # bare metal
kubectl logs -n observability -l app=openobserve --tail=100  # Kubernetes

# Test ingestion directly
curl -u admin:password \
  -X POST \
  -H "Content-Type: application/json" \
  "https://openobserve.yourdomain.com/api/default/test-stream/_json" \
  -d '[{"_timestamp":"2026-07-01T00:00:00Z","level":"info","message":"test"}]'

# Test object storage connectivity (check OO logs for S3/GCS errors)
kubectl logs -n observability -l app=openobserve | grep -i "storage\|s3\|gcs\|azure" | tail -20

What's next

  • SSL certificate monitoring — Vigilmon automatically tracks TLS certificate expiry for your OpenObserve endpoint
  • Ingestion rate alerting — pair Vigilmon's external checks with OpenObserve's own alerting rules to catch sudden ingestion volume drops
  • Multi-region probing — Vigilmon's multi-region consensus prevents false positives when a single network path to OpenObserve has a transient issue

Keep your observability platform healthy with vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles