How to Monitor Opstrace with Vigilmon (HTTP + TCP + Alerts)

Opstrace is an open-source observability platform that bundles Prometheus, Loki, Cortex, and Grafana into a single Kubernetes-native installation, giving teams a self-hosted alternative to commercial observability SaaS. Its API gateway acts as the single ingress point for all queries, dashboards, and metric ingestion — if it goes down, engineers lose visibility into every other system they monitor. Vigilmon fills the blind spot that Opstrace itself cannot fill: external uptime and latency checks that keep watching even when your observability stack is the thing that's broken. You can add your first monitor for free, no credit card required.

Why Opstrace needs external monitoring

API gateway crash silences all dashboards. Opstrace routes all Grafana queries and Prometheus remote-write traffic through a single ingress gateway. When that gateway crashes or the TLS certificate expires, every dashboard goes blank simultaneously. Kubernetes liveness probes may still report the pod as healthy if the HTTP listener has not exited cleanly, so PagerDuty and Alertmanager — both of which depend on the same stack — never fire.
Cortex compactor OOM kills block querying silently. The Cortex compactor handles long-term metric storage. An out-of-memory kill leaves the compactor pod in a CrashLoopBackOff while the query path returns stale or incomplete data with HTTP 200 responses. Engineers querying dashboards see flat lines rather than errors, and the failure is invisible without an external synthetic check.
Loki distributor backpressure drops log lines without errors. When the Loki distributor is overwhelmed by log volume, it begins dropping ingest requests and returns HTTP 429 to log shippers. Log pipelines back off silently, and the observability platform appears healthy in Kubernetes while hours of logs go missing. An external monitor polling the Loki push endpoint detects the non-200 response immediately.
Tenant authentication service outage locks out all users. Opstrace implements per-tenant authentication via a dedicated service that validates tokens before forwarding requests to Cortex or Loki. If that service becomes unreachable, all API calls return HTTP 401 regardless of credential validity. Because the auth service itself generates no user-visible alerts, the outage looks like a user error until it is reported manually.
Node eviction disrupts the Prometheus ruler, breaking alerting. Opstrace runs the Prometheus ruler as a StatefulSet. If a node is evicted under memory pressure, the ruler pod may take several minutes to reschedule, during which no alerting rules evaluate. The teams that rely on Opstrace to alert them are unaware they have lost alert coverage — exactly when external monitoring matters most.

What you'll need

A running Opstrace cluster with its API gateway accessible over HTTPS (cloud load balancer, NodePort, or Ingress)
The public or LAN hostname for your Opstrace tenant endpoint, for example https://system.opstrace.example.com
A free Vigilmon account — monitors start running in under a minute with no credit card required

Step 1: Expose Opstrace's health endpoint

Verify that your Opstrace installation is running and locate the API gateway URL before adding Vigilmon monitors.

# Check all Opstrace pods are running
kubectl get pods -n opstrace

# Sample output:
# NAME                                      READY   STATUS    RESTARTS   AGE
# api-gateway-7d9f6b5c8-xk2pq              1/1     Running   0          3d
# cortex-compactor-0                        1/1     Running   0          3d
# cortex-ingester-0                         1/1     Running   0          3d
# loki-distributor-6b8d7f9c4-m3nw7         1/1     Running   0          3d
# prometheus-operator-5c7d8b6f9-r4vt2      1/1     Running   0          3d
# grafana-84f9c7d6b-p8ks1                  1/1     Running   0          3d

# Get the external hostname or IP of the API gateway
kubectl get ingress -n opstrace

# Sample output:
# NAME          CLASS   HOSTS                          ADDRESS         PORTS     AGE
# api-gateway   nginx   system.opstrace.example.com   203.0.113.42    80, 443   3d

# Test the system health endpoint (returns 200 when the API gateway is up)
curl -I https://system.opstrace.example.com/-/ready

# Sample output:
# HTTP/2 200
# content-type: application/json
# x-opstrace-tenant: system

# Test the Cortex ready endpoint through the gateway
curl -I https://system.opstrace.example.com/api/v1/health

# Sample output:
# HTTP/2 200
# content-type: text/plain; charset=utf-8

If the Ingress is not yet configured, create one that exposes the API gateway service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-gateway
  namespace: opstrace
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - system.opstrace.example.com
      secretName: opstrace-tls
  rules:
    - host: system.opstrace.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-gateway
                port:
                  number: 443

Apply it and confirm the ADDRESS field is populated before proceeding:

kubectl apply -f opstrace-ingress.yaml
kubectl get ingress -n opstrace --watch

Step 2: Set up HTTP monitoring in Vigilmon

Add an HTTP monitor for the Opstrace ready endpoint:

Log in to vigilmon.online and click Add Monitor.
Set Monitor type to HTTP.
Enter the URL: https://system.opstrace.example.com/-/ready
Set Check interval to 60 seconds and Expected status code to 200.
Click Save Monitor — Vigilmon will run the first check within a minute.

Add a monitor for Opstrace's API gateway

Repeat the process for the tenant API endpoint, which exercises the full request path including authentication:

Click Add Monitor again.
Set Monitor type to HTTP.
Enter the URL: https://system.opstrace.example.com/api/v1/health
Set Expected status code to 200 and Timeout to 10 seconds.
Optionally add a Keyword match for ready to confirm the response body is substantive, not a cached empty reply.
Click Save Monitor.

Why external monitoring catches what Kubernetes misses

Kubernetes liveness and readiness probes run from inside the cluster. They check whether a container process is alive on its local network interface, but they cannot detect failures at the load balancer tier, expired TLS certificates, DNS misconfiguration, or routing problems that affect external clients. A pod can pass its liveness probe while returning connection timeouts to every user outside the cluster. Vigilmon polls from external vantage points over the public internet, replicating the exact experience of an engineer or end-user trying to open a dashboard. This means Vigilmon can detect an outage even when every Kubernetes probe reports green and Alertmanager itself is unreachable.

Step 3: Monitor Opstrace's TCP port

A TCP monitor confirms that the API gateway is accepting connections at the transport layer, independently of TLS negotiation or HTTP response parsing. This catches scenarios where the TLS handshake hangs or the ingress controller crashes without freeing its socket.

In Vigilmon, click Add Monitor.
Set Monitor type to TCP.
Enter Host: system.opstrace.example.com and Port: 443.
Set Check interval to 60 seconds.
Click Save Monitor.

If your cluster exposes the API gateway via a NodePort rather than a cloud load balancer, patch the service to use a fixed port:

# Patch the api-gateway service to use NodePort 30443
kubectl patch svc api-gateway -n opstrace \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/type","value":"NodePort"},
       {"op":"add","path":"/spec/ports/0/nodePort","value":30443}]'

# Confirm the NodePort is assigned
kubectl get svc api-gateway -n opstrace
# NAME          TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
# api-gateway   NodePort   10.96.45.201   <none>        443:30443/TCP   3d

Use <node-ip>:30443 as the TCP monitor host and port in this case.

Step 4: Configure alert channels

Email alerts

In Vigilmon, go to Alert Channels and click Add Channel.
Choose Email and enter the address that should receive notifications (for example, oncall@example.com or a team distribution list).
Select the monitors created in Steps 2 and 3 and assign this channel to all of them.
Click Save — Vigilmon will send a test email to confirm delivery.

Webhook alerts

For integration with Slack, PagerDuty, or an internal runbook system, add a webhook channel:

In Alert Channels, click Add Channel and choose Webhook.
Paste your endpoint URL (for example, a Slack incoming webhook or a PagerDuty Events API v2 endpoint).
Vigilmon will POST the following JSON payload when a monitor transitions to down:

{
  "monitor_id": "mon_opstrace_api_gateway",
  "monitor_name": "Opstrace API Gateway /-/ready",
  "status": "down",
  "previous_status": "up",
  "checked_at": "2026-07-11T14:32:07Z",
  "response_time_ms": null,
  "status_code": null,
  "error": "Connection timed out after 10000ms",
  "url": "https://system.opstrace.example.com/-/ready"
}

Assign all Opstrace monitors to this channel and click Save.

Runbook: kubectl diagnostics to run when an alert fires

#!/bin/bash
# Opstrace alert runbook — run these commands in order when Vigilmon fires

NAMESPACE=opstrace

echo "=== Pod status ==="
kubectl get pods -n $NAMESPACE

echo "=== API gateway logs (last 100 lines) ==="
kubectl logs -n $NAMESPACE -l app=api-gateway --tail=100

echo "=== Ingress controller events ==="
kubectl describe ingress api-gateway -n $NAMESPACE

echo "=== Cortex ingester status ==="
kubectl get pods -n $NAMESPACE -l app=cortex-ingester
kubectl logs -n $NAMESPACE -l app=cortex-ingester --tail=50

echo "=== Loki distributor status ==="
kubectl get pods -n $NAMESPACE -l app=loki-distributor
kubectl logs -n $NAMESPACE -l app=loki-distributor --tail=50

echo "=== Recent Kubernetes events in opstrace namespace ==="
kubectl get events -n $NAMESPACE --sort-by='.lastTimestamp' | tail -20

echo "=== Node resource pressure ==="
kubectl describe nodes | grep -A5 "Conditions:"

echo "=== TLS certificate expiry ==="
echo | openssl s_client -connect system.opstrace.example.com:443 -servername system.opstrace.example.com 2>/dev/null \
  | openssl x509 -noout -dates

Step 5: Create a public status page

A Vigilmon status page lets your team and any dependent teams check Opstrace health without querying the cluster directly — useful when the cluster itself is inaccessible.

In Vigilmon, navigate to Status Pages and click New Status Page.
Give it a name such as Opstrace Observability Platform and choose a subdomain like opstrace-status.vigilmon.page.
Add all three monitors created in this tutorial: the /-/ready HTTP monitor, the /api/v1/health HTTP monitor, and the TCP port 443 monitor.
Optionally add a custom logo or description so that end-users understand what the page covers.
Click Publish — the page is immediately accessible and updates in real time as monitor results come in.

Share the status page URL with your infrastructure team, on-call rotation, and any teams whose pipelines ship metrics or logs into Opstrace so they can self-serve during incidents without needing cluster access.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | https://system.opstrace.example.com/-/ready | HTTP | API gateway process crash, ingress routing failure, TLS certificate expiry, DNS resolution failure | | https://system.opstrace.example.com/api/v1/health | HTTP | Cortex or Loki backend degradation that surfaces as a non-200 response through the gateway | | system.opstrace.example.com:443 | TCP | Network-level connectivity loss, ingress controller hang, port exhaustion before TLS negotiation |

# Quick-reference diagnostic one-liners when a Vigilmon alert fires

# 1. Are all Opstrace pods running?
kubectl get pods -n opstrace | grep -v Running

# 2. What do the API gateway logs say?
kubectl logs -n opstrace -l app=api-gateway --tail=50 --since=5m

# 3. Is the Cortex compactor in CrashLoopBackOff?
kubectl get pod -n opstrace -l app=cortex-compactor

# 4. Is the Loki distributor returning 429s?
kubectl logs -n opstrace -l app=loki-distributor --tail=50 | grep -i "429\|rate limit\|backpressure"

# 5. How long until the TLS cert expires?
echo | openssl s_client -connect system.opstrace.example.com:443 \
  -servername system.opstrace.example.com 2>/dev/null \
  | openssl x509 -noout -enddate

# 6. Are there node pressure events?
kubectl get events -n opstrace --field-selector reason=Evicted

What's next

SSL certificate monitoring — Vigilmon can alert you days before the TLS certificate on system.opstrace.example.com expires, giving you time to renew before the API gateway starts rejecting connections. Add an SSL monitor pointing at the same hostname.
Heartbeat monitoring for Opstrace's own alerting pipeline — Configure the Opstrace Prometheus ruler to POST a heartbeat to a Vigilmon dead-man's-switch endpoint on a fixed interval. If the ruler crashes or alert evaluation halts, Vigilmon detects the missing heartbeat and fires an alert through a separate channel that does not depend on the Opstrace stack.
Dependent component monitors — Add individual HTTP monitors for the Grafana UI (/api/health), the Loki push endpoint (/loki/api/v1/push), and the Cortex remote-write endpoint (/api/v1/push) to pinpoint which subsystem is degraded during a partial outage, rather than knowing only that the platform is unhealthy.

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.