Monitoring Apache SkyWalking with Vigilmon: OAP Server Health, UI Uptime, Storage Connectivity & SSL Alerts

Apache SkyWalking is a distributed APM and observability platform trusted by organizations running large-scale microservice architectures — collecting traces, metrics, logs, and profiling data from thousands of services simultaneously. SkyWalking's Observability Analysis Platform (OAP) aggregates data from language agents (Java, Go, Python, Node.js, and more), stores it in Elasticsearch, BanyanDB, or other backends, and exposes it through a GraphQL API and a built-in UI. When the OAP server is unreachable, all agents lose their telemetry sink, your SkyWalking UI goes dark, and your team loses visibility into service health at exactly the moment they need it most. Vigilmon gives you external monitoring of SkyWalking itself: OAP health endpoints, SkyWalking UI availability, storage backend TCP connectivity, and SSL certificate validity — so the system that monitors your applications is itself monitored.

What You'll Build

• An HTTP monitor on the SkyWalking OAP server health endpoint
• An HTTP monitor on the SkyWalking UI to detect frontend outages
• A TCP monitor on your Elasticsearch or BanyanDB port (SkyWalking's storage backend)
• A TCP monitor on the OAP gRPC collector port (used by language agents)
• SSL certificate monitoring for your SkyWalking UI domain
• Alerting rules that distinguish OAP failures from storage failures

Prerequisites

• Apache SkyWalking OAP Server 9.0+ (self-hosted)
• SkyWalking UI deployed (typically on port 8080)
• Storage backend: Elasticsearch 7/8, OpenSearch, or BanyanDB
• OAP REST API port accessible (default: 12800)
• A free account at vigilmon.online

Step 1: Understand the SkyWalking Architecture and Failure Modes

SkyWalking consists of three independently deployable components that can fail independently:

OAP Server (Observability Analysis Platform): The core data pipeline. Receives trace segments, metrics, and logs from agents via gRPC (port 11800) and HTTP (port 12800), processes them through the OAL (Observability Analysis Language) engine, and writes to the storage backend. When the OAP goes down, all language agents buffer data locally (if configured) or lose telemetry permanently.

SkyWalking UI: A static web application served separately from the OAP, communicating with OAP's GraphQL API (/graphql). If the UI is down, engineers cannot view traces or metrics — but the OAP is still collecting data. A UI outage is separate from an OAP outage.

Storage Backend: Elasticsearch, OpenSearch, or BanyanDB. If the storage backend is unreachable, OAP cannot write data and begins queuing in memory until it exhausts its buffer, then drops telemetry. TCP connectivity to the storage port is the earliest signal of this failure.

Step 2: Monitor the OAP Server Health Endpoint

SkyWalking OAP exposes a health check endpoint on its REST port:

# OAP health endpoint (default REST port 12800)
curl http://skywalking.example.com:12800/internal/l7check
# Returns HTTP 200 when healthy

# Alternative: check the GraphQL endpoint responds
curl -X POST http://skywalking.example.com:12800/graphql \
  -H "Content-Type: application/json" \
  -d '{"query":"{ version }"}'
# Returns: {"data":{"version":"9.7.0"}}

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: http://skywalking.example.com:12800/internal/l7check.
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Label: SkyWalking OAP health.
7. Click Save.

Port 12800 firewall rules: The OAP REST port is sometimes restricted to internal networks. If it is not externally reachable, place a lightweight reverse proxy (Nginx, Traefik) in front of OAP that exposes a /health path publicly and proxies to localhost:12800/internal/l7check. Monitor the proxy endpoint instead.

Step 3: Monitor the SkyWalking UI

The SkyWalking UI is a separate web service (often running on port 8080 or behind a reverse proxy):

# Check the SkyWalking UI is serving the frontend
curl -I https://skywalking-ui.example.com/
# Should return 200 with HTML

# Check the UI can reach the OAP GraphQL API
curl https://skywalking-ui.example.com/graphql \
  -H "Content-Type: application/json" \
  -d '{"query":"{ version }"}'

1. Add Monitor → HTTP.
2. URL: https://skywalking-ui.example.com/.
3. Check interval: 2 minutes.
4. Expected status: 200.
5. Keyword: SkyWalking (appears in the page title) or another stable string in the UI HTML.
6. Label: SkyWalking UI.
7. Click Save.

Step 4: Monitor Storage Backend TCP Connectivity

SkyWalking requires a persistent TCP connection to its storage backend. A storage outage immediately impacts OAP's ability to write telemetry:

# Test Elasticsearch connectivity
nc -zv elasticsearch.example.com 9200

# Test OpenSearch connectivity
nc -zv opensearch.example.com 9200

# Test BanyanDB gRPC port
nc -zv banyandb.example.com 17912

For Elasticsearch/OpenSearch:

1. Add Monitor → TCP.
2. Host: elasticsearch.example.com.
3. Port: 9200.
4. Check interval: 2 minutes.
5. Response timeout: 10 seconds.
6. Label: Elasticsearch (SkyWalking storage).
7. Click Save.

For BanyanDB (SkyWalking's native storage):

1. Add Monitor → TCP.
2. Host: banyandb.example.com.
3. Port: 17912.
4. Check interval: 2 minutes.
5. Label: BanyanDB (SkyWalking storage).
6. Click Save.

Step 5: Monitor the OAP gRPC Collector Port

Language agents (Java, Go, Node.js) connect to the OAP's gRPC endpoint on port 11800. If this port becomes unreachable, agents fail to deliver trace segments:

# Test gRPC collector port connectivity
nc -zv skywalking.example.com 11800

1. Add Monitor → TCP.
2. Host: skywalking.example.com.
3. Port: 11800.
4. Check interval: 2 minutes.
5. Response timeout: 10 seconds.
6. Label: SkyWalking OAP gRPC collector.
7. Click Save.

Agent buffering: Most SkyWalking agents have a local buffer that queues segments when the OAP is unreachable. However, this buffer is finite (default 5000 segments in the Java agent). If the OAP is down for more than a few minutes in a high-throughput service, trace data is lost. Early detection via this TCP monitor is critical.

Step 6: Monitor SSL Certificates

If your SkyWalking UI is served over HTTPS (strongly recommended for production), monitor the certificate expiry:

openssl s_client -connect skywalking-ui.example.com:443 2>/dev/null | openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: skywalking-ui.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

If the OAP REST API is also served over TLS (via a reverse proxy), add a second SSL monitor for that domain.

Step 7: Configure Alerting for SkyWalking-Specific Scenarios

In Vigilmon under Settings → Notifications, wire up your alert channels:

| Monitor | Trigger | Action | |---|---|---| | OAP health | Non-200 or timeout | Check OAP process; inspect OAP logs for storage write errors; verify application.yml storage config | | SkyWalking UI | Non-200 or keyword missing | Restart UI container; check reverse proxy config; verify OAP GraphQL API reachable from UI host | | Elasticsearch TCP | Connection refused or timeout | Check Elasticsearch cluster health; disk space on data nodes; heap pressure | | BanyanDB TCP | Connection refused or timeout | Check BanyanDB process; inspect BanyanDB logs | | OAP gRPC port | Connection refused or timeout | Firewall rule may have dropped port 11800; check OAP process binding | | SSL certificate | < 30 days to expiry | Renew certificate; check Let's Encrypt certbot cron; inspect Nginx TLS config |

Correlation pattern: If both the OAP health monitor and the Elasticsearch TCP monitor fire simultaneously, the root cause is almost always the storage backend. If only the OAP health monitor fires while Elasticsearch TCP is green, the OAP itself has failed — check OAP heap and GC logs.

Common SkyWalking Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | OAP process OOM kill | OAP health monitor fires; gRPC port monitor fires | | Elasticsearch index full (disk quota) | OAP health eventually unhealthy; ES TCP still open | | Elasticsearch data node restarting | ES TCP monitor fires; OAP health follows | | SkyWalking UI container stopped | UI monitor fires; OAP health stays green | | Firewall blocks port 11800 from agent network | gRPC TCP monitor fires; agents lose telemetry | | BanyanDB partition leader election | Storage TCP monitor flaps; OAP queues data | | OAP config error after upgrade | OAP fails to start; health and gRPC monitors fire | | SSL certificate expired on UI domain | SSL monitor fires 30 days before | | Reverse proxy misconfigured after infra change | UI 502; health endpoint via proxy also fails | | Storage backend disk full | OAP health 503; writes rejected |

SkyWalking gives your applications deep observability — but the monitoring platform itself must be monitored from the outside to be reliable. Vigilmon provides the external vantage point that SkyWalking cannot give itself: continuous health checks on the OAP server, the UI, the storage backend, and the gRPC collector, so your team knows within 60 seconds when the APM platform has gone dark and telemetry is being lost.

Start monitoring your SkyWalking installation in under 5 minutes — register free at vigilmon.online.