Monitoring Nacos with Vigilmon

Nacos is the service registry and dynamic configuration management platform that Spring Cloud Alibaba microservices depend on for discovering each other and receiving real-time configuration updates. When Nacos goes down, microservices can't register themselves, can't locate their dependencies, and stop receiving configuration changes — a cascading failure that affects every service in the mesh simultaneously.

Vigilmon gives you external visibility into Nacos server health, service registration, configuration push integrity, Raft consensus, and gRPC port availability — so you catch failures before your entire microservices fleet starts logging connection refused errors.

Why Nacos Monitoring Matters

Nacos is infrastructure for your infrastructure. A Nacos outage causes:

• Service discovery failure — microservices cannot register or discover peers; load balancing breaks
• Configuration push stops — services run on stale configuration after their in-memory cached values expire
• Health check failures accumulate — Nacos stops health-checking registered instances; unhealthy services continue receiving traffic
• gRPC registration breaks — Nacos 2.x high-throughput gRPC registrations fail silently when port 9848 is down

External monitoring from Vigilmon detects these failures before your microservices start logging Connection refused to their log aggregators.

What You'll Set Up

• HTTP uptime monitor for the Nacos API and management console (port 8848)
• gRPC port liveness check (port 9848, Nacos 2.x)
• Service registry health endpoint (instance count trend)
• Configuration push health check
• Raft consensus health for clustered deployments
• MySQL backend connectivity check
• Backup completion heartbeat

Step 1: Monitor the Nacos API Health

Nacos exposes Spring Boot Actuator endpoints. The primary health probe is:

GET http://your-nacos-host:8848/nacos/actuator/health

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter: http://your-nacos-host:8848/nacos/actuator/health
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Response body contains, enter "status":"UP".
7. Set Response time threshold to 5000ms.
8. Click Save.

For Nacos clusters behind a VIP or load balancer, point the monitor at the VIP address. If Nacos is secured with authentication, use Vigilmon's Request headers setting to add a valid Authorization header or query the health endpoint from an unauthenticated context (Nacos actuator health is typically open by default).

Step 2: TCP Monitor for the gRPC Port (Nacos 2.x)

Nacos 2.x uses gRPC on port 9848 for high-throughput service registration and health heartbeats from clients. The HTTP API on port 8848 can be healthy while the gRPC endpoint is unreachable — which causes all Nacos 2.x SDK clients to fail registration silently.

1. In Vigilmon, click Add Monitor → TCP Port.
2. Host: your-nacos-host, Port: 9848.
3. Interval: 1 minute.
4. Alert channel: same channel as the API monitor.

This TCP check confirms that the gRPC listener is accepting connections. A failure here while the HTTP endpoint is healthy indicates a gRPC configuration problem or port binding failure.

Step 3: Build a Nacos Health Sidecar

Add a sidecar HTTP service that queries Nacos's management APIs and exposes granular health signals:

Python Nacos Health Sidecar

# nacos_health.py
import os
import requests
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

NACOS_URL = os.environ.get("NACOS_URL", "http://localhost:8848")
NACOS_USERNAME = os.environ.get("NACOS_USERNAME", "nacos")
NACOS_PASSWORD = os.environ.get("NACOS_PASSWORD", "nacos")
MYSQL_DSN = os.environ.get("NACOS_MYSQL_DSN", "")

def get_nacos_token():
    """Authenticate and get Nacos access token."""
    try:
        resp = requests.post(
            f"{NACOS_URL}/nacos/v1/auth/login",
            data={"username": NACOS_USERNAME, "password": NACOS_PASSWORD},
            timeout=5,
        )
        return resp.json().get("accessToken", "")
    except Exception:
        return ""


@app.get("/health/nacos/registry")
def registry_health():
    """Check service registry — verify instance count is non-zero."""
    min_instances = int(os.environ.get("MIN_NACOS_INSTANCES", "1"))
    token = get_nacos_token()
    try:
        headers = {"Authorization": f"Bearer {token}"} if token else {}
        resp = requests.get(
            f"{NACOS_URL}/nacos/v1/ns/catalog/instances",
            params={"pageNo": 1, "pageSize": 1},
            headers=headers,
            timeout=5,
        )
        data = resp.json()
        total = data.get("count", 0)
        if total < min_instances:
            return JSONResponse(status_code=503, content={
                "status": "degraded",
                "reason": "instance_count_below_threshold",
                "instance_count": total,
                "threshold": min_instances,
            })
        return {"status": "ok", "instance_count": total}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})


@app.get("/health/nacos/config-push")
def config_push_health():
    """Verify Nacos can return a known configuration — proxy for config push health."""
    namespace = os.environ.get("NACOS_HEALTH_NAMESPACE", "public")
    data_id = os.environ.get("NACOS_HEALTH_DATAID", "")
    group = os.environ.get("NACOS_HEALTH_GROUP", "DEFAULT_GROUP")

    if not data_id:
        return {"status": "ok", "note": "config_push_check_not_configured"}

    token = get_nacos_token()
    try:
        headers = {"Authorization": f"Bearer {token}"} if token else {}
        resp = requests.get(
            f"{NACOS_URL}/nacos/v1/cs/configs",
            params={"dataId": data_id, "group": group, "tenant": namespace},
            headers=headers,
            timeout=5,
        )
        if resp.status_code == 200 and resp.text:
            return {"status": "ok", "config_size_bytes": len(resp.text)}
        return JSONResponse(status_code=503, content={
            "status": "degraded",
            "reason": "config_fetch_failed",
            "http_status": resp.status_code,
        })
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})


@app.get("/health/nacos/cluster")
def cluster_health():
    """Check Nacos cluster Raft consensus (cluster mode only)."""
    token = get_nacos_token()
    try:
        headers = {"Authorization": f"Bearer {token}"} if token else {}
        resp = requests.get(
            f"{NACOS_URL}/nacos/v1/ns/raft/state",
            headers=headers,
            timeout=5,
        )
        if resp.status_code != 200:
            return JSONResponse(status_code=503, content={
                "status": "degraded",
                "reason": "raft_state_unreachable",
            })
        data = resp.json()
        leader = data.get("leader", "")
        if not leader:
            return JSONResponse(status_code=503, content={
                "status": "degraded",
                "reason": "no_raft_leader",
                "cluster_state": data,
            })
        return {"status": "ok", "raft_leader": leader}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})


@app.get("/health/nacos/mysql")
def mysql_health():
    """Check MySQL backend connectivity (cluster mode only)."""
    if not MYSQL_DSN:
        return {"status": "ok", "note": "mysql_not_configured"}

    try:
        import pymysql
        conn = pymysql.connect(
            **dict(item.split("=") for item in MYSQL_DSN.split(";")),
            connect_timeout=3,
        )
        conn.cursor().execute("SELECT 1")
        conn.close()
        return {"status": "ok"}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Run the sidecar on port 8099:

uvicorn nacos_health:app --host 0.0.0.0 --port 8099

Configure environment variables:

NACOS_URL=http://localhost:8848
NACOS_USERNAME=nacos
NACOS_PASSWORD=your-password
NACOS_HEALTH_DATAID=health-probe   # a known config key to verify push path
NACOS_HEALTH_GROUP=DEFAULT_GROUP
MIN_NACOS_INSTANCES=5              # alert if fewer instances registered

Step 4: Heartbeat for Configuration Backup Jobs

Nacos configuration data (namespaces, configs, permissions) must be backed up regularly. Nacos does not have built-in backup alerting — if your backup cron job silently fails, you lose disaster recovery coverage without knowing it.

1. In Vigilmon, click Add Monitor → Heartbeat.
2. Name: nacos-config-backup.
3. Expected interval: 1440 minutes (daily backup).
4. Grace period: 120 minutes.
5. Save and copy the heartbeat URL.

Wire the ping into your backup script:

#!/bin/bash
# nacos-backup.sh

# Export Nacos configuration via API
curl -u "${NACOS_USERNAME}:${NACOS_PASSWORD}" \
  "http://localhost:8848/nacos/v1/cs/configs/export?group=DEFAULT_GROUP" \
  -o "/backup/nacos-configs-$(date +%Y%m%d).zip"

if [ $? -eq 0 ]; then
  curl -s "${VIGILMON_HEARTBEAT_URL}" > /dev/null
else
  echo "Nacos backup failed — heartbeat NOT sent" >&2
  exit 1
fi

Step 5: Add Vigilmon Monitors for Each Health Endpoint

Add a Vigilmon HTTP monitor for each sidecar endpoint:

| Endpoint | Expected | Interval | |---|---|---| | /health/nacos/registry | 200, body "status":"ok" | 1 min | | /health/nacos/config-push | 200, body "status":"ok" | 2 min | | /health/nacos/cluster | 200, body "status":"ok" | 1 min | | /health/nacos/mysql | 200, body "status":"ok" | 1 min |

Step 6: Configure Alert Channels and Routing

| Monitor | Alert Channel | Suggested Priority | |---|---|---| | Nacos actuator health | Slack + PagerDuty | P1 — all service discovery fails | | gRPC port TCP | Slack + PagerDuty | P1 — all Nacos 2.x registrations fail | | Service registry instances | Slack | P2 — mass deregistration event | | Config push health | Slack | P2 — services running stale config | | Raft cluster health | Slack | P2 — consensus lost in cluster mode | | MySQL backend health | Slack | P2 — persistence unavailable | | Backup heartbeat | Email | P3 — backup missing |

Recommended alert settings:

• Consecutive failures before alert: 2 — avoids false positives from brief authentication token expiry
• Response time threshold: 5000ms — Nacos authentication + registry queries can be slow under load
• Maintenance windows: suppress during Nacos version upgrades and cluster topology changes

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP — actuator health | :8848/nacos/actuator/health | Nacos server down | | TCP — gRPC port | :9848 | gRPC registration failures (Nacos 2.x) | | HTTP — registry | :8099/health/nacos/registry | Mass service deregistration | | HTTP — config push | :8099/health/nacos/config-push | Configuration push path failure | | HTTP — cluster | :8099/health/nacos/cluster | Raft leader loss in cluster mode | | HTTP — MySQL | :8099/health/nacos/mysql | Persistence backend failure | | Heartbeat — backup | Heartbeat URL | Backup job missed or failed |

Nacos is the nervous system of Spring Cloud Alibaba deployments — when it fails, every microservice in the mesh feels it. With Vigilmon watching the API health, gRPC port, registry state, and configuration push path, you catch failures before they cascade across your entire service fleet.

Get started free at vigilmon.online — your first Nacos monitor is running in under two minutes.