tutorial

How to Monitor Apache Dubbo RPC Service Health with Vigilmon

Apache Dubbo RPC failures — provider unavailability, registry disconnects, and load balancer routing errors — are invisible to HTTP uptime checks. Learn how to monitor Dubbo provider health, registry connectivity, and consumer-side call success with Vigilmon HTTP probes and heartbeat monitors.

Apache Dubbo is a high-performance RPC framework widely used in Java microservices architectures across East Asia and increasingly globally. Dubbo services communicate over a custom binary protocol (Dubbo protocol) or HTTP/2, register with a service registry (Zookeeper, Nacos, or Consul), and use client-side load balancing. When a Dubbo provider goes down, the consumer retries across available instances — which can mask the failure completely until all providers are gone. When the registry itself is unreachable, providers can't register new instances and consumers can't discover them, but existing connections continue working until they fail.

Vigilmon gives you external visibility into Dubbo service health through HTTP probe monitoring and heartbeat monitors. This tutorial covers both.

Why Dubbo Needs External Monitoring

Dubbo's resilience features — retries, failover, load balancing — are excellent for handling transient failures, but they make serious problems invisible:

  • Provider unavailability masked by failover: when 2 of 3 providers are down, Dubbo silently routes all traffic to the surviving provider — no errors surface, but capacity is critically degraded
  • Registry disconnects: when Nacos or ZooKeeper is unreachable, providers can't heartbeat and consumers can't discover new instances — but cached addresses keep existing calls working until the providers restart
  • Thread pool overflow: Dubbo's server-side thread pool is fixed-size by default (dubbo.provider.threads=200); when exhausted, new requests receive ThreadPoolExhaustedException — a runtime error, not a health check failure
  • Serialization errors: a schema mismatch between provider and consumer (common after rolling upgrades) causes DecodeException on every call — callers get errors but no health check reports a failure

External monitoring with Vigilmon adds:

  • Proactive alerting when Dubbo provider health endpoints report degraded state
  • Heartbeat monitoring so you know immediately when consumer-side RPC calls stop succeeding
  • HTTP probe monitoring for Dubbo's built-in QoS port and admin endpoints
  • Multi-region availability checking from outside your network perimeter

Step 1: Build a Dubbo Health Endpoint

Dubbo includes a built-in Quality of Service (QoS) port (default: 22222) with health commands. You can also expose a custom HTTP health endpoint via Dubbo's REST protocol or a Spring Boot Actuator integration.

Using Dubbo's Built-In QoS Port

Dubbo's QoS server exposes telnet-compatible commands over HTTP:

# Check overall Dubbo liveness
curl http://localhost:22222/live

# Check readiness (are all services registered?)
curl http://localhost:22222/ready

# List registered providers
curl http://localhost:22222/ls

Point Vigilmon directly at http://your-provider:22222/live for basic liveness. For production, expose these via a reverse proxy with authentication.

Java Spring Boot Health Indicator (Dubbo + Actuator)

// DubboHealthIndicator.java
import org.apache.dubbo.config.spring.ServiceBean;
import org.apache.dubbo.rpc.model.ApplicationModel;
import org.apache.dubbo.rpc.model.ServiceMetadata;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
import java.util.*;

@Component
public class DubboHealthIndicator implements HealthIndicator {

    @Override
    public Health health() {
        try {
            ApplicationModel appModel = ApplicationModel.defaultModel();
            if (appModel == null) {
                return Health.down().withDetail("reason", "dubbo_not_initialized").build();
            }

            // Check that all exported services are registered
            Collection<ServiceMetadata> services = appModel.getApplicationServiceRepository()
                .getAllServices();

            if (services.isEmpty()) {
                return Health.down()
                    .withDetail("reason", "no_services_registered")
                    .build();
            }

            return Health.up()
                .withDetail("service_count", services.size())
                .withDetail("services", services.stream()
                    .map(s -> s.getServiceKey())
                    .toList())
                .build();
        } catch (Exception e) {
            return Health.down().withException(e).build();
        }
    }
}

This exposes /actuator/health/dubbo automatically via Spring Boot Actuator.

Custom HTTP Health Endpoint (JAX-RS or Spring MVC)

// DubboHealthController.java
import org.apache.dubbo.common.URL;
import org.apache.dubbo.registry.RegistryService;
import org.springframework.web.bind.annotation.*;
import java.util.*;

@RestController
@RequestMapping("/health")
public class DubboHealthController {

    private final RegistryService registryService;

    public DubboHealthController(RegistryService registryService) {
        this.registryService = registryService;
    }

    @GetMapping("/dubbo")
    public ResponseEntity<Map<String, Object>> health() {
        Map<String, Object> response = new LinkedHashMap<>();

        try {
            // Check registry connectivity
            boolean registryAvailable = checkRegistryConnectivity();
            response.put("registry_available", registryAvailable);

            if (!registryAvailable) {
                response.put("status", "degraded");
                response.put("reason", "registry_unreachable");
                return ResponseEntity.status(503).body(response);
            }

            response.put("status", "ok");
            return ResponseEntity.ok(response);
        } catch (Exception e) {
            response.put("status", "down");
            response.put("error", e.getMessage());
            return ResponseEntity.status(503).body(response);
        }
    }

    private boolean checkRegistryConnectivity() {
        // Attempt a lightweight registry operation to verify connectivity
        try {
            registryService.lookup(URL.valueOf("dubbo://0.0.0.0/health-check?interface=health"));
            return true;
        } catch (Exception e) {
            return false;
        }
    }
}

Node.js Sidecar (Querying QoS Port)

// health/dubbo.js
const express = require('express');
const http = require('http');

const app = express();

const QOS_HOST = process.env.DUBBO_QOS_HOST || 'localhost';
const QOS_PORT = parseInt(process.env.DUBBO_QOS_PORT || '22222');

function fetchQos(path) {
  return new Promise((resolve, reject) => {
    const req = http.get({ host: QOS_HOST, port: QOS_PORT, path, timeout: 5000 }, res => {
      let body = '';
      res.on('data', chunk => { body += chunk; });
      res.on('end', () => resolve({ status: res.statusCode, body }));
    });
    req.on('error', reject);
    req.on('timeout', () => { req.destroy(); reject(new Error('timeout')); });
  });
}

app.get('/health/dubbo', async (req, res) => {
  try {
    const live = await fetchQos('/live');
    if (live.status !== 200 || !live.body.toLowerCase().includes('ok')) {
      return res.status(503).json({
        status: 'down',
        reason: 'liveness_check_failed',
        response: live.body.trim(),
      });
    }

    const ready = await fetchQos('/ready');
    if (ready.status !== 200 || !ready.body.toLowerCase().includes('ok')) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'readiness_check_failed',
        response: ready.body.trim(),
      });
    }

    return res.status(200).json({
      status: 'ok',
      live: true,
      ready: true,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3010);

Python Health Sidecar

# health_dubbo.py
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)

QOS_BASE = f"http://{os.environ.get('DUBBO_QOS_HOST', 'localhost')}:{os.environ.get('DUBBO_QOS_PORT', '22222')}"

@app.route('/health/dubbo')
def health():
    try:
        live = requests.get(f'{QOS_BASE}/live', timeout=5)
        if live.status_code != 200 or 'ok' not in live.text.lower():
            return jsonify({'status': 'down', 'reason': 'liveness_failed', 'response': live.text.strip()}), 503

        ready = requests.get(f'{QOS_BASE}/ready', timeout=5)
        if ready.status_code != 200 or 'ok' not in ready.text.lower():
            return jsonify({'status': 'degraded', 'reason': 'readiness_failed', 'response': ready.text.strip()}), 503

        return jsonify({'status': 'ok', 'live': True, 'ready': True})
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3010)

Step 2: Configure Vigilmon HTTP Monitor for Dubbo

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Dubbo health endpoint: https://your-app.example.com/actuator/health/dubbo (or http://your-provider:22222/live via a reverse proxy for the QoS port)
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"UP" (Actuator) or "status":"ok" (sidecar)
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the QoS readiness check:

  • URL: http://your-provider:22222/ready (via reverse proxy)
  • Expected: 200, body contains: OK
  • Interval: 2 minutes
  • Alert channel: P1 pager — readiness failure means the provider is not accepting new registrations

Vigilmon's multi-region probe consensus filters transient blips from genuine provider failures.

Step 3: Heartbeat Monitoring for Dubbo RPC Call Success

Provider health monitoring catches down providers, but not degraded call success rates. A provider can be "ready" while returning errors on every call due to a downstream database failure or a serialization mismatch introduced by a recent deployment.

Vigilmon heartbeat monitors catch these: your consumer-side code pings Vigilmon after each successful RPC call batch.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: order-service-dubbo-consumer
  3. Set the expected interval: 5 minutes
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a Dubbo Consumer (Spring Boot)

// OrderProcessor.java — consumer-side, calls a Dubbo provider
import org.apache.dubbo.config.annotation.DubboReference;
import org.springframework.scheduling.annotation.Scheduled;
import java.net.http.*;
import java.net.URI;

@Component
public class OrderProcessor {

    @DubboReference(version = "1.0.0", timeout = 5000, retries = 2)
    private InventoryService inventoryService;

    private final HttpClient http = HttpClient.newHttpClient();
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");

    @Scheduled(fixedDelay = 60000)
    public void processOrderBatch() {
        try {
            List<Order> orders = orderRepository.findPending();
            int successCount = 0;

            for (Order order : orders) {
                try {
                    // RPC call to Dubbo provider
                    InventoryResult result = inventoryService.reserve(order.getItems());
                    if (result.isSuccess()) {
                        order.markReserved();
                        successCount++;
                    }
                } catch (RpcException e) {
                    log.error("RPC failed for order {}: {}", order.getId(), e.getMessage());
                }
            }

            if (successCount > 0) {
                // At least one successful RPC — signal Vigilmon
                pingHeartbeat();
            }
        } catch (Exception e) {
            log.error("Order batch processing failed: {}", e.getMessage());
        }
    }

    private void pingHeartbeat() {
        if (heartbeatUrl == null) return;
        try {
            http.sendAsync(
                HttpRequest.newBuilder(URI.create(heartbeatUrl)).GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        } catch (Exception ignored) {}
    }
}

Wire It Into a Dubbo Filter (Cross-Cutting Success Tracking)

// VigilmonHeartbeatFilter.java — applied on the consumer side
import org.apache.dubbo.common.extension.Activate;
import org.apache.dubbo.rpc.*;
import java.net.http.*;
import java.net.URI;
import java.util.concurrent.atomic.*;

@Activate(group = CommonConstants.CONSUMER)
public class VigilmonHeartbeatFilter implements Filter {

    private final HttpClient http = HttpClient.newHttpClient();
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final AtomicLong lastPingMs = new AtomicLong(0);
    private static final long PING_INTERVAL_MS = 60_000;

    @Override
    public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
        Result result = invoker.invoke(invocation);

        if (!result.hasException()) {
            long now = System.currentTimeMillis();
            if (now - lastPingMs.get() > PING_INTERVAL_MS) {
                lastPingMs.set(now);
                pingHeartbeat();
            }
        }

        return result;
    }

    private void pingHeartbeat() {
        if (heartbeatUrl == null) return;
        try {
            http.sendAsync(
                HttpRequest.newBuilder(URI.create(heartbeatUrl)).GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        } catch (Exception ignored) {}
    }
}

Register the filter in META-INF/dubbo/org.apache.dubbo.rpc.Filter:

vigilmonHeartbeat=com.example.filter.VigilmonHeartbeatFilter

And activate it in your consumer configuration:

dubbo:
  consumer:
    filter: vigilmonHeartbeat

Step 4: Alert Routing for Dubbo Failures

Dubbo failures range from a single provider instance going down (handled by failover) to all providers failing (full service outage) to registry disconnects (new instances can't register, but existing connections survive). Route alerts by severity:

| Monitor | Alert Channel | Priority | |---|---|---| | HTTP: /actuator/health/dubbo | Slack + PagerDuty | P1 | | HTTP: QoS /live endpoint | Slack + PagerDuty | P1 | | HTTP: QoS /ready endpoint | Slack + PagerDuty | P1 | | Heartbeat: consumer RPC success | Slack + PagerDuty | P1 | | Heartbeat: provider business operation | Slack + email | P2 |

Set response time thresholds as early warning signals:

  • Alert at 1000ms for the QoS endpoints (slow QoS responses indicate JVM GC pressure)
  • Alert at 3000ms for the actuator health endpoint
  • Alert at 5000ms for the consumer heartbeat endpoint (slow consumer-side processing is often the first sign of provider degradation)

For production microservices, set up a status page in Vigilmon showing each Dubbo provider's health alongside the consumers that depend on them — this makes cascade root-cause analysis instant during incidents, especially when rolling deployments introduce serialization mismatches.

Summary

Dubbo failures are distributed failures — a provider going down triggers silent failover, a registry disconnect degrades service discovery, and a serialization mismatch causes every RPC call to fail while the provider health check passes. External monitoring catches these before they become incidents:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /actuator/health/dubbo | Dubbo application model state, registered service count | | HTTP monitor on QoS /live | Provider JVM liveness | | HTTP monitor on QoS /ready | Provider registration readiness and registry connectivity | | Heartbeat monitor (consumer-side) | End-to-end RPC call success across all providers |

Get started free at vigilmon.online — your first Dubbo service monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles