tutorial

How to Monitor Apache Calcite Health and Query Planning Availability with Vigilmon

Apache Calcite query planning failures and stale schema metadata silently break SQL-over-anything data layers. Learn how to monitor Calcite-based services, detect planner stalls, and track client application health with Vigilmon HTTP probes and heartbeat monitors.

Apache Calcite is the SQL parsing and query planning engine embedded in Drill, Flink, Hive, Kylin, and dozens of other data frameworks — but when the planner's metadata cache grows stale, a rule set produces an infinite loop, or an adapter loses its connection to the underlying data source, your SQL queries start returning wrong results or hanging indefinitely. Calcite itself is a library, not a server, so there is no built-in dashboard to tell you something has gone wrong; failures surface as timeouts or exceptions inside whichever framework embeds it.

Vigilmon gives you external visibility into Calcite-powered services through HTTP probe monitoring (via a custom health sidecar or the REST API of the embedding framework) and heartbeat monitoring for your query client applications. This tutorial covers both.

Why Calcite-Powered Services Need External Monitoring

Calcite's internal diagnostics are excellent for developers but invisible to operators at runtime. External monitoring with Vigilmon adds:

  • Proactive alerting when the HTTP endpoint fronting your Calcite-based query service becomes unreachable (indicating a JVM crash or GC storm)
  • Query planner liveness detection via a test query that exercises the full parse → validate → optimize → execute pipeline
  • Heartbeat monitoring so you know immediately when a client application stops successfully executing queries through Calcite
  • Schema metadata staleness detection by checking that a known table or view resolves correctly after a metadata refresh

These layers work together: the JVM can be alive while the Calcite planner hangs on a recursive rule application, and the planner can work correctly while an adapter has silently dropped its connection to a backend store.

Step 1: Build a Calcite Health Endpoint

Because Calcite is embedded, you need a health endpoint in the embedding application. The endpoint should exercise the full query path, not just check that the JVM is alive.

Java Spring Boot Health Endpoint (Embedded Calcite)

// CalciteHealthController.java
@RestController
public class CalciteHealthController {

    @Autowired
    private CalciteConnectionPool calcitePool;

    @GetMapping("/health/calcite")
    public ResponseEntity<Map<String, Object>> checkHealth() {
        Map<String, Object> status = new LinkedHashMap<>();
        long start = System.currentTimeMillis();

        try (Connection conn = calcitePool.getConnection()) {
            // Execute a lightweight probe query — checks parse, validate, plan, and execute
            PreparedStatement stmt = conn.prepareStatement(
                "SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLES"
            );
            ResultSet rs = stmt.executeQuery();
            rs.next();
            long tableCount = rs.getLong(1);
            long elapsed = System.currentTimeMillis() - start;

            if (elapsed > 5000) {
                status.put("status", "degraded");
                status.put("reason", "planner_slow");
                status.put("elapsed_ms", elapsed);
                return ResponseEntity.status(503).body(status);
            }

            status.put("status", "ok");
            status.put("schema_tables", tableCount);
            status.put("elapsed_ms", elapsed);
            return ResponseEntity.ok(status);

        } catch (SQLException e) {
            status.put("status", "down");
            status.put("error", e.getMessage());
            return ResponseEntity.status(503).body(status);
        }
    }
}

Python Health Endpoint (pyarrow + Calcite via JDBC/ODBC bridge)

# calcite_health.py
from flask import Flask, jsonify
import jaydebeapi
import os
import time

app = Flask(__name__)

CALCITE_JDBC_URL = os.environ.get('CALCITE_JDBC_URL', 'jdbc:calcite:model=/app/model.json')
JDBC_DRIVER = 'org.apache.calcite.jdbc.Driver'
JDBC_JAR = os.environ.get('CALCITE_JDBC_JAR', '/opt/calcite/calcite-avatica.jar')

@app.route('/health/calcite')
def calcite_health():
    start = time.time()
    try:
        conn = jaydebeapi.connect(
            JDBC_DRIVER,
            CALCITE_JDBC_URL,
            {},
            JDBC_JAR
        )
        cursor = conn.cursor()
        cursor.execute("SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLES")
        row = cursor.fetchone()
        elapsed_ms = int((time.time() - start) * 1000)
        conn.close()

        if elapsed_ms > 5000:
            return jsonify({'status': 'degraded', 'reason': 'planner_slow',
                            'elapsed_ms': elapsed_ms}), 503

        return jsonify({'status': 'ok', 'table_count': row[0],
                        'elapsed_ms': elapsed_ms}), 200

    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8091)

Using the Avatica REST Endpoint

If your service uses Apache Avatica (Calcite's RPC layer), it exposes HTTP on port 8765 by default. You can probe it directly:

# Check Avatica server liveness
curl -i http://calcite-avatica.example.com:8765/

# Send a minimal statement request to verify the planner is alive
curl -s -X POST http://calcite-avatica.example.com:8765 \
  -H 'Content-Type: application/json' \
  -d '{"request": "openConnection", "connectionId": "health-probe-001", "info": {}}'

Step 2: Configure Vigilmon HTTP Monitor for Calcite

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Calcite health endpoint: https://your-app.example.com/health/calcite
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 8000ms (Calcite planning can involve multi-step rule chain execution)
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the Avatica endpoint if applicable:

  • URL: http://calcite-avatica.example.com:8765/
  • Expected: 200
  • Interval: 2 minutes
  • Alert channel: on-call channel

Vigilmon's multi-region probe consensus prevents false positives from transient network blips between probe nodes and your query service.

Step 3: Heartbeat Monitoring for Calcite Query Clients

Planner and Avatica health checks are necessary — but not sufficient. A client application can be stuck retrying a failed query, producing results from a stale cache, or silently swallowing SQLException responses, while the service metrics look healthy.

Vigilmon heartbeat monitors detect silent client stalls: your application pings Vigilmon after each successful query execution batch. If pings stop, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: calcite-query-client
  3. Set the expected interval: 5 minutes (adjust to your query throughput)
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Java Query Client with Heartbeat

// CalciteQueryService.java
import java.sql.*;
import java.net.http.*;
import java.net.URI;

public class CalciteQueryService {

    private final Connection calciteConn;
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final HttpClient httpClient = HttpClient.newHttpClient();
    private int queriesSinceHeartbeat = 0;
    private static final int HEARTBEAT_EVERY = 100;

    public ResultSet executeQuery(String sql) throws SQLException {
        PreparedStatement stmt = calciteConn.prepareStatement(sql);
        ResultSet rs = stmt.executeQuery();

        queriesSinceHeartbeat++;
        if (queriesSinceHeartbeat >= HEARTBEAT_EVERY) {
            pingVigilmon();
            queriesSinceHeartbeat = 0;
        }
        return rs;
    }

    private void pingVigilmon() {
        try {
            HttpRequest req = HttpRequest.newBuilder()
                .uri(URI.create(heartbeatUrl))
                .GET().build();
            httpClient.sendAsync(req, HttpResponse.BodyHandlers.discarding());
        } catch (Exception e) {
            // Non-fatal: heartbeat failure must not interrupt query execution
        }
    }
}

Python Query Client with Heartbeat

# calcite_query_worker.py
import jaydebeapi, requests, os, time

HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']
HEARTBEAT_INTERVAL = 60  # seconds

conn = jaydebeapi.connect(
    'org.apache.calcite.jdbc.Driver',
    os.environ['CALCITE_JDBC_URL'],
    {},
    os.environ['CALCITE_JDBC_JAR']
)
cursor = conn.cursor()

last_heartbeat = time.time()
queries_executed = 0

for sql in get_queries_to_run():
    cursor.execute(sql)
    rows = cursor.fetchall()
    process_results(rows)
    queries_executed += 1

    if time.time() - last_heartbeat > HEARTBEAT_INTERVAL:
        try:
            requests.get(HEARTBEAT_URL, timeout=5)
        except Exception:
            pass
        last_heartbeat = time.time()

print(f"Executed {queries_executed} queries")

Step 4: Alert Routing for Calcite Failures

Calcite failures cascade through the embedding stack: a planner infinite loop causes the query thread to hang, which exhausts the connection pool, which causes all clients to queue on getConnection(), which surfaces as a blanket timeout across every service that queries through Calcite. Alert routing should reflect this cascade order.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Calcite health /health/calcite | Slack + PagerDuty | P1 | | Avatica REST endpoint | Slack | P2 | | Heartbeat: query client application | Slack + email | P2 | | Heartbeat: batch analytics job | Email | P3 |

Set response time thresholds for early warning:

  • Alert at 4000ms for the health endpoint (slow planning signals a runaway optimization rule)
  • Alert at 10000ms for Avatica (slow serialization signals GC pressure or network congestion)

For query-critical applications, enable two consecutive failures before alerting — Calcite's adaptive rule pruning can briefly spike planning time during schema refreshes without representing a real outage.

Summary

Calcite failures are invisible at the application layer because the planner is embedded. External monitoring catches planner stalls, adapter disconnections, and client query stalls before they become data pipeline outages:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/calcite | Planner liveness, schema metadata validity, query round-trip time | | HTTP monitor on Avatica REST | RPC layer availability, serialization health | | Heartbeat monitor | Client application liveness, query execution throughput |

Get started free at vigilmon.online — your first Calcite monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles