Monitoring Apache Empire-db Applications with Vigilmon

Apache Empire-db is a Java library that provides a type-safe, SQL-generating abstraction layer over relational databases. Rather than mapping objects to rows (as Hibernate does), Empire-db models the database schema as Java objects and generates explicit SQL statements through a fluent query builder — giving developers full control over the SQL generated while maintaining compile-time type safety. Applications using Empire-db query the database through DBCommand, DBReader, and DBRecord objects; the framework handles connection acquisition from a pool, SQL generation, result mapping, and transaction management. When the underlying database becomes unreachable, the connection pool exhausts under load, query plans regress after schema changes, or transaction isolation conflicts accumulate, Empire-db applications degrade in ways that are often invisible at the framework level. Vigilmon gives you external monitoring for Empire-db application liveness, JDBC connection pool health, query execution latency, transaction throughput, and database schema consistency so you catch data layer failures before they reach end users.

What You'll Set Up

• Empire-db application health endpoint monitoring
• JDBC connection pool utilization tracking
• Query execution latency and slow query monitoring
• Transaction throughput and error rate tracking
• Database schema consistency verification

Prerequisites

• A Java application using Apache Empire-db 3.0+
• A JDBC connection pool (HikariCP, c3p0, or Apache DBCP2)
• Access to a health endpoint or JMX metrics from your application
• A free Vigilmon account

Step 1: Monitor Application Health

Empire-db applications need a health endpoint that validates the entire data access stack — not just that the JVM is alive, but that the database connection, schema, and Empire-db session initialization are all working. Add a health endpoint to your application that performs a lightweight Empire-db probe query:

Add a health servlet or Spring Boot health indicator to your application:

// HealthCheckServlet.java
@WebServlet("/health")
public class HealthCheckServlet extends HttpServlet {

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        try {
            // Acquire a connection and run a trivial Empire-db query
            DBDatabase db = ApplicationDatabase.getInstance();
            try (Connection conn = db.getConnection()) {
                DBCommand cmd = db.createCommand();
                cmd.select(db.USERS.count()); // Replace with any table in your schema
                int count = db.querySingleInt(cmd, -1, conn);

                if (count >= 0) {
                    resp.setStatus(200);
                    resp.getWriter().write("{\"status\":\"ok\",\"db\":\"connected\"}");
                } else {
                    resp.setStatus(503);
                    resp.getWriter().write("{\"status\":\"error\",\"db\":\"query failed\"}");
                }
            }
        } catch (Exception e) {
            resp.setStatus(503);
            resp.getWriter().write("{\"status\":\"error\",\"message\":\"" +
                e.getMessage().replace("\"", "'") + "\"}");
        }
    }
}

1. In Vigilmon, click Add Monitor → HTTP(S).
2. Enter the URL: https://your-app.internal/health
3. Set Check interval to 1 minute.
4. Set Expected HTTP status to 200.
5. Optionally set a keyword match for "status":"ok" to validate the response body.

This single monitor catches JVM crashes, Empire-db session initialization failures, database TCP connectivity issues, and authentication failures in one check.

Step 2: Monitor JDBC Connection Pool Utilization

Empire-db relies on a JDBC connection pool to acquire database connections for each query and transaction. When the pool is exhausted — because queries are slow, transactions are held open too long, or the pool size is undersized for the application's concurrency — getConnection() calls block and eventually timeout. Connection pool saturation is one of the most common causes of Empire-db application degradation under load. Expose pool metrics via JMX or a custom endpoint:

For HikariCP (the recommended pool for Empire-db applications):

// MetricsServlet.java — add to your application
@WebServlet("/metrics/pool")
public class PoolMetricsServlet extends HttpServlet {

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        HikariDataSource ds = (HikariDataSource) ApplicationDatabase.getDataSource();
        HikariPoolMXBean pool = ds.getHikariPoolMXBean();

        int active = pool.getActiveConnections();
        int idle = pool.getIdleConnections();
        int waiting = pool.getThreadsAwaitingConnection();
        int total = pool.getTotalConnections();
        int maxPool = ds.getMaximumPoolSize();

        resp.setContentType("application/json");

        if (waiting > 5 || (active * 100 / maxPool) > 90) {
            resp.setStatus(503);
        } else {
            resp.setStatus(200);
        }

        resp.getWriter().write(String.format(
            "{\"active\":%d,\"idle\":%d,\"waiting\":%d,\"total\":%d,\"max\":%d}",
            active, idle, waiting, total, maxPool
        ));
    }
}

1. In Vigilmon, click Add Monitor → HTTP(S).
2. Enter the URL: https://your-app.internal/metrics/pool
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.

For environments without an embedded HTTP server, use a cron heartbeat with a JMX probe:

#!/bin/bash
# /usr/local/bin/empire-db-pool-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
APP_HOST="localhost"
JMX_PORT="9999"
POOL_BEAN="com.zaxxer.hikari:type=Pool (HikariPool-1)"
MAX_WAIT_THREADS=5

WAITING=$(java -jar /opt/jmxterm.jar <<EOF 2>/dev/null
open ${APP_HOST}:${JMX_PORT}
bean "${POOL_BEAN}"
get ThreadsAwaitingConnection
close
EOF
echo "$WAITING" | grep -oP '\d+' | tail -1)

WAITING="${WAITING:-0}"

if [ "$WAITING" -le "$MAX_WAIT_THREADS" ]; then
  echo "Connection pool OK: ${WAITING} threads waiting"
  curl -s "$HEARTBEAT_URL"
else
  echo "Connection pool saturated: ${WAITING} threads waiting for connection"
  exit 1
fi

Set the Vigilmon heartbeat to 5 minutes. Sustained connection waiting (not brief spikes during traffic bursts) indicates either pool undersizing or long-held transactions in Empire-db operations — look for DBRecord instances that are opened for update but not committed.

Step 3: Monitor Query Execution Latency

Empire-db's fluent query builder generates explicit SQL, but slow queries still kill application throughput. After schema changes, index drops, or data volume growth, query plans that were fast become table scans. Monitor query latency using a custom timing endpoint or database slow-query logs:

Add a latency probe endpoint to your application:

// LatencyProbeServlet.java
@WebServlet("/metrics/latency")
public class LatencyProbeServlet extends HttpServlet {

    private static final long MAX_LATENCY_MS = 500;

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        DBDatabase db = ApplicationDatabase.getInstance();

        try (Connection conn = db.getConnection()) {
            // Run a representative read query (adjust to your schema)
            DBCommand cmd = db.createCommand();
            cmd.select(db.ORDERS.count());
            cmd.where(db.ORDERS.STATUS.is("PENDING"));

            long start = System.currentTimeMillis();
            int count = db.querySingleInt(cmd, -1, conn);
            long elapsed = System.currentTimeMillis() - start;

            resp.setContentType("application/json");

            if (elapsed <= MAX_LATENCY_MS && count >= 0) {
                resp.setStatus(200);
                resp.getWriter().write(
                    String.format("{\"latency_ms\":%d,\"status\":\"ok\"}", elapsed));
            } else {
                resp.setStatus(503);
                resp.getWriter().write(
                    String.format("{\"latency_ms\":%d,\"status\":\"slow\"}", elapsed));
            }
        } catch (Exception e) {
            resp.setStatus(503);
            resp.getWriter().write("{\"status\":\"error\"}");
        }
    }
}

1. In Vigilmon, click Add Monitor → HTTP(S).
2. Enter the URL: https://your-app.internal/metrics/latency
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.

For monitoring slow queries at the database level, use a cron heartbeat:

#!/bin/bash
# /usr/local/bin/empire-db-slow-queries.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
DB_HOST="db.internal"
DB_USER="monitor"
DB_PASS="monitor-password"
DB_NAME="appdb"
SLOW_QUERY_THRESHOLD_SEC=2
MAX_SLOW_QUERIES=5

# Count slow queries in the last 10 minutes (PostgreSQL example)
SLOW_COUNT=$(psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -t -c "
  SELECT count(*)
  FROM pg_stat_statements
  WHERE mean_exec_time > ${SLOW_QUERY_THRESHOLD_SEC}000
    AND calls > 10
    AND query ILIKE '%empire_%'
" 2>/dev/null | tr -d ' ' || echo 0)

SLOW_COUNT="${SLOW_COUNT:-0}"

if [ "$SLOW_COUNT" -le "$MAX_SLOW_QUERIES" ]; then
  echo "Query latency OK: ${SLOW_COUNT} slow queries detected"
  curl -s "$HEARTBEAT_URL"
else
  echo "Slow queries detected: ${SLOW_COUNT} queries exceeding ${SLOW_QUERY_THRESHOLD_SEC}s avg"
  exit 1
fi

Set the Vigilmon heartbeat to 10 minutes. Slow query proliferation after schema migrations is the most common Empire-db performance issue — run EXPLAIN ANALYZE on the specific queries Empire-db generates (enable SQL logging via DBDatabase.setErrorHandler()) to identify missing indexes.

Step 4: Monitor Transaction Throughput and Error Rates

Empire-db transactions (DBRecord.update(), DBRecord.delete()) must be explicitly committed and rolled back. When transaction error rates increase — due to constraint violations, serialization failures, or deadlocks — application error logs fill with Empire-db DBException entries, user operations fail, and data integrity errors accumulate. Track transaction error rates via application metrics:

// TransactionMetricsFilter.java — instrument Empire-db operations
public class TransactionMetrics {

    private static final AtomicLong successCount = new AtomicLong(0);
    private static final AtomicLong errorCount = new AtomicLong(0);
    private static final AtomicLong rollbackCount = new AtomicLong(0);

    public static void recordSuccess() { successCount.incrementAndGet(); }
    public static void recordError() { errorCount.incrementAndGet(); }
    public static void recordRollback() { rollbackCount.incrementAndGet(); }

    // Call this from your Empire-db transaction wrapper
    public static void executeTransaction(DBDatabase db, Connection conn,
            Runnable work) {
        try {
            work.run();
            conn.commit();
            recordSuccess();
        } catch (Exception e) {
            try { conn.rollback(); } catch (Exception re) { /* ignore */ }
            recordError();
            recordRollback();
            throw new RuntimeException(e);
        }
    }
}

// Expose via metrics endpoint
@WebServlet("/metrics/transactions")
public class TransactionMetricsServlet extends HttpServlet {
    private static final double MAX_ERROR_RATE = 0.05; // 5% error rate

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        long success = TransactionMetrics.getSuccessCount();
        long errors = TransactionMetrics.getErrorCount();
        long total = success + errors;

        double errorRate = total > 0 ? (double) errors / total : 0.0;

        resp.setContentType("application/json");

        if (errorRate <= MAX_ERROR_RATE) {
            resp.setStatus(200);
        } else {
            resp.setStatus(503);
        }

        resp.getWriter().write(String.format(
            "{\"success\":%d,\"errors\":%d,\"error_rate\":%.4f}",
            success, errors, errorRate
        ));
    }
}

1. In Vigilmon, click Add Monitor → HTTP(S).
2. Enter the URL: https://your-app.internal/metrics/transactions
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.

A sustained error rate above 5% in Empire-db transactions typically indicates a recurring constraint violation (check your DBRecord.update() calls for missing required fields), a deadlock pattern (review transaction ordering across concurrent request paths), or a database-level issue like a failed migration leaving the schema in an inconsistent state.

Step 5: Monitor Database Schema Consistency

Empire-db models the database schema as Java classes — DBTable, DBColumn, and DBRelation objects that mirror the live database structure. When database migrations add columns, rename tables, or change constraints without updating the corresponding Empire-db schema classes (or vice versa), queries silently generate incorrect SQL or throw DBException at runtime. Monitor schema consistency with a startup validation check:

#!/bin/bash
# /usr/local/bin/empire-db-schema-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
APP_URL="https://your-app.internal"
TIMEOUT=30

# Call a schema validation endpoint exposed by the application
HTTP_CODE=$(curl -s -o /tmp/empire-schema-response -w "%{http_code}" \
  --max-time "$TIMEOUT" \
  "${APP_URL}/admin/schema-check")

if [ "$HTTP_CODE" = "200" ]; then
  SCHEMA_OK=$(grep -c '"status":"ok"' /tmp/empire-schema-response || echo 0)
  if [ "$SCHEMA_OK" -gt 0 ]; then
    echo "Schema consistent with Empire-db model"
    curl -s "$HEARTBEAT_URL"
  else
    echo "Schema inconsistency detected"
    cat /tmp/empire-schema-response
    exit 1
  fi
else
  echo "Schema check endpoint failed: HTTP $HTTP_CODE"
  exit 1
fi

Add the schema validation endpoint to your application:

@WebServlet("/admin/schema-check")
public class SchemaCheckServlet extends HttpServlet {

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        DBDatabase db = ApplicationDatabase.getInstance();
        StringBuilder issues = new StringBuilder();

        try (Connection conn = db.getConnection()) {
            // Use Empire-db's built-in schema update check
            DBSchemaSyncer syncer = new DBSchemaSyncer(db);
            List<String> ddlStatements = syncer.getUpdateDDLScript(conn, false);

            if (ddlStatements.isEmpty()) {
                resp.setStatus(200);
                resp.getWriter().write("{\"status\":\"ok\",\"drift\":false}");
            } else {
                resp.setStatus(503);
                // Don't expose DDL in production — just signal drift
                resp.getWriter().write(String.format(
                    "{\"status\":\"drift\",\"pending_changes\":%d}",
                    ddlStatements.size()
                ));
            }
        } catch (Exception e) {
            resp.setStatus(503);
            resp.getWriter().write("{\"status\":\"error\"}");
        }
    }
}

Set the Vigilmon heartbeat to 15 minutes. Schema drift is typically introduced after a DBA runs a manual migration or a migration tool (Flyway, Liquibase) applies a change that wasn't reflected in the Empire-db schema classes — catching it within 15 minutes prevents a full deployment cycle of debugging.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Empire-db alerts to your backend and DBA teams:

1. Go to Alert Channels in Vigilmon.
2. Add a Slack webhook to #backend-alerts — data layer failures need immediate attention from developers.
3. Add PagerDuty for the application health and connection pool monitors — these affect all users.
4. Add email notification for slow query and schema consistency monitors (these need investigation, not immediate response).
5. Set Consecutive failures before alert to 2 for the transaction error rate monitor — brief spikes during deployments are normal.

Add maintenance windows during deployments:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "EMPIRE_DB_MONITOR_ID",
    "duration_minutes": 10,
    "reason": "Application deployment with schema migration"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (health) | /health with Empire-db probe query | JVM crash, DB connectivity, session init failure | | HTTP monitor (pool) | /metrics/pool connection pool bean | Pool exhaustion, long-held transactions | | HTTP monitor (latency) | /metrics/latency probe query | Slow queries, missing indexes, data volume growth | | HTTP monitor (transactions) | /metrics/transactions error rate | Constraint violations, deadlocks, schema issues | | Cron heartbeat (schema) | /admin/schema-check DDL drift | Post-migration schema drift, missing columns |

Empire-db's explicit SQL generation is an asset for debugging query performance — but it also means that schema drift, pool exhaustion, and slow queries are application-layer concerns that Empire-db itself won't surface as errors until a query fails at runtime. Vigilmon's external health monitoring catches these failure modes before they accumulate into data integrity issues or user-visible outages.

Start monitoring for free at vigilmon.online.