tutorial

How to Monitor AWS QLDB Uptime and Health with Vigilmon

AWS QLDB journal lag, QLDB Streams Kinesis shard iterator failures, and silent IAM permission revocations can corrupt your audit ledger without triggering a CloudWatch alarm. Learn how to monitor AWS QLDB externally with Vigilmon HTTP probes and heartbeat monitors for QLDB Streams export jobs and scheduled verification tasks.

AWS QLDB (Quantum Ledger Database) is a fully managed ledger database that provides a cryptographically verifiable history of all data changes — making it the backbone of audit trails, financial transaction logs, and compliance systems. But "cryptographically verifiable" and "actively monitored" are two different things. QLDB is accessible only from within a VPC or via the AWS SDK over HTTPS — there is no publicly reachable endpoint you can probe directly. A QLDB Streams export to Kinesis silently stops when the IAM role attached to the stream loses kinesis:PutRecord permissions; CloudWatch emits a metric, but no alert fires unless you explicitly configured an alarm. A PartiQL query that hits a document with a deeply nested QLDB history revision can time out silently on the application side without QLDB reporting an error. And the cryptographic verification operation (qldb:GetDigest, qldb:GetRevision, qldb:GetBlock) that proves your ledger has not been tampered with is asynchronous — a verification job that fails partway through produces an incomplete proof without surfacing a health signal.

Vigilmon gives you external visibility into AWS QLDB health through HTTP probe monitoring and heartbeat monitoring for QLDB Streams export jobs and scheduled verification tasks. This tutorial walks through both.

Why AWS QLDB Monitoring Needs More Than CloudWatch

AWS CloudWatch, the QLDB console, and VPC health dashboards tell you QLDB is reachable at the network layer. They cannot tell you:

  • Whether QLDB is reachable from your application servers with a valid IAM credential and is accepting authenticated PartiQL transactions
  • Whether your QLDB Streams export to Kinesis is actively delivering journal records — or has silently stalled due to an IAM role change, Kinesis shard limit, or a transient network partition
  • Whether a scheduled cryptographic verification job that proves the ledger's integrity has run successfully and produced a valid proof
  • Whether the QLDB journal lag — the delay between a committed transaction and its appearance in QLDB Streams — has grown beyond your compliance SLA
  • Whether an IAM permission revocation has silently blocked your application from inserting new transaction records into the ledger
  • Whether a PartiQL batch query that auditors use to reconcile ledger state with your operational database has timed out or returned incomplete results

These failure modes can violate audit trail completeness requirements long before your users see an error. External monitoring through Vigilmon catches them by probing the actual read/write paths your application relies on.

Step 1: Build an AWS QLDB Health Endpoint

QLDB is accessible only via the AWS SDK — it is not reachable via a raw TCP or HTTP probe. Deploy a Lambda function or ECS sidecar inside your VPC that uses the QLDB SDK to perform an authenticated PartiQL transaction and exposes the result via API Gateway or an internal ALB.

Node.js / Lambda Example (exposed via API Gateway)

// health/qldb.js — deployed as a Lambda behind API Gateway
const { QldbDriver, RetryConfig } = require('amazon-qldb-driver-nodejs');

let driver = null;

function getDriver() {
  if (driver) return driver;
  driver = new QldbDriver(
    process.env.QLDB_LEDGER_NAME,
    {
      region: process.env.AWS_REGION || 'us-east-1',
      // IAM credentials are picked up from the Lambda execution role
    },
    new RetryConfig(3)
  );
  return driver;
}

exports.handler = async (event) => {
  try {
    const qldbDriver = getDriver();
    const writeStart = Date.now();

    // Execute a lightweight PartiQL transaction
    // Insert a sentinel health-check document; table must exist in the ledger
    await qldbDriver.executeLambda(async (txn) => {
      // Use INSERT INTO ... ON CONFLICT DO NOTHING pattern to avoid duplicate accumulation
      const result = await txn.execute(
        'INSERT INTO _VigilmonHealth VALUE {\'probe\': \'vigilmon\', \'checked_at\': `now()`}'
      );
      return result;
    });

    const writeMs = Date.now() - writeStart;

    // Verify the ledger is accepting reads — select from the health table
    const readStart = Date.now();
    let rowCount = 0;
    await qldbDriver.executeLambda(async (txn) => {
      const result = await txn.execute(
        'SELECT COUNT(*) AS cnt FROM _VigilmonHealth'
      );
      for await (const page of result) {
        rowCount = page.getResultList().length;
      }
    });
    const readMs = Date.now() - readStart;

    return {
      statusCode: 200,
      body: JSON.stringify({ status: 'ok', write_ms: writeMs, read_ms: readMs }),
    };
  } catch (err) {
    const errType = err.constructor?.name || 'UnknownError';
    // InvalidSessionException — session expired or ledger unavailable
    // ResourceNotFoundException — ledger or table does not exist
    // AccessDeniedException — IAM permission revoked
    return {
      statusCode: 503,
      body: JSON.stringify({ status: 'down', error: err.message, error_type: errType }),
    };
  }
};

First-time setup: Create the _VigilmonHealth table in your ledger before deploying the health endpoint:

aws qldb execute-statement \
  --ledger-name YOUR_LEDGER_NAME \
  --statement "CREATE TABLE _VigilmonHealth"

Python Example (ECS sidecar or Lambda)

# health_qldb.py — Lambda handler or FastAPI endpoint inside VPC
import os
import time
from pyqldb.driver.qldb_driver import QldbDriver
from botocore.exceptions import ClientError

LEDGER_NAME = os.environ["QLDB_LEDGER_NAME"]
driver = QldbDriver(ledger_name=LEDGER_NAME)

def check_qldb_health():
    try:
        write_start = time.monotonic()

        def write_probe(txn):
            txn.execute_statement(
                "INSERT INTO _VigilmonHealth VALUE {'probe': 'vigilmon', 'ts': ?}",
                int(time.time())
            )

        driver.execute_lambda(write_probe)
        write_ms = int((time.monotonic() - write_start) * 1000)

        read_start = time.monotonic()

        def read_probe(txn):
            result = txn.execute_statement(
                "SELECT COUNT(*) AS cnt FROM _VigilmonHealth"
            )
            return list(result)

        rows = driver.execute_lambda(read_probe)
        read_ms = int((time.monotonic() - read_start) * 1000)

        return {"status": "ok", "write_ms": write_ms, "read_ms": read_ms}, 200

    except ClientError as e:
        code = e.response["Error"]["Code"]
        return {"status": "down", "error": str(e), "aws_error_code": code}, 503

def lambda_handler(event, context):
    import json
    result, status_code = check_qldb_health()
    return {
        "statusCode": status_code,
        "body": json.dumps(result),
    }

Verify the endpoint before wiring up Vigilmon:

curl -i https://your-api.example.com/health/qldb
# HTTP/1.1 200 OK
# {"status":"ok","write_ms":38,"read_ms":22}

Step 2: Configure Vigilmon HTTP Monitor for AWS QLDB

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-api.example.com/health/qldb
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously, requiring multi-region consensus before opening an incident. This eliminates false alarms from transient single-probe network issues and distinguishes genuine QLDB service degradation from Lambda cold-start latency spikes.

Setting a Higher Response Time Threshold for QLDB

QLDB PartiQL transactions have higher baseline latency than traditional SQL queries — a write + read round-trip via the QLDB driver can take 100–400ms under normal conditions due to the cryptographic journal commit overhead. Configure your Vigilmon response time thresholds accordingly:

  • Alert at 5000ms — signals QLDB is under load or the ledger is experiencing elevated journal commit latency
  • Set a P1 page threshold at 15000ms to alert before Vigilmon marks the monitor fully down

Step 3: Heartbeat Monitoring for QLDB Streams and Verification Jobs

QLDB Streams Monitoring

QLDB Streams exports a continuous stream of journal records to Kinesis Data Streams. A stream that stops delivering records means your downstream consumers — real-time audit dashboards, compliance systems, or event-driven microservices — are operating on stale data. The stream can stall due to:

  • The IAM execution role losing kinesis:PutRecord or kinesis:DescribeStream permissions
  • Kinesis shard limits being exceeded without auto-scaling enabled
  • A transient QLDB service event that pauses the journal

Vigilmon heartbeat monitors detect these stalls: a lightweight consumer reads the Kinesis stream and pings Vigilmon whenever it successfully processes records. If pings stop, Vigilmon fires an alert.

Scheduled Verification Job Monitoring

QLDB's cryptographic digest verification (GetDigest, GetRevision, GetBlock) proves the ledger has not been tampered with. For compliance requirements, this verification should run on a schedule. A verification job that fails partway through produces an incomplete proof without raising an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: qldb-streams-kinesis (or qldb-digest-verification)
  3. Set the expected interval: 5 minutes for QLDB Streams consumer; 24 hours for digest verification
  4. Set the grace period: 15 minutes for streams; 2 hours for verification
  5. Save — copy the unique heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Python Heartbeat Example for QLDB Streams Consumer

# qldb_streams_consumer.py — runs as a Lambda triggered by Kinesis
import os
import json
import requests
import time

HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]
last_heartbeat_at = 0
HEARTBEAT_INTERVAL_SECONDS = 240  # 4 minutes (Vigilmon grace: 15 min)

def lambda_handler(event, context):
    global last_heartbeat_at

    records_processed = 0
    for record in event.get("Records", []):
        try:
            payload = json.loads(record["kinesis"]["data"])
            qldb_event_type = payload.get("recordType", "UNKNOWN")

            if qldb_event_type == "REVISION_DETAILS":
                # Process the QLDB journal revision
                revision = payload.get("payload", {}).get("revision", {})
                process_revision(revision)
                records_processed += 1

        except Exception as e:
            print(f"Failed to process Kinesis record: {e}")
            # Re-raise to mark the Lambda as failed and trigger Kinesis retry
            raise

    print(f"Processed {records_processed} QLDB revision records")

    # Ping Vigilmon if records were processed and enough time has passed
    now = time.time()
    if records_processed > 0 and (now - last_heartbeat_at) >= HEARTBEAT_INTERVAL_SECONDS:
        requests.get(HEARTBEAT_URL, timeout=5)
        last_heartbeat_at = now
        print("Heartbeat sent to Vigilmon")

def process_revision(revision):
    # Your revision processing logic
    pass

Python Heartbeat Example for QLDB Digest Verification

# qldb_verify.py — run as a scheduled Lambda or ECS task
import os
import boto3
import requests
from pyqldb.driver.qldb_driver import QldbDriver

LEDGER_NAME = os.environ["QLDB_LEDGER_NAME"]
TABLE_NAME = os.environ.get("QLDB_TABLE_TO_VERIFY", "Transactions")
HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]

qldb_client = boto3.client("qldb")
qldb_session_client = boto3.client("qldb-session")
driver = QldbDriver(ledger_name=LEDGER_NAME)

def verify_ledger_digest():
    print(f"Starting QLDB digest verification for ledger: {LEDGER_NAME}")

    # Get the current digest
    digest_response = qldb_client.get_digest(Name=LEDGER_NAME)
    tip_address = digest_response["DigestTipAddress"]
    digest = digest_response["Digest"]

    print(f"Ledger tip block: {tip_address}")
    print(f"Digest: {digest.hex() if isinstance(digest, bytes) else digest}")

    # Get a recent document to verify its revision proof
    def get_recent_doc_id(txn):
        result = txn.execute_statement(
            f"SELECT metadata.id FROM {TABLE_NAME} LIMIT 1"
        )
        rows = list(result)
        if not rows:
            raise ValueError(f"No documents found in table: {TABLE_NAME}")
        return rows[0]["id"]

    doc_id = driver.execute_lambda(get_recent_doc_id)

    # Get the block address for this document
    revision_response = qldb_client.get_revision(
        Name=LEDGER_NAME,
        BlockAddress=tip_address,
        DocumentId=doc_id,
        DigestTipAddress=tip_address,
    )

    proof = revision_response.get("Proof")
    if not proof:
        raise ValueError("No proof returned from GetRevision — verification incomplete")

    print(f"Cryptographic proof obtained for document {doc_id}")
    print("QLDB digest verification successful")

    # Ping Vigilmon — verification completed
    requests.get(HEARTBEAT_URL, timeout=5)
    print("Heartbeat sent to Vigilmon")

def lambda_handler(event, context):
    verify_ledger_digest()

if __name__ == "__main__":
    verify_ledger_digest()

Step 4: Alert Routing and Response Time Thresholds

Configure alert routing in Vigilmon to match the severity of each QLDB failure mode:

| Monitor | Alert Channel | Priority | |---|---|---| | QLDB health /health/qldb | Slack + PagerDuty | P1 | | Heartbeat: QLDB Streams Kinesis consumer | Slack + email | P1 | | Heartbeat: digest verification job | Slack + email | P2 |

Set response time thresholds as early warnings before full availability failures:

  • Alert at 5000ms for the health endpoint — QLDB journal commit latency above 5 seconds indicates ledger capacity pressure or an AWS service event
  • Alert at `15000ms** as a P1 threshold — signals the ledger is severely degraded or the Lambda health-check function is cold-starting excessively
  • For QLDB Streams: configure a P1 heartbeat — a stalled Kinesis stream means your downstream compliance consumers have a data gap in the audit trail, which may trigger a compliance incident

Summary

AWS QLDB failures surface in subtle ways — silent IAM permission revocations blocking ledger writes, stalled Kinesis Streams creating audit trail gaps, and incomplete cryptographic verification jobs — long before your compliance team discovers the gap in the audit log. Vigilmon gives you external visibility across the full failure surface:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/qldb | Authenticated PartiQL write/read, IAM credential validity, ledger availability | | Heartbeat monitor: QLDB Streams consumer | Kinesis export delivery continuity and journal lag detection | | Heartbeat monitor: digest verification | Cryptographic proof completeness and scheduled verification job health |

Get started free at vigilmon.online — your first QLDB monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles