Monitoring AWS Keyspaces with Vigilmon: Managed Cassandra Health, CQL Availability & Background Job Heartbeats

AWS Keyspaces (for Apache Cassandra) is a fully managed, serverless, highly available Cassandra-compatible database service. You get Cassandra's CQL query language and table model without managing nodes, repairs, or compaction — but the managed model introduces its own class of failures. SigV4 authentication misconfigurations, capacity mode throttling (provisioned vs. on-demand), CQL driver version incompatibilities, and missing table-specific configurations can all degrade your application while Keyspaces' CloudWatch metrics report no cluster-level problems. Vigilmon adds an independent external monitoring layer: HTTP health checks on your Keyspaces-connected services that probe real CQL query availability and alert before users encounter errors.

What You'll Build

• A health endpoint that executes a real CQL query against AWS Keyspaces
• A Vigilmon HTTP monitor with appropriate timeout and assertion settings
• Throttling detection to catch capacity mode issues early
• Cron heartbeat monitoring for background Keyspaces workers
• An alerting strategy tuned to AWS Keyspaces' serverless failure modes

Prerequisites

• An AWS account with a Keyspaces keyspace and table
• IAM credentials or an IAM role with cassandra:Select and cassandra:Modify permissions
• The Amazon Keyspaces service endpoint certificate for your region
• An application service connecting to Keyspaces via a CQL driver
• A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Keyspaces Service

Add a /health route that executes a lightweight CQL query against a dedicated health row to verify real connectivity, authentication, and query availability.

Node.js (cassandra-driver with SigV4 auth)

// routes/health.js
import cassandra from 'cassandra-driver';
import { SigV4AuthProvider } from 'aws-sigv4-auth-cassandra-plugin';

const authProvider = new SigV4AuthProvider({ region: process.env.AWS_REGION });

const client = new cassandra.Client({
  contactPoints: [`cassandra.${process.env.AWS_REGION}.amazonaws.com`],
  localDataCenter: process.env.AWS_REGION,
  authProvider,
  sslOptions: { rejectUnauthorized: true },
  protocolOptions: { port: 9142 },
  socketOptions: { connectTimeout: 5000, readTimeout: 8000 },
  queryOptions: { consistency: cassandra.types.consistencies.localQuorum },
});

const HEALTH_TABLE = process.env.KEYSPACES_HEALTH_TABLE || 'healthcheck';
const KEYSPACE = process.env.KEYSPACES_KEYSPACE;

export async function healthHandler(req, res) {
  const checks = {};
  let ok = true;
  const start = Date.now();

  // Check 1: Write probe — verify write availability
  try {
    await client.execute(
      `INSERT INTO ${KEYSPACE}.${HEALTH_TABLE} (id, ts) VALUES (?, ?) USING TTL 300`,
      ['vigilmon', new Date()],
      { prepare: true }
    );
    checks.write = 'ok';
    checks.writeLatencyMs = Date.now() - start;
  } catch (err) {
    checks.write = `error: ${err.message}`;
    // Detect throttling specifically
    if (err.message?.includes('Rate exceeded') || err.code === 0x1200) {
      checks.writeThrottled = true;
    }
    ok = false;
  }

  // Check 2: Read probe — verify read availability
  const readStart = Date.now();
  try {
    const result = await client.execute(
      `SELECT id, ts FROM ${KEYSPACE}.${HEALTH_TABLE} WHERE id = ?`,
      ['vigilmon'],
      { prepare: true }
    );
    checks.read = result.rows.length > 0 ? 'ok' : 'row not found';
    checks.readLatencyMs = Date.now() - readStart;
    if (result.rows.length === 0) ok = false;
  } catch (err) {
    checks.read = `error: ${err.message}`;
    if (err.message?.includes('Rate exceeded') || err.code === 0x1200) {
      checks.readThrottled = true;
    }
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    service: 'keyspaces-service',
    checks,
  });
}

Python (cassandra-driver with SigV4)

# routers/health.py
import os, time, datetime
from cassandra.cluster import Cluster
from cassandra.auth import PlainTextAuthProvider
from cassandra.policies import DCAwareRoundRobinPolicy
from ssl import SSLContext, PROTOCOL_TLS_CLIENT, CERT_REQUIRED
from fastapi import APIRouter

router = APIRouter()

# AWS Keyspaces requires SigV4 via SSL on port 9142
# Use the cassandra-sigv4 package for SigV4 auth
from cassandra_sigv4.auth import SigV4AuthProvider as SigV4

_auth_provider = SigV4(
    region=os.environ['AWS_REGION'],
)

_ssl_context = SSLContext(PROTOCOL_TLS_CLIENT)
_ssl_context.check_hostname = True
_ssl_context.verify_mode = CERT_REQUIRED
_ssl_context.load_verify_locations('/path/to/sf-class2-root.crt')

_cluster = Cluster(
    contact_points=[f"cassandra.{os.environ['AWS_REGION']}.amazonaws.com"],
    port=9142,
    auth_provider=_auth_provider,
    ssl_context=_ssl_context,
    connect_timeout=5,
)
_session = _cluster.connect()

KEYSPACE = os.environ['KEYSPACES_KEYSPACE']
HEALTH_TABLE = os.environ.get('KEYSPACES_HEALTH_TABLE', 'healthcheck')

@router.get("/health")
async def health():
    checks = {}
    ok = True
    start = time.monotonic()

    # Check 1: Write probe
    try:
        _session.execute(
            f"INSERT INTO {KEYSPACE}.{HEALTH_TABLE} (id, ts) VALUES (%s, %s) USING TTL 300",
            ('vigilmon', datetime.datetime.utcnow()),
        )
        checks['write'] = 'ok'
        checks['writeLatencyMs'] = int((time.monotonic() - start) * 1000)
    except Exception as e:
        msg = str(e)
        checks['write'] = f'error: {msg}'
        if 'Rate exceeded' in msg:
            checks['writeThrottled'] = True
        ok = False

    # Check 2: Read probe
    read_start = time.monotonic()
    try:
        result = _session.execute(
            f"SELECT id, ts FROM {KEYSPACE}.{HEALTH_TABLE} WHERE id = %s",
            ('vigilmon',),
        )
        rows = list(result)
        checks['read'] = 'ok' if rows else 'row not found'
        checks['readLatencyMs'] = int((time.monotonic() - read_start) * 1000)
        if not rows:
            ok = False
    except Exception as e:
        msg = str(e)
        checks['read'] = f'error: {msg}'
        if 'Rate exceeded' in msg:
            checks['readThrottled'] = True
        ok = False

    return {
        'status': 'ok' if ok else 'degraded',
        'service': 'keyspaces-service',
        'checks': checks,
    }

Required health table schema

Create a minimal health table in your Keyspaces keyspace before running the health checks:

CREATE TABLE IF NOT EXISTS your_keyspace.healthcheck (
  id text PRIMARY KEY,
  ts timestamp
) WITH default_time_to_live = 300;

The 5-minute TTL ensures the health probe row is automatically cleaned up and doesn't accumulate storage.

Step 2: Configure the Vigilmon HTTP Monitor

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: your service's health endpoint (e.g. https://api.example.com/health).
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. JSON assertion: path status, expected value ok.
7. Click Save.

Tip: AWS Keyspaces enforces a warmup period for provisioned tables — newly created tables can take several minutes to reach full throughput. If you're running health checks on a recently provisioned table, use on-demand capacity mode for your health table to avoid throttling during initial scale-up.

Throttling detection: If your health endpoint returns writeThrottled: true or readThrottled: true, add a JSON assertion on these fields. A throttled response still returns HTTP 503, but the throttle flag helps you distinguish capacity mode exhaustion from a genuine connectivity failure.

Step 3: Heartbeat Monitoring for Background Keyspaces Jobs

AWS Keyspaces is often used in time-series and event log workloads with scheduled batch writers. Use Vigilmon's Heartbeat monitor to confirm these jobs run on schedule.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to match your job frequency.
3. Copy the heartbeat URL.
4. Ping the heartbeat at the end of each successful run:

// Node.js — scheduled Keyspaces batch writer
import cassandra from 'cassandra-driver';
import fetch from 'node-fetch';

export async function runBatchWriter(client) {
  try {
    const batch = [];
    const events = await fetchPendingEvents();  // your data source

    for (const event of events) {
      batch.push({
        query: 'INSERT INTO events_log (id, ts, payload) VALUES (?, ?, ?)',
        params: [event.id, event.timestamp, JSON.stringify(event.data)],
      });
    }

    await client.batch(batch, { prepare: true });

    // Signal success to Vigilmon
    await fetch(process.env.VIGILMON_HEARTBEAT_URL, { method: 'POST' });
    console.log(`Wrote ${events.length} events — heartbeat sent.`);
  } catch (err) {
    console.error('Batch write failed — heartbeat NOT sent:', err.message);
  }
}

# Python — scheduled Keyspaces event writer
import os, requests

def run_batch_writer(session):
    try:
        events = fetch_pending_events()  # your data source
        batch = BatchStatement()
        prepared = session.prepare(
            'INSERT INTO events_log (id, ts, payload) VALUES (?, ?, ?)'
        )
        for event in events:
            batch.add(prepared, (event['id'], event['timestamp'], str(event['data'])))
        session.execute(batch)

        # Signal success to Vigilmon
        requests.post(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)
        print(f'Wrote {len(events)} events — heartbeat sent.')
    except Exception as e:
        print(f'Batch write failed — heartbeat NOT sent: {e}')

Set the heartbeat grace period to 3× your expected job interval. If your batch runs every 15 minutes, set the grace period to 45 minutes.

Step 4: Alert Routing

| Monitor type | Target | Alert channel | Priority | |---|---|---|---| | HTTP | /health (write + read CQL probe) | Email + PagerDuty | Critical | | HTTP | /health (latency assertion) | Slack | Warning | | HTTP | /health with writeThrottled assertion | Slack + Email | High | | Cron Heartbeat | Time-series / event log writer | Email + PagerDuty | Critical | | Cron Heartbeat | Batch analytics job | Slack | Warning |

Configure consecutive failures to 2 before alerting — the CQL driver's connection re-establishment after a brief network interruption can cause a single probe timeout that resolves on the next check.

What Vigilmon Catches That CloudWatch Misses

| Scenario | CloudWatch | Vigilmon | |---|---|---| | SigV4 credentials expired or rotated | No app-layer metric | Health probe returns auth error → alert | | Table throttled (provisioned capacity exhausted) | UserErrors metric exists | Health endpoint returns throttled flag + 503 | | IAM policy change blocks CQL operations | No application-layer alert | Write or read probe fails immediately | | Batch writer job silently stopped | No job-level metric | Heartbeat grace period expires → alert | | CQL driver version incompatibility after update | Not detectable by metrics | Health probe query fails — alert fires | | External connectivity to Keyspaces endpoint lost | Not externally observable | Vigilmon probe is external and independent |

AWS Keyspaces delivers Cassandra compatibility without operational overhead, but the serverless model creates failure modes — throttling, IAM changes, driver updates — that managed metrics alone won't catch at the application layer. External health checks with Vigilmon give you the independent signal your users experience.

Start monitoring your AWS Keyspaces applications in under 5 minutes — register free at vigilmon.online.