tutorial

Monitoring ReadySet with Vigilmon

ReadySet is a wire-compatible SQL caching proxy that accelerates MySQL and PostgreSQL reads — but cache hit rate drops and CDC replication lag can silently degrade performance. Here's how to monitor ReadySet end-to-end with Vigilmon.

ReadySet is an open source SQL caching layer that sits between your application and your upstream MySQL or PostgreSQL database. It speaks the native wire protocol — your application thinks it's talking directly to the database — but ReadySet automatically caches the results of frequently-executed SELECT queries in memory and serves them without touching the primary database.

Unlike Redis or Memcached, ReadySet doesn't require application-level cache management. It uses Change Data Capture (CDC) — MySQL binlog or PostgreSQL logical replication — to automatically invalidate cache entries when the upstream data changes. The result is transparent query acceleration with no application code changes.

That power comes with operational considerations. Cache hit rate, CDC replication lag, and upstream connectivity are all critical to ReadySet's correctness. Vigilmon gives you continuous visibility into all of them.

What You'll Set Up

  • TCP availability monitor for the ReadySet proxy endpoint
  • Heartbeat monitors for cache hit rate, CDC lag, and memory utilization
  • Upstream database connectivity monitoring
  • Alert thresholds for stale data, cache saturation, and adapter health

Prerequisites

  • ReadySet deployed with upstream MySQL (port 3306) or PostgreSQL (port 5432) configured
  • ReadySet Server and Adapter processes running
  • A free Vigilmon account

Step 1: Monitor ReadySet Proxy Availability

ReadySet's core job is to be an always-available query proxy. If the proxy endpoint is down, queries either fail entirely or fall back to the upstream database at degraded performance (depending on your application's connection handling).

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to TCP Port.
  3. Enter your ReadySet server hostname and port:
    • PostgreSQL-compatible mode: 5432
    • MySQL-compatible mode: 3306
  4. Set Check interval to 1 minute.
  5. Click Save.

A TCP check confirms the ReadySet Adapter process is listening. For a more complete health check, add an HTTP monitor if ReadySet exposes a health endpoint (check your version's documentation for /health or /status):

http://readyset.yourdomain.com:6034/health

ReadySet exposes internal metrics on port 6034 by default. A successful response confirms both the Adapter and Server processes are running.

Step 2: Monitor Cache Hit Rate

The cache hit rate is the most important ReadySet performance metric. A high hit rate (>80%) means ReadySet is delivering its core value — queries served from memory without touching the database. A drop in hit rate means:

  • New query patterns have emerged that ReadySet hasn't cached yet
  • A cache was dropped (e.g., after a ReadySet restart)
  • Query parameters are too varied for caching to be effective

Add a heartbeat that checks ReadySet's internal metrics for hit rate:

#!/usr/bin/env python3
"""
Queries ReadySet's Prometheus metrics endpoint for cache hit/miss counters
and alerts if the hit rate drops below the expected threshold.
"""
import requests
import sys
import re

READYSET_METRICS = "http://readyset.yourdomain.com:6034/metrics"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_CACHE_HIT_ID"
MIN_HIT_RATE = 0.70  # alert if cache hit rate drops below 70%

resp = requests.get(READYSET_METRICS, timeout=10)
resp.raise_for_status()
text = resp.text

def extract_metric(text, name):
    for line in text.splitlines():
        if line.startswith(name + " ") or line.startswith(name + "{"):
            match = re.search(r"\} (\S+)$", line) or re.search(r" (\S+)$", line)
            if match:
                return float(match.group(1))
    return None

# ReadySet exposes readyset_query_log_cache_hit_total and readyset_query_log_cache_miss_total
hits = extract_metric(text, "readyset_query_log_cache_hit_total")
misses = extract_metric(text, "readyset_query_log_cache_miss_total")

if hits is None or misses is None:
    # Metrics not yet available (ReadySet just started)
    requests.get(HEARTBEAT_URL, timeout=5)
    sys.exit(0)

total = hits + misses
hit_rate = hits / total if total > 0 else 1.0

if hit_rate >= MIN_HIT_RATE:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print(f"Cache hit rate {hit_rate:.1%} below threshold {MIN_HIT_RATE:.1%}")
    sys.exit(1)

Schedule every 5 minutes.

Step 3: Monitor CDC Replication Lag

ReadySet maintains cache freshness by tracking changes from the upstream database via CDC. For MySQL, it reads the binary log (binlog). For PostgreSQL, it consumes a logical replication slot. Replication lag is the time between a data change in the upstream database and ReadySet updating its cached results.

Lag above 1 second means ReadySet may be serving stale data — a correctness problem, not just a performance problem.

Add a replication lag probe:

#!/usr/bin/env python3
"""
Checks ReadySet's replication lag from its internal metrics.
Alert threshold: >1000ms indicates potentially stale cache results.
"""
import requests
import sys
import re

READYSET_METRICS = "http://readyset.yourdomain.com:6034/metrics"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_LAG_ID"
MAX_LAG_MS = 1000  # 1 second

resp = requests.get(READYSET_METRICS, timeout=10)
resp.raise_for_status()

# ReadySet exposes readyset_replication_lag_ms or similar
for line in resp.text.splitlines():
    if "replication_lag" in line and not line.startswith("#"):
        match = re.search(r"(\S+)$", line)
        if match:
            lag_ms = float(match.group(1))
            if lag_ms <= MAX_LAG_MS:
                requests.get(HEARTBEAT_URL, timeout=5)
            else:
                print(f"Replication lag {lag_ms:.0f}ms exceeds threshold {MAX_LAG_MS}ms")
                sys.exit(1)
            sys.exit(0)

# No lag metric found — treat as healthy (metrics may vary by version)
requests.get(HEARTBEAT_URL, timeout=5)

Schedule every 2 minutes. Replication lag spikes often indicate:

  • Upstream database write surge that CDC can't keep up with
  • Network congestion between ReadySet and the upstream database
  • Replication slot falling behind (PostgreSQL)

Step 4: Monitor Upstream Database Connectivity

ReadySet must maintain a persistent CDC connection to the upstream database. If this connection is lost, ReadySet stops processing change events and begins serving increasingly stale data — without necessarily returning an error to clients.

Add a TCP monitor for your upstream database:

For PostgreSQL upstream:

  1. Click Add Monitor → TCP Port.
  2. Enter your upstream PostgreSQL hostname and port 5432.
  3. Check interval: 1 minute.
  4. Name it "ReadySet upstream PostgreSQL".

For MySQL upstream:

  1. Click Add Monitor → TCP Port.
  2. Enter your upstream MySQL hostname and port 3306.
  3. Check interval: 1 minute.
  4. Name it "ReadySet upstream MySQL".

Also add a replication slot health heartbeat for PostgreSQL to catch slot bloat:

#!/usr/bin/env python3
"""
Checks the ReadySet replication slot's retained WAL size.
A growing slot means ReadySet is falling behind upstream write rate.
"""
import psycopg2
import requests
import sys

DB_CONFIG = {
    "host": "postgres.yourdomain.com",
    "port": 5432,
    "user": "monitor",
    "password": "YOUR_MONITOR_PASSWORD",
    "dbname": "postgres",
}
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_SLOT_ID"
MAX_LAG_BYTES = 100 * 1024 * 1024  # 100 MB

conn = psycopg2.connect(**DB_CONFIG)
with conn.cursor() as cur:
    cur.execute("""
        SELECT slot_name, active, pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS lag_bytes
        FROM pg_replication_slots
        WHERE plugin = 'pgoutput' OR slot_name ILIKE '%readyset%'
    """)
    rows = cur.fetchall()
conn.close()

if not rows:
    print("No ReadySet replication slot found")
    sys.exit(1)

for slot_name, active, lag_bytes in rows:
    if not active:
        print(f"Replication slot {slot_name} is inactive")
        sys.exit(1)
    if lag_bytes > MAX_LAG_BYTES:
        print(f"Slot {slot_name} lag {lag_bytes / 1024 / 1024:.1f}MB exceeds threshold")
        sys.exit(1)

requests.get(HEARTBEAT_URL, timeout=5)

Schedule every 5 minutes.

Step 5: Monitor Memory Utilization

ReadySet stores all cached query results in memory. When memory utilization is high, the operating system may evict pages or ReadySet may begin dropping caches, causing a sudden hit rate collapse as caches need to be rebuilt from the upstream database.

Add a memory utilization heartbeat:

#!/usr/bin/env python3
"""Checks ReadySet's memory usage against a configurable threshold."""
import requests
import sys
import re

READYSET_METRICS = "http://readyset.yourdomain.com:6034/metrics"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_MEMORY_ID"
MAX_MEMORY_BYTES = 6 * 1024 ** 3  # 6 GB — adjust to your ReadySet instance size

resp = requests.get(READYSET_METRICS, timeout=10)
resp.raise_for_status()

for line in resp.text.splitlines():
    if "process_resident_memory_bytes" in line and not line.startswith("#"):
        match = re.search(r"(\S+)$", line)
        if match:
            mem_bytes = float(match.group(1))
            if mem_bytes < MAX_MEMORY_BYTES:
                requests.get(HEARTBEAT_URL, timeout=5)
            else:
                gb = mem_bytes / 1024 ** 3
                print(f"ReadySet memory {gb:.1f}GB exceeds threshold")
                sys.exit(1)
            sys.exit(0)

requests.get(HEARTBEAT_URL, timeout=5)

Schedule every 5 minutes. Set the Vigilmon heartbeat interval to 10 minutes to avoid false alerts during normal memory fluctuations.

Step 6: Monitor ReadySet Server Process Health

The ReadySet Server maintains the in-memory cache state. The ReadySet Adapter processes handle client connections and proxy them to the Server. Frequent Server restarts indicate stability issues — possibly related to memory pressure, upstream connectivity problems, or bugs in cache invalidation.

Add a restart-count heartbeat:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_SERVER_ID"
MAX_RESTARTS=3  # alert if readyset-server restarted more than 3 times today

# Check systemd restart count (adjust unit name to match your deployment)
RESTARTS=$(systemctl show readyset-server --property=NRestarts --value 2>/dev/null || echo "0")

if [ "${RESTARTS:-0}" -le "$MAX_RESTARTS" ]; then
    curl -s "$HEARTBEAT_URL" > /dev/null
else
    echo "ReadySet Server restarted ${RESTARTS} times today" >&2
fi

For Docker/Kubernetes deployments, check container restart count instead:

RESTARTS=$(docker inspect readyset-server --format '{{.RestartCount}}' 2>/dev/null || echo "0")

Schedule every 15 minutes.

Step 7: Configure Alert Routing

ReadySet failures have different urgency levels depending on whether they affect correctness (stale data) or performance (cache miss):

| Monitor | Urgency | Alert Action | |---------|---------|-------------| | Proxy TCP down | Critical | Page on-call immediately | | CDC replication lag >1s | High | Alert within 2 minutes | | Cache hit rate <70% | Medium | Slack alert to database team | | Upstream DB connectivity lost | Critical | Page on-call immediately | | Replication slot lag >100MB | High | Alert within 5 minutes | | Memory >85% threshold | High | Alert to ops team | | Server restart count high | Medium | Slack alert to database team |

In Vigilmon:

  1. Go to Settings → Alerts.
  2. Add your PagerDuty, Slack, or email channels.
  3. Apply escalation policies per monitor.

CDC replication lag is the highest-correctness risk. When ReadySet is serving stale data, applications may make decisions based on outdated records. Alert on this within 2 minutes of detection.

Key Metrics Summary

| Monitor | Type | Interval | Alert Threshold | |---------|------|----------|----------------| | ReadySet proxy (TCP) | TCP | 1 min | 1 failure | | ReadySet health endpoint | HTTP | 1 min | 1 failure | | Cache hit rate | Heartbeat | 5 min | 1 miss (<70%) | | CDC replication lag | Heartbeat | 2 min | 1 miss (>1s) | | Upstream DB (TCP) | TCP | 1 min | 1 failure | | Replication slot lag | Heartbeat | 5 min | 1 miss (>100MB) | | Memory utilization | Heartbeat | 5 min | 2 misses (>85%) | | Server restart count | Heartbeat | 15 min | 1 miss (>3/day) |

Conclusion

ReadySet's transparency is its strength and its risk. Because it speaks the native database protocol, your application has no visibility into whether queries are hitting the cache or falling through to the upstream database. A silent cache collapse — from a restart, memory pressure, or CDC falling behind — looks to your application like a suddenly slow database.

Vigilmon gives you the external visibility that your application can't provide: continuous checks on the proxy endpoint, cache hit rate, CDC replication lag, and upstream connectivity. You'll know the moment ReadySet starts serving stale or uncached results, long before your users notice query latency climbing.

Get started at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles