tutorial

Monitoring Skytable with Vigilmon

Skytable is a high-performance NoSQL database written in Rust with BlueQL, ACID guarantees, and in-memory operation with optional persistence. Here's how to monitor daemon health, REST API, query throughput, and data integrity with Vigilmon.

Skytable (formerly TerrabaseDB) is an open-source, high-performance NoSQL database written in Rust. It provides a custom query language called BlueQL, supports key-value, document, and structured table data models within named keyspaces, and delivers ACID guarantees with ANSI-standard isolation levels. Skytable runs primarily in-memory with optional disk persistence and uses the Skyhash binary protocol for high-throughput client communication. The skyd daemon exposes two ports: TCP 2003 for the Skyhash client protocol and HTTP 2004 for the management REST API. Vigilmon covers both ports with TCP and HTTP monitors, while cron heartbeats handle persistence health and data integrity checks.

What You'll Set Up

  • TCP liveness check on Skyhash protocol port 2003
  • REST API health check on port 2004
  • Query throughput and latency monitoring
  • Memory usage alert before OOM risk
  • Disk persistence health via cron heartbeat
  • Connection pool and keyspace size monitoring

Prerequisites

  • Skytable skyd daemon running and accessible
  • TCP port 2003 (Skyhash) and 2004 (REST/HTTP management) reachable
  • curl or skytable-client CLI available for scripted checks
  • A free Vigilmon account

Step 1: TCP Liveness Check on the Skyhash Port

The Skyhash protocol port (default 2003) is the primary client interface. If it goes down, all application reads and writes via the Skytable client library fail immediately:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to TCP.
  3. Enter your-skytable-host:2003.
  4. Set Check interval to 30 seconds.
  5. Set Alert threshold to 2 consecutive failures.
  6. Click Save.

A TCP failure on 2003 means the skyd daemon has stopped or the host is unreachable.

Step 2: REST API Health Check on Port 2004

Skytable 0.8+ exposes an HTTP management interface on port 2004. Use this as your primary HTTP health probe since it returns a structured response:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter http://your-skytable-host:2004/health (check your Skytable version docs for the exact health endpoint path; earlier versions may use / or /v1/health).
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Set Alert threshold to 2 consecutive failures.
  6. Click Save.

If your Skytable version does not expose a /health path, use the root REST API endpoint and check for a 200 response:

# Test the health endpoint manually first
curl -s -o /dev/null -w "%{http_code}" http://your-skytable-host:2004/

For versions where you need to authenticate the REST API, set the Custom HTTP headers field in Vigilmon:

Authorization: Token YOUR_SKYTABLE_TOKEN

Step 3: Monitor Query Throughput

Skytable does not expose a built-in Prometheus metrics endpoint in all versions, so you need a sidecar or scripted check that queries the management API for throughput statistics. If the REST API exposes a /v1/metrics or /v1/stats endpoint, poll it:

#!/bin/bash
# /usr/local/bin/skytable-throughput-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_THROUGHPUT_HEARTBEAT_ID"
STATS_URL="http://127.0.0.1:2004/v1/metrics"

RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "$STATS_URL")

if [ "$RESPONSE" = "200" ]; then
  # REST API is responsive — metrics are available
  curl -s "$HEARTBEAT_URL" > /dev/null
fi
# If the REST API is down or errors, heartbeat is skipped → Vigilmon alerts

For a richer throughput check that uses the Skytable Rust client library to issue BlueQL queries and measure response time:

use skytable::{Connection, Query};
use std::time::Instant;

async fn throughput_check() -> Result<(), Box<dyn std::error::Error>> {
    let mut con = Connection::new("127.0.0.1", 2003)?;

    let start = Instant::now();
    let mut count = 0u64;

    // Run 1000 test queries and measure throughput
    for i in 0..1000 {
        let q = Query::from("SET").arg(format!("bench_key_{}", i)).arg("ok");
        con.run_query::<String>(q)?;
        count += 1;
    }

    let elapsed = start.elapsed().as_millis();
    let ops_per_sec = (count * 1000) / elapsed as u64;

    // Cleanup
    for i in 0..1000 {
        let q = Query::from("DEL").arg(format!("bench_key_{}", i));
        con.run_query::<String>(q)?;
    }

    println!("Throughput: {} ops/sec", ops_per_sec);
    Ok(())
}

Run this as a scheduled job and emit a heartbeat only when throughput is within acceptable bounds.

Step 4: Memory Usage Alert

Skytable is primarily an in-memory database. Monitor the process RSS memory to catch runaway data accumulation before the kernel OOM-kills the skyd process:

#!/bin/bash
# /usr/local/bin/skytable-memory-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_MEMORY_HEARTBEAT_ID"
THRESHOLD_PERCENT=80  # alert at 80% of system RAM

TOTAL_KB=$(grep MemTotal /proc/meminfo | awk '{print $2}')
SKYD_PID=$(pgrep -x skyd)

if [ -z "$SKYD_PID" ]; then
  echo "skyd not running"
  exit 1
fi

RSS_KB=$(awk '/^VmRSS/{print $2}' /proc/"$SKYD_PID"/status)
USED_PERCENT=$(( RSS_KB * 100 / TOTAL_KB ))

if [ "$USED_PERCENT" -lt "$THRESHOLD_PERCENT" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi
# If over threshold, heartbeat is skipped → Vigilmon alerts

Schedule every 2 minutes:

*/2 * * * * /usr/local/bin/skytable-memory-check.sh

An 80% threshold gives the OS enough headroom to avoid swapping Skytable's working set, which would cause severe latency spikes.

Step 5: Disk Persistence Health

If you run Skytable with persistence enabled (the bgsave or snapshot feature in your Skytable configuration), monitor persistence job completion via a cron heartbeat:

#!/bin/bash
# /usr/local/bin/skytable-persist-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PERSIST_HEARTBEAT_ID"
SKYTABLE_DATA_DIR="/var/lib/skytable"
MAX_AGE_MINUTES=30  # alert if no persistence file updated in 30 minutes

# Find the most recently modified persistence file
LATEST_MOD=$(find "$SKYTABLE_DATA_DIR" -name "*.bin" -o -name "*.sdb" \
  2>/dev/null | xargs stat -c %Y 2>/dev/null | sort -n | tail -1)

if [ -z "$LATEST_MOD" ]; then
  # No persistence files found — skip heartbeat if persistence is expected
  exit 1
fi

NOW=$(date +%s)
AGE_MINUTES=$(( (NOW - LATEST_MOD) / 60 ))

if [ "$AGE_MINUTES" -lt "$MAX_AGE_MINUTES" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

Add a Vigilmon cron heartbeat with a 35-minute interval (allowing 5 minutes of slack beyond the 30-minute max age check):

*/10 * * * * /usr/local/bin/skytable-persist-check.sh

Step 6: Connection Count Monitoring

Track the number of active client connections to catch application reconnect failures or connection pool exhaustion. Use the management REST API if it exposes connection stats, or check via the OS:

#!/bin/bash
# /usr/local/bin/skytable-connections-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CONN_HEARTBEAT_ID"
MIN_CONNECTIONS=1  # alert if zero connections (application lost connectivity)

CONN_COUNT=$(ss -tn state established "( dport = :2003 )" | grep -c "2003")

if [ "$CONN_COUNT" -ge "$MIN_CONNECTIONS" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

This is particularly useful in production where you expect a constant pool of connections from your application. Zero connections when the app should be running indicates a reconnect failure.

Step 7: Keyspace Size Alert

Runaway data accumulation (a bug writing duplicate keys, a cache that never expires) can exhaust memory silently. Monitor the total key count via BlueQL if your Skytable version supports it via the REST API:

#!/bin/bash
# /usr/local/bin/skytable-keyspace-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_KEYSPACE_HEARTBEAT_ID"
MAX_KEYS=10000000  # alert if total keys exceed 10M

# Query keyspace count via REST API (adapt to your Skytable version's API)
RESPONSE=$(curl -s "http://127.0.0.1:2004/v1/keyspaces")
KEY_COUNT=$(echo "$RESPONSE" | python3 -c "import sys,json; d=json.load(sys.stdin); print(sum(t.get('entries',0) for ks in d.get('keyspaces',[]) for t in ks.get('tables',[])))" 2>/dev/null)

if [ -n "$KEY_COUNT" ] && [ "$KEY_COUNT" -lt "$MAX_KEYS" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

Adapt the key count extraction to match your Skytable REST API response shape.

Step 8: Data Integrity Check

Schedule a periodic checksum or sanity-check against Skytable's persistence files to detect storage corruption:

#!/bin/bash
# /usr/local/bin/skytable-integrity-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_INTEGRITY_HEARTBEAT_ID"
SKYTABLE_DATA_DIR="/var/lib/skytable"

# Check for checksum errors in persistence files using Skytable's built-in verify
# (Adapt this to whatever integrity command your Skytable version provides)
if skyd --check-data "$SKYTABLE_DATA_DIR" 2>&1 | grep -qi "error\|corrupt\|checksum"; then
  echo "DATA INTEGRITY ERROR detected in Skytable persistence files"
  # Do not ping heartbeat → Vigilmon alerts
else
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

Run this weekly or after each persistence flush. The alert window for this heartbeat should match your check cadence plus a buffer.

Step 9: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and connect Slack, PagerDuty, email, or a webhook.
  2. For the TCP monitor (port 2003): 2 consecutive failures before alerting.
  3. For the REST API monitor (port 2004): 2 consecutive failures.
  4. For memory and connection heartbeats: 1 missed ping — these are polled frequently.
  5. For persistence and integrity heartbeats: 1 missed ping — any persistence failure is immediately significant.
  6. Route memory alerts to your engineering on-call channel and integrity alerts to both on-call and your data team.

Summary

| Monitor | Type | Interval | Alert Threshold | |---|---|---|---| | TCP port 2003 (Skyhash) | TCP | 30s | 2 failures | | HTTP port 2004 (REST API) | HTTP | 1m | 2 failures | | Query throughput | Cron heartbeat | 5m | 1 missed ping | | Memory usage | Cron heartbeat | 2m | 1 missed ping | | Disk persistence health | Cron heartbeat | 10m | 1 missed ping | | Connection count | Cron heartbeat | 2m | 1 missed ping | | Keyspace size | Cron heartbeat | 15m | 1 missed ping | | Data integrity | Cron heartbeat | 1h | 1 missed ping |

Skytable's Rust foundation gives it strong memory safety guarantees, but no database is immune to operational failure. Vigilmon wraps both the Skyhash client port and the management REST API in uptime checks, while the cron heartbeat pattern covers persistence and integrity checks that have no HTTP endpoint equivalent. The result is full observability across both the data path and the operational health of your Skytable deployment.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles