tutorial

Monitoring Aerospike with Vigilmon: Real-Time NoSQL Health, Cluster Availability & Application-Layer Uptime Checks

Practical guide to uptime and health monitoring for Aerospike — HTTP health endpoints for Aerospike-connected services, namespace and node availability probing, migration detection, and independent external monitoring with Vigilmon.

Aerospike is built for sub-millisecond reads and writes at massive scale, but that speed only delivers value when every node in your cluster is healthy. Namespace storage approaching capacity, cluster migrations in progress after a node failure, client connection pool exhaustion, and replication factor violations can all degrade your application silently while Aerospike's internal metrics remain within bounds. Vigilmon adds an independent external layer: HTTP health checks on your Aerospike-connected services that probe real cluster connectivity, surface namespace pressure, and alert on application-level failures before users notice.

This tutorial shows you how to build meaningful monitoring for Aerospike-backed applications using Vigilmon.

What You'll Build

  • A health endpoint that probes real Aerospike node connectivity and namespace health
  • Namespace storage and eviction pressure detection
  • A Vigilmon HTTP monitor with appropriate timeout and assertion settings
  • A heartbeat monitor for Aerospike-backed background workers
  • A cluster-aware alerting strategy

Prerequisites

  • A running Aerospike cluster (single-node dev or multi-node production)
  • An application connected to Aerospike (any runtime)
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Aerospike Service

Add a /health endpoint that performs a lightweight Aerospike operation to verify real connectivity and check namespace storage pressure.

Node.js (aerospike — official client)

// routes/health.js
import Aerospike from 'aerospike';

const client = Aerospike.client({
  hosts: process.env.AEROSPIKE_HOSTS || 'localhost:3000',
  log: { level: Aerospike.log.OFF },
  policies: {
    timeout: 5000,
  },
});

await client.connect();

export async function healthHandler(req, res) {
  const checks = {};
  let ok = true;
  const start = Date.now();

  // Check 1: Basic connectivity — write and read a sentinel key
  try {
    const key = new Aerospike.Key('test', 'health', 'vigilmon-probe');
    await client.put(key, { ts: Date.now() });
    const record = await client.get(key);
    checks.connectivity = 'ok';
    checks.connectivityLatencyMs = Date.now() - start;
    checks.probeValue = record.bins.ts;
  } catch (err) {
    checks.connectivity = `error: ${err.message}`;
    ok = false;
  }

  // Check 2: Cluster info — node count and migrations
  try {
    const info = await client.infoAll('statistics');
    const nodes = Object.keys(info);
    checks.nodeCount = nodes.length;

    let migrationsInProgress = false;
    for (const node of nodes) {
      const stats = info[node];
      const migrating = stats.info?.match(/migrate_progress_send=(\d+)/)?.[1];
      if (migrating && parseInt(migrating) > 0) {
        migrationsInProgress = true;
      }
    }

    checks.migrationsInProgress = migrationsInProgress;
    if (migrationsInProgress) {
      checks.clusterStatus = 'migrating — cluster rebalancing';
      // Degraded but not down during expected rebalancing
    } else {
      checks.clusterStatus = 'stable';
    }
  } catch (err) {
    checks.clusterStatus = `error: ${err.message}`;
    ok = false;
  }

  // Check 3: Namespace storage pressure
  const namespace = process.env.AEROSPIKE_NAMESPACE || 'test';
  try {
    const nsInfo = await client.infoAll(`namespace/${namespace}`);
    const firstNode = Object.values(nsInfo)[0]?.info || '';
    const usedBytesMatch = firstNode.match(/device_used_bytes=(\d+)/);
    const totalBytesMatch = firstNode.match(/device_total_bytes=(\d+)/);
    const stopWritesMatch = firstNode.match(/stop_writes=(\w+)/);

    if (usedBytesMatch && totalBytesMatch) {
      const used = parseInt(usedBytesMatch[1]);
      const total = parseInt(totalBytesMatch[1]);
      const pct = total > 0 ? Math.round((used / total) * 100) : 0;
      checks.namespaceStoragePct = pct;

      const maxStoragePct = parseInt(process.env.MAX_STORAGE_PCT || '80', 10);
      if (pct > maxStoragePct) {
        checks.namespaceHealth = `storage high: ${pct}%`;
        ok = false;
      } else {
        checks.namespaceHealth = 'ok';
      }
    }

    if (stopWritesMatch?.[1] === 'true') {
      checks.namespaceHealth = 'stop-writes active — namespace full';
      ok = false;
    }
  } catch (err) {
    checks.namespaceHealth = `error: ${err.message}`;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    service: 'aerospike-service',
    checks,
  });
}

Python (aerospike — official client)

# routers/health.py
import os, time, aerospike
from fastapi import APIRouter
from fastapi.responses import JSONResponse

router = APIRouter()

config = {
    'hosts': [(h.split(':')[0], int(h.split(':')[1]) if ':' in h else 3000)
              for h in os.environ.get('AEROSPIKE_HOSTS', 'localhost:3000').split(',')],
    'timeout': 5000,
}

aero_client = aerospike.client(config).connect()

@router.get("/health")
async def health():
    checks = {}
    ok = True
    start = time.monotonic()

    # Check 1: Write/read probe
    try:
        key = ('test', 'health', 'vigilmon-probe')
        aero_client.put(key, {'ts': int(time.time() * 1000)})
        _, _, record = aero_client.get(key)
        checks['connectivity'] = 'ok'
        checks['connectivityLatencyMs'] = int((time.monotonic() - start) * 1000)
    except Exception as e:
        checks['connectivity'] = f'error: {e}'
        ok = False

    # Check 2: Cluster node count via info
    try:
        request = 'statistics'
        policy = {'timeout': 5000}
        info_map = aero_client.info_all(request, policy)
        checks['nodeCount'] = len(info_map)

        migrating = any(
            'migrate_progress_send' in str(v) and
            'migrate_progress_send=0' not in str(v)
            for v in info_map.values()
        )
        checks['migrationsInProgress'] = migrating
        checks['clusterStatus'] = 'migrating' if migrating else 'stable'
    except Exception as e:
        checks['clusterStatus'] = f'error: {e}'
        ok = False

    # Check 3: Namespace storage
    namespace = os.environ.get('AEROSPIKE_NAMESPACE', 'test')
    try:
        ns_info = aero_client.info_all(f'namespace/{namespace}', {'timeout': 5000})
        first_val = list(ns_info.values())[0] if ns_info else ''
        info_str = str(first_val)

        import re
        used_match = re.search(r'device_used_bytes=(\d+)', info_str)
        total_match = re.search(r'device_total_bytes=(\d+)', info_str)
        stop_writes = 'stop_writes=true' in info_str

        if used_match and total_match:
            used = int(used_match.group(1))
            total = int(total_match.group(1))
            pct = round(used / total * 100, 1) if total > 0 else 0
            checks['namespaceStoragePct'] = pct
            max_pct = float(os.environ.get('MAX_STORAGE_PCT', '80'))
            if pct > max_pct:
                checks['namespaceHealth'] = f'storage high: {pct}%'
                ok = False
            else:
                checks['namespaceHealth'] = 'ok'

        if stop_writes:
            checks['namespaceHealth'] = 'stop-writes active — namespace full'
            ok = False
    except Exception as e:
        checks['namespaceHealth'] = f'error: {e}'
        ok = False

    status_code = 200 if ok else 503
    return JSONResponse(status_code=status_code, content={
        'status': 'ok' if ok else 'degraded',
        'service': 'aerospike-service',
        'checks': checks,
    })

Go (aerospike-client-go)

// internal/health/handler.go
package health

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "os"
    "time"

    as "github.com/aerospike/aerospike-client-go/v7"
)

func Handler(client *as.Client) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        checks := map[string]any{}
        ok := true

        // Check 1: Write/read probe
        start := time.Now()
        key, _ := as.NewKey("test", "health", "vigilmon-probe")
        bins := as.BinMap{"ts": time.Now().UnixMilli()}
        writePolicy := as.NewWritePolicy(0, 0)
        writePolicy.Timeout = 5 * time.Second
        if err := client.Put(writePolicy, key, bins); err != nil {
            checks["connectivity"] = "error: " + err.Error()
            ok = false
        } else {
            readPolicy := as.NewPolicy()
            readPolicy.Timeout = 5 * time.Second
            if _, err := client.Get(readPolicy, key); err != nil {
                checks["connectivity"] = "error: " + err.Error()
                ok = false
            } else {
                checks["connectivity"] = "ok"
                checks["connectivityLatencyMs"] = time.Since(start).Milliseconds()
            }
        }

        // Check 2: Cluster node count
        nodes := client.GetNodes()
        checks["nodeCount"] = len(nodes)
        if len(nodes) == 0 {
            checks["clusterStatus"] = "no nodes connected"
            ok = false
        } else {
            checks["clusterStatus"] = fmt.Sprintf("%d nodes connected", len(nodes))
        }

        // Check 3: Namespace storage via info
        namespace := os.Getenv("AEROSPIKE_NAMESPACE")
        if namespace == "" {
            namespace = "test"
        }
        if len(nodes) > 0 {
            infoMap, err := as.RequestNodeInfo(nodes[0], "namespace/"+namespace)
            if err == nil {
                infoStr := infoMap["namespace/"+namespace]
                checks["namespaceInfoAvailable"] = len(infoStr) > 0
                if len(infoStr) > 0 {
                    checks["namespaceHealth"] = "ok"
                    if contains(infoStr, "stop_writes=true") {
                        checks["namespaceHealth"] = "stop-writes active"
                        ok = false
                    }
                }
            }
        }

        w.Header().Set("Content-Type", "application/json")
        if !ok {
            w.WriteHeader(http.StatusServiceUnavailable)
        }
        _ = json.NewEncoder(w).Encode(map[string]any{
            "status":  map[bool]string{true: "ok", false: "degraded"}[ok],
            "service": "aerospike-service",
            "checks":  checks,
        })
    }
}

func contains(s, substr string) bool {
    return len(s) >= len(substr) && (s == substr || len(s) > 0 &&
        func() bool {
            for i := 0; i <= len(s)-len(substr); i++ {
                if s[i:i+len(substr)] == substr {
                    return true
                }
            }
            return false
        }())
}

Deploy and verify the endpoint manually before wiring it into Vigilmon:

curl -i https://your-app.example.com/health
# HTTP/1.1 200 OK
# {"status":"ok","service":"aerospike-service","checks":{"connectivity":"ok","nodeCount":3,"clusterStatus":"stable","namespaceHealth":"ok"}}

Step 2: Configure the Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: your service's health endpoint (e.g. https://api.example.com/health).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. JSON assertion: path status, expected value ok.
  7. Save the monitor.

Vigilmon probes from multiple geographic regions simultaneously. A single-region blip will not page you; multi-region consensus is required before opening an incident. This is particularly valuable for Aerospike because individual node failures may not impact all clients equally depending on your rack-awareness configuration.

Tip: Create a second monitor pointing to a /health/detailed endpoint that returns node count and namespace storage percentage. Use a JSON assertion on checks.namespaceStoragePct to alert before the namespace reaches stop-writes capacity.

Step 3: Heartbeat Monitoring for Aerospike-Backed Workers

Aerospike is commonly used as a session store, rate-limiter backend, or high-throughput event counter. Workers that depend on Aerospike will fail silently when the cluster is unavailable or when namespace writes are stopped. Vigilmon's heartbeat monitors detect this: your worker pings Vigilmon after each successful cycle, and if pings stop arriving, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat.
  2. Set the name: aerospike-worker.
  3. Set the expected interval: 2 minutes (adjust to your job frequency).
  4. Set the grace period: 5 minutes.
  5. Save — copy the unique heartbeat URL.

Wire It Into Your Worker

Node.js:

import Aerospike from 'aerospike';
import fetch from 'node-fetch';

async function runWorkerCycle() {
  const key = new Aerospike.Key('events', 'queue', `job-${Date.now()}`);
  await client.put(key, { processed: true, ts: Date.now() });

  // Ping Vigilmon only after successful Aerospike write
  await fetch(process.env.VIGILMON_HEARTBEAT_URL).catch(() => {});
}

setInterval(runWorkerCycle, 60_000);

Python:

import os, time, requests, aerospike

def run_worker_cycle():
    key = ('events', 'queue', f'job-{int(time.time() * 1000)}')
    client.put(key, {'processed': True, 'ts': int(time.time())})

    # Only ping Vigilmon after successful Aerospike operation
    requests.get(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)

while True:
    try:
        run_worker_cycle()
    except Exception as e:
        print(f"Worker cycle failed — heartbeat NOT sent: {e}")
    time.sleep(60)

Step 4: Cluster Migration Alerting

Aerospike performs data migrations when a node joins or leaves the cluster. Migrations are expected during maintenance but can last minutes to hours depending on data volume. During migrations, replication factor guarantees may be temporarily reduced.

Surface migration state in your health endpoint (as shown in Step 1) and create a separate Vigilmon monitor with an assertion on checks.migrationsInProgress:

  • false: cluster stable — green
  • true: migrations in progress — Slack notification (P2, not a page)

This gives your team visibility into cluster rebalancing without triggering an on-call page for planned maintenance.

Step 5: Alerting Strategy

| Alert channel | When to use | |---|---| | PagerDuty | Connectivity check fails (cluster unreachable) | | Slack webhook | Migrations detected (cluster rebalancing) | | Slack + Email | Namespace storage > 80% (approaching stop-writes) | | PagerDuty | stop_writes=true (writes rejected — active outage) | | PagerDuty | Heartbeat missed (worker stopped processing) | | Status page | Any user-facing service backed by Aerospike |

Set response time thresholds as an early warning:

  • Alert at 10ms for direct Aerospike operation latency (sub-millisecond is normal; 10ms+ signals cluster pressure)
  • Alert at 500ms for health endpoint response time

What Vigilmon Catches That Internal Monitoring Misses

| Scenario | Internal check | Vigilmon | |---|---|---| | Namespace stop-writes active | Aerospike logs only | /health endpoint exposes flag | | Node loss reduces replication factor | Cluster-internal event | Migration state surfaced in health | | Client connection pool exhausted | No client-side metric | Health endpoint probe fails — 503 fires | | Worker silently stopped writing | Process appears running | Heartbeat grace period expires → alert | | Aerospike reachable from cluster but not app servers | Node-level check only | External HTTP probe from app's perspective | | Monitoring pipeline (CloudWatch/Prometheus) fails | Alerts silently lost | Vigilmon is external — unaffected |

Aerospike's speed advantage disappears when your application cannot reach the cluster or when namespace writes are stopped. External health checks with Vigilmon give you the independent signal you need — before latency spikes reach your users.

Start monitoring your Aerospike applications in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles