How to Monitor Riak Uptime and Cluster Health with Vigilmon

Riak is a distributed key-value database built for high availability — it can survive node failures, tolerate network partitions, and serve reads and writes even during cluster reconfigurations. But "available by design" does not mean you can skip monitoring. Riak clusters fail in subtle ways: a ring with too many pending claims, overloaded vnodes that drop requests silently, or a split-brain where two ring partitions both accept writes but cannot reconcile them. When Riak degrades, your application may still receive responses — just wrong ones.

Vigilmon gives you external visibility into Riak availability through HTTP probe monitoring and heartbeat monitoring for Riak-dependent background processes. This tutorial walks through both.

Why Riak Monitoring Matters

Riak's built-in stats endpoint (/stats) provides internal node metrics. It cannot tell you:

• Whether Riak is reachable from your application servers across the network
• Whether read/write quorum is being met (R and W settings vs. available replicas)
• Whether your ring is balanced — unbalanced rings cause hot vnodes and degraded performance
• Whether handoff operations are consuming cluster resources and causing write timeouts
• Whether your Riak-dependent background workers have silently stopped processing

These failures look healthy from an internal stats perspective while causing real user-facing data errors. External monitoring through Vigilmon tests the actual path your application uses.

Step 1: Build a Riak Health Endpoint

Riak's built-in /ping HTTP endpoint returns OK when the node is alive. You can monitor this directly in Vigilmon, but a more thorough check validates that reads and writes are actually working. Create a thin wrapper that performs a test put/get cycle.

Node.js Example

// riak-health.js
const express = require('express');
const Riak = require('basho-riak-client');

const app = express();

const nodes = [
  new Riak.Node({ remoteAddress: process.env.RIAK_HOST || '127.0.0.1', remotePort: 8087 }),
];
const cluster = new Riak.Cluster({ nodes });
const client = new Riak.Client(cluster);

app.get('/health/riak', (req, res) => {
  const ts = Date.now().toString();
  const key = `healthcheck-${ts}`;

  client.storeValue({
    bucket: '_healthcheck',
    key,
    value: new Riak.Commands.KV.RiakObject().setValue(ts),
  }, (err) => {
    if (err) {
      return res.status(503).json({ status: 'down', phase: 'write', error: err.message });
    }

    client.fetchValue({ bucket: '_healthcheck', key }, (err2, result) => {
      if (err2 || !result || result.isNotFound) {
        return res.status(503).json({ status: 'degraded', phase: 'read', error: err2?.message });
      }

      return res.status(200).json({ status: 'ok', latency_ms: Date.now() - parseInt(ts) });
    });
  });
});

app.get('/health/riak/ping', async (req, res) => {
  try {
    const response = await fetch(`http://${process.env.RIAK_HOST}:8098/ping`);
    const text = await response.text();
    if (text.trim() === 'OK') {
      return res.status(200).json({ status: 'ok' });
    }
    return res.status(503).json({ status: 'degraded', response: text });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Python Example

# riak_health.py
import os, time
import riak
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
client = riak.RiakClient(host=os.environ.get('RIAK_HOST', '127.0.0.1'), pb_port=8087)
bucket = client.bucket('_healthcheck')

@app.get('/health/riak')
def riak_health():
    try:
        ts = int(time.time() * 1000)
        key = f'healthcheck-{ts}'

        obj = bucket.new(key, data=str(ts))
        obj.store()

        fetched = bucket.get(key)
        if not fetched.exists:
            return JSONResponse(status_code=503, content={
                'status': 'degraded',
                'reason': 'read_after_write_failed',
            })

        return {'status': 'ok', 'latency_ms': int(time.time() * 1000) - ts}
    except Exception as e:
        return JSONResponse(status_code=503, content={
            'status': 'down',
            'error': str(e),
        })

Verify your endpoint manually before adding it to Vigilmon:

curl -i https://your-app.example.com/health/riak
# HTTP/1.1 200 OK
# {"status":"ok","latency_ms":15}

Step 2: Monitor Riak's Built-In Ping Endpoint

Riak exposes a /ping endpoint on its HTTP interface (default port 8098). You can point Vigilmon directly at this if your Riak node is publicly accessible or reachable from Vigilmon probes:

• URL: http://your-riak-host:8098/ping
• Expected body contains: OK
• Status code: 200

This is the fastest path to getting a monitor running. It only validates that the Riak HTTP interface is responding — it does not test quorum reads/writes. Combine it with the application-level health endpoint from Step 1 for full coverage.

Step 3: Configure Vigilmon HTTP Monitors

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your Riak health endpoint: https://your-app.example.com/health/riak
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 2000ms
6. Under Alert channels, assign your Slack or email channel
7. Save the monitor

Repeat for each Riak node if you run a multi-node cluster — use a naming convention like [riak-node-1] /health, [riak-node-2] /health, etc. Group them into a single Status Page for a unified cluster view.

What This Catches

| Failure | Riak Internal Stats | Vigilmon | |---|---|---| | Node process crash | ✓ | ✓ | | Network partition from app servers | ✗ | ✓ | | Write quorum not met | ✗ | ✓ | | Overloaded vnodes dropping requests | ✗ | ✓ | | HTTP interface misconfiguration | ✗ | ✓ |

Step 4: Heartbeat Monitoring for Riak-Dependent Workers

Background workers that use Riak as a job store or event log will stop silently if the cluster becomes unreachable. The worker blocks on a get/put and never completes another unit of work.

Vigilmon heartbeat monitors catch this: your worker sends a ping to Vigilmon after each successful processing cycle. If the ping stops, Vigilmon opens an incident.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Name it: riak-background-worker
3. Set the expected interval: 5 minutes
4. Set the grace period: 10 minutes
5. Save — copy the heartbeat URL

Wire Into Your Worker

import os, requests, time
import riak

client = riak.RiakClient(host=os.environ.get('RIAK_HOST', '127.0.0.1'), pb_port=8087)
queue_bucket = client.bucket('job_queue')

def process_jobs():
    while True:
        jobs = queue_bucket.get_index('status_bin', 'pending')
        for key in jobs.results:
            job_obj = queue_bucket.get(key)
            if job_obj.exists:
                process(job_obj.data)
                job_obj.data['status'] = 'done'
                job_obj.store()

        # Heartbeat: ping Vigilmon after each processing cycle
        try:
            requests.get(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)
        except Exception:
            pass

        time.sleep(60)

Step 5: Key Metrics and Alert Routing

Riak clusters can degrade before they fail completely. Route alerts by severity:

1. Riak health endpoint monitor → Slack + PagerDuty (P1 — reads/writes failing)
2. Riak ping monitor (per node) → Slack only (P2 — single node down, cluster degraded but alive)
3. Worker heartbeat monitor → Slack + email (P2 — background jobs stalled)
4. Response time alert at 1500ms → Slack only (early warning for vnode pressure)

For a 5-node cluster with n_val=3, you can tolerate two node failures while still meeting quorum. Monitor each node individually — when two nodes show degraded, escalate immediately because the third failure will take down write quorum.

Summary

Riak's distributed design is resilient, but silent degradation is its biggest operational risk. Vigilmon gives you external visibility before users notice data inconsistencies or timeouts:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/riak | Read/write quorum, node reachability | | HTTP monitor on /ping (per node) | Individual node liveness | | Heartbeat monitor | Background worker liveness |

Get started free at vigilmon.online — your first Riak monitor is running in under two minutes.