tutorial

How to Monitor AWS Neptune Uptime and Health with Vigilmon

AWS Neptune failovers, loader job failures, and Gremlin endpoint degradation can impact your graph database silently. Learn how to monitor Neptune uptime externally with Vigilmon HTTP probes, heartbeat monitors for bulk loader jobs, and alert routing for cluster health.

AWS Neptune is a fully managed graph database — but "managed" does not mean "automatically monitored from your application's perspective." A Neptune cluster failover takes 30–60 seconds; during that window your application's connection pool sees timeout errors that CloudWatch does not surface until after the fact. A bulk loader job that ingests your knowledge graph can fail silently with a CloudWatch log entry nobody checks. A Gremlin traversal endpoint that returns HTTP 200 with an empty result set due to a schema mismatch looks healthy to an uptime checker, but returns wrong data.

Vigilmon gives you external visibility into AWS Neptune health through HTTP probe monitoring and heartbeat monitoring for Neptune Loader jobs and graph processing pipelines. This tutorial walks through both.

Why AWS Neptune Monitoring Needs More Than Process Checks

AWS CloudWatch, the Neptune console, and VPC health dashboards tell you the cluster endpoint is reachable. They cannot tell you:

  • Whether Neptune is reachable from your application servers across the VPC at the network layer
  • Whether a failover is in progress and your writer endpoint is temporarily unavailable
  • Whether the read replica your analytics tier queries has fallen behind the writer
  • Whether a Neptune Loader job has failed with a partial import
  • Whether a Gremlin or SPARQL query is timing out due to graph traversal depth limits
  • Whether your application's connection pool to Neptune is exhausted under concurrent traversals

These are the failure modes that produce silent data errors or cascading timeouts without clean alerts. External monitoring through Vigilmon catches them by probing the actual connectivity and logic paths your application relies on.

Step 1: Build a Neptune Health Endpoint

AWS Neptune does not expose an HTTP health endpoint your application can call from outside the VPC without a proxy layer. Deploy a thin Lambda function or sidecar service inside your VPC that runs the health check and exposes it via API Gateway or an internal ALB.

Node.js / Lambda Example (exposed via API Gateway)

// health/neptune.js — deployed as a Lambda behind API Gateway
const { DriverRemoteConnection, Graph } = require('gremlin');

let connection;
function getClient() {
  if (!connection) {
    const endpoint = process.env.NEPTUNE_ENDPOINT; // e.g. wss://xxx.neptune.amazonaws.com:8182/gremlin
    connection = new DriverRemoteConnection(endpoint, {
      mimeType: 'application/vnd.gremlin-v2.0+json',
      pingEnabled: false,
    });
  }
  return connection;
}

exports.handler = async (event) => {
  try {
    const conn = getClient();
    const graph = new Graph();
    const g = graph.traversal().withRemote(conn);

    // 1. Verify graph traversal works (count vertices — lightweight probe)
    const count = await g.V().limit(1).count().next();
    if (count.value === undefined) {
      return { statusCode: 503, body: JSON.stringify({ status: 'degraded', reason: 'traversal_failed' }) };
    }

    // 2. Check cluster status via Neptune management API (requires IAM)
    let readerLag = null;
    try {
      const endpoint = process.env.NEPTUNE_CLUSTER_ENDPOINT;
      const statusResp = await fetch(
        `https://${endpoint}/status`,
        { signal: AbortSignal.timeout(3000) }
      );
      if (statusResp.ok) {
        const status = await statusResp.json();
        // Neptune /status returns {"status":"healthy"} when all is well
        if (status.status !== 'healthy') {
          return {
            statusCode: 503,
            body: JSON.stringify({ status: 'degraded', reason: 'cluster_unhealthy', cluster_status: status.status }),
          };
        }
      }
    } catch {
      // /status endpoint may not be reachable from within Lambda — non-fatal
    }

    return {
      statusCode: 200,
      body: JSON.stringify({ status: 'ok', vertex_probe: count.value }),
    };
  } catch (err) {
    return { statusCode: 503, body: JSON.stringify({ status: 'down', error: err.message }) };
  }
};

Python (FastAPI / sidecar in ECS) Example

# health_neptune.py — run as a sidecar container in ECS with VPC access
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from gremlin_python.driver import client as gremlin_client
from gremlin_python.driver.serializer import GraphSONSerializersV2d0
import httpx
import os

app = FastAPI()

NEPTUNE_WS = os.environ["NEPTUNE_ENDPOINT"]    # wss://xxx.neptune.amazonaws.com:8182/gremlin
NEPTUNE_HTTP = os.environ["NEPTUNE_HTTP_ENDPOINT"]  # https://xxx.neptune.amazonaws.com:8182

_client = None

def get_gremlin_client():
    global _client
    if _client is None:
        _client = gremlin_client.Client(
            NEPTUNE_WS,
            "g",
            message_serializer=GraphSONSerializersV2d0(),
        )
    return _client

@app.get("/health/neptune")
async def neptune_health():
    try:
        # 1. Gremlin traversal probe
        gc = get_gremlin_client()
        result = gc.submit("g.V().limit(1).count()").all().result()
        if not isinstance(result, list):
            return JSONResponse(status_code=503, content={"status": "degraded", "reason": "traversal_failed"})

        # 2. Neptune cluster /status check
        async with httpx.AsyncClient(timeout=5) as http:
            status_resp = await http.get(f"{NEPTUNE_HTTP}/status")
            if status_resp.status_code == 200:
                status_data = status_resp.json()
                if status_data.get("status") != "healthy":
                    return JSONResponse(status_code=503, content={
                        "status": "degraded",
                        "reason": "cluster_unhealthy",
                        "cluster_status": status_data.get("status"),
                    })

        return {"status": "ok", "vertex_probe": result[0] if result else 0}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Verify the endpoint before wiring up Vigilmon:

curl -i https://your-api.example.com/health/neptune
# HTTP/1.1 200 OK
# {"status":"ok","vertex_probe":1}

Step 2: Configure Vigilmon HTTP Monitor for AWS Neptune

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-api.example.com/health/neptune
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously, requiring multi-region consensus before opening an incident. This eliminates false alarms from transient single-probe network hiccups.

Monitoring Reader and Writer Endpoints Separately

Neptune exposes a writer endpoint and one or more reader endpoints. Create separate monitors:

  • [neptune-writer] /health/neptune?role=writer — immediate P1 page on failure; writes stop
  • [neptune-reader-1] /health/neptune?role=reader — P2 Slack alert; analytics degraded
  • [neptune-reader-2] /health/neptune?role=reader — P2 Slack alert

Pass the role query parameter to your health handler to direct the probe at the appropriate Neptune endpoint.

Use Vigilmon's status page grouping to surface all Neptune monitors in a single pane for on-call engineers.

Step 3: Heartbeat Monitoring for Neptune Loader Jobs and Graph Pipelines

Neptune Bulk Loader jobs import large graph datasets from S3 into Neptune. These jobs are asynchronous — you submit the request and poll for completion. When a loader job fails due to malformed data, an IAM permission error, or a timeout, it logs the failure but does not notify your team. Graph processing pipelines that periodically refresh materialized traversal caches or re-compute recommendations also run silently.

Vigilmon heartbeat monitors detect silent stalls: your pipeline pings Vigilmon after each successful completion. If pings stop arriving within the expected window, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: neptune-loader-pipeline
  3. Set the expected interval: 6 hours (adjust to your loader schedule)
  4. Set the grace period: 1 hour
  5. Save — copy the unique heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a Neptune Loader Poller

// neptune-loader.js
const axios = require('axios');

const NEPTUNE_HTTP = process.env.NEPTUNE_HTTP_ENDPOINT; // https://xxx.neptune.amazonaws.com:8182

async function submitAndWaitForLoader(s3Uri, iamRoleArn) {
  // Submit the loader job
  const submitResp = await axios.post(`${NEPTUNE_HTTP}/loader`, {
    source: s3Uri,
    format: 'csv',
    iamRoleArn,
    region: process.env.AWS_REGION,
    failOnError: 'TRUE',
  });
  const loadId = submitResp.data.payload.loadId;
  console.log(`Loader job submitted: ${loadId}`);

  // Poll for completion
  let status = 'LOAD_IN_PROGRESS';
  while (status === 'LOAD_IN_PROGRESS' || status === 'LOAD_NOT_STARTED') {
    await new Promise(r => setTimeout(r, 30_000));
    const statusResp = await axios.get(`${NEPTUNE_HTTP}/loader/${loadId}`);
    status = statusResp.data.payload.overallStatus?.status;
    console.log(`Loader status: ${status}`);
  }

  if (status !== 'LOAD_COMPLETED') {
    throw new Error(`Loader job failed with status: ${status}`);
  }

  // Ping Vigilmon on successful completion
  await axios.get(process.env.VIGILMON_HEARTBEAT_URL, { timeout: 5000 }).catch(() => {});
  console.log('Loader job completed successfully');
}

submitAndWaitForLoader(process.env.S3_DATA_URI, process.env.NEPTUNE_IAM_ROLE_ARN)
  .catch(err => {
    console.error('Loader job failed:', err.message);
    process.exit(1); // Heartbeat stops — Vigilmon alerts within grace period
  });

For graph analytics pipelines that recompute traversal caches:

# graph_analytics.py
import requests
import os
from gremlin_python.driver import client as gremlin_client
from gremlin_python.driver.serializer import GraphSONSerializersV2d0

def recompute_recommendations():
    gc = gremlin_client.Client(
        os.environ["NEPTUNE_ENDPOINT"],
        "g",
        message_serializer=GraphSONSerializersV2d0(),
    )
    try:
        # Example: find top-connected vertices for recommendation engine
        result = gc.submit(
            "g.V().hasLabel('Product').order().by(__.in('PURCHASED').count(), desc).limit(100).id()"
        ).all().result()
        print(f"Recomputed recommendations for {len(result)} products")

        # Ping Vigilmon on successful completion
        requests.get(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)
    finally:
        gc.close()

if __name__ == "__main__":
    recompute_recommendations()

Step 4: Failover Detection and Alert Routing

Neptune automatic failovers promote a read replica to writer in under 60 seconds — but during that window your application sees connection timeouts. Configure your application to detect failover quickly by connecting to the cluster endpoint (which automatically resolves to the current writer) and setting short connection timeouts:

// Expose failover status in your health endpoint
// Neptune's /status endpoint returns {"status":"healthy"} during normal operation
// and {"status":"available"} immediately after a failover completes
const statusResp = await fetch(`${NEPTUNE_HTTP}/status`, { signal: AbortSignal.timeout(2000) });
const { status } = await statusResp.json();
if (status !== 'healthy') {
  return { statusCode: 503, body: JSON.stringify({
    status: 'degraded',
    reason: 'post_failover',
    cluster_status: status,
  })};
}

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Neptune writer /health/neptune | Slack + PagerDuty | P1 | | Neptune reader endpoints | Slack | P2 | | Heartbeat: bulk loader job | Slack + email | P2 | | Heartbeat: analytics pipeline | Email | P3 |

Set response time thresholds as early warnings:

  • Alert at 2000ms for the health endpoint (a simple vertex count traversal should be fast)
  • Alert at 10000ms for Gremlin traversal endpoints in production (signals deep traversal or missing index)

Summary

AWS Neptune failures surface in subtle ways — silent failovers, failed loader jobs, reader replica lag, traversal timeouts — long before your users see errors. Vigilmon gives you external visibility across the full failure surface:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/neptune | Gremlin traversal, cluster status, failover detection | | HTTP monitor per endpoint (writer/reader) | Role-specific availability during failovers | | Heartbeat monitor | Loader job completion, analytics pipeline liveness |

Get started free at vigilmon.online — your first Neptune monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles