How to Monitor JanusGraph with Vigilmon

JanusGraph is a distributed graph database built on Apache TinkerPop, designed to store and query graphs containing hundreds of billions of vertices and edges. But its distributed architecture introduces a class of failure modes that process monitors cannot detect: the JVM process stays alive while the storage backend (Cassandra, HBase, or BerkeleyDB) has disconnected, Gremlin traversals time out under partition pressure, Elasticsearch index writes fall behind vertex mutations, or a schema management lock left by a dead JVM blocks all schema operations indefinitely. Because JanusGraph does not expose a native HTTP health endpoint — health is only observable through Gremlin traversal results or a custom sidecar — these failures often go undetected until application queries start returning errors or hanging.

Vigilmon gives you external visibility into JanusGraph's Gremlin Server, storage backend connectivity, and graph analytics jobs through HTTP probe monitoring and heartbeat monitors. This tutorial covers all three layers.

Why JanusGraph Monitoring Needs More Than Process Checks

A running janusgraph.sh process or a healthy Gremlin Server JVM tells you almost nothing about whether the graph database is actually functional. The failure modes that cause real outages are deeper:

• Storage backend disconnect: Cassandra or HBase becomes unavailable; JanusGraph accepts connections but throws PermanentBackendException on every read and write
• Transaction timeout: Long-running traversals exceed the transaction timeout (default 60 seconds), causing silent rollbacks that look like successful queries to the caller
• Index inconsistency: Elasticsearch or Solr index writes lag behind vertex/edge mutations; queries return incomplete results with no error signal
• Schema management lock contention: A JVM crash during a schema change leaves a distributed lock held in the storage backend, blocking all subsequent schema operations until the lock TTL expires (up to 200 seconds by default)
• Gremlin Server thread pool exhaustion: All server worker threads are occupied by slow traversals; new connections queue and eventually time out
• HBase region server failures: With HBase as the storage backend, region server failures cause tablet unavailability for the vertex ranges they own — the JanusGraph process is healthy, but a fraction of the graph is unreadable
• BerkeleyDB lock table overflow: In development/single-node deployments, BerkeleyDB can exhaust its lock table under concurrent writes, causing transaction failures without a clear storage-layer error

Step 1: Build a JanusGraph Health Endpoint

JanusGraph exposes health through Gremlin traversals, not HTTP. You need a lightweight sidecar that connects to Gremlin Server on port 8182 (WebSocket), runs a probe traversal, checks storage backend connectivity via the JanusGraph management API, and exposes the result as an HTTP endpoint that Vigilmon can probe.

Install the dependencies:

pip install gremlinpython fastapi uvicorn

Health Sidecar

# janusgraph_health.py
import os
import asyncio
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from gremlin_python.driver import client, serializer
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal

app = FastAPI()

GREMLIN_HOST = os.environ.get("JANUSGRAPH_HOST", "localhost")
GREMLIN_PORT = int(os.environ.get("JANUSGRAPH_PORT", "8182"))
GREMLIN_URL = f"ws://{GREMLIN_HOST}:{GREMLIN_PORT}/gremlin"


def probe_gremlin() -> dict:
    """
    Opens a short-lived Gremlin connection, runs a lightweight traversal,
    and checks the JanusGraph management API for storage backend status.
    Returns a dict with status, backend_status, and vertex_count_sample.
    """
    try:
        gremlin_client = client.Client(
            GREMLIN_URL,
            "g",
            message_serializer=serializer.GraphSONSerializersV2d0(),
        )

        # Lightweight traversal: count vertices with a limit to avoid full scans
        result = gremlin_client.submit(
            "g.V().limit(1).count().next()"
        ).all().result()
        vertex_sample = result[0] if result else 0

        # Check storage backend connectivity via JanusGraph management API
        # mgmt.isOpen() returns false if the backend connection has dropped
        backend_result = gremlin_client.submit(
            "graph.openManagement().isOpen()"
        ).all().result()
        backend_open = backend_result[0] if backend_result else False

        # Check for active schema lock (held by dead JVM = schema ops blocked)
        lock_result = gremlin_client.submit(
            "graph.openManagement().getOpenInstances().size()"
        ).all().result()
        open_instances = lock_result[0] if lock_result else 0

        gremlin_client.close()

        if not backend_open:
            return {
                "status": "down",
                "reason": "storage_backend_disconnected",
                "vertex_sample": vertex_sample,
                "open_instances": open_instances,
            }

        return {
            "status": "ok",
            "backend_open": backend_open,
            "vertex_sample": vertex_sample,
            "open_instances": open_instances,
        }

    except Exception as exc:
        return {"status": "down", "error": str(exc)}


@app.get("/health/janusgraph")
def health():
    result = probe_gremlin()
    status_code = 200 if result.get("status") == "ok" else 503
    return JSONResponse(content=result, status_code=status_code)


@app.get("/health/janusgraph/schema")
def schema_health():
    """
    Dedicated endpoint for schema lock contention.
    Returns 503 if open_instances suggests a ghost JVM is holding a lock.
    In production, more than one open instance per JanusGraph node is suspicious.
    """
    try:
        gremlin_client = client.Client(
            GREMLIN_URL,
            "g",
            message_serializer=serializer.GraphSONSerializersV2d0(),
        )
        result = gremlin_client.submit(
            "graph.openManagement().getOpenInstances().toList()"
        ).all().result()
        gremlin_client.close()

        instance_count = len(result) if result else 0
        max_expected = int(os.environ.get("JANUSGRAPH_MAX_INSTANCES", "3"))

        if instance_count > max_expected:
            return JSONResponse(
                status_code=503,
                content={
                    "status": "degraded",
                    "reason": "possible_schema_lock_contention",
                    "open_instances": instance_count,
                    "max_expected": max_expected,
                },
            )

        return JSONResponse(
            status_code=200,
            content={"status": "ok", "open_instances": instance_count},
        )
    except Exception as exc:
        return JSONResponse(
            status_code=503,
            content={"status": "down", "error": str(exc)},
        )


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8090)

Run the sidecar alongside your JanusGraph instance:

JANUSGRAPH_HOST=127.0.0.1 JANUSGRAPH_PORT=8182 python janusgraph_health.py

Test that the endpoint responds correctly:

curl -i http://localhost:8090/health/janusgraph
# Expected: HTTP 200 {"status":"ok","backend_open":true,...}

curl -i http://localhost:8090/health/janusgraph/schema
# Expected: HTTP 200 {"status":"ok","open_instances":1}

Storage Backend Notes

The graph.openManagement().isOpen() call exercises the storage backend connection path. If Cassandra or HBase becomes unreachable after the initial JanusGraph startup, this call will block and eventually throw PermanentBackendException — which the sidecar catches and maps to a 503. For BerkeleyDB (single-node/embedded deployments), the same probe works but the failure mode is different: BerkeleyDB errors surface as com.sleepycat.je.LockConflictException in the exception message rather than a connection timeout.

For HBase backends, add the HBase REST API as a separate monitor target (see Step 2) to distinguish JanusGraph-layer failures from HBase region server failures.

Step 2: Configure Vigilmon HTTP Monitor for JanusGraph

Core Gremlin Server Monitor

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your health sidecar endpoint: https://your-host.example.com/health/janusgraph
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 8000ms (Gremlin traversals over Cassandra can be slower than simple HTTP checks; tighten this once you have baseline data)
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Schema Lock Monitor

Add a second monitor for schema lock contention:

• URL: https://your-host.example.com/health/janusgraph/schema
• Expected: 200, body contains "status":"ok"
• Interval: 5 minutes (schema locks self-resolve after TTL; frequent checks add unnecessary Gremlin load)
• Alert channel: Slack + email (schema locks require manual intervention — force-closing the ghost instance via mgmt.forceCloseInstance())

Monitoring Multiple Gremlin Server Instances

In production, JanusGraph is typically deployed as a cluster of Gremlin Server instances backed by a shared storage cluster. Create a separate Vigilmon monitor for each Gremlin Server node:

| Monitor Name | URL | Interval | |---|---|---| | JanusGraph node 1 | https://graph1.internal/health/janusgraph | 1 min | | JanusGraph node 2 | https://graph2.internal/health/janusgraph | 1 min | | JanusGraph node 3 | https://graph3.internal/health/janusgraph | 1 min | | Schema lock check | https://graph1.internal/health/janusgraph/schema | 5 min | | HBase REST API | http://hbase-master:8080/version/cluster | 1 min |

Monitoring each Gremlin Server instance individually lets you detect partial failures (a single node disconnected from the storage cluster) that would be invisible if you only monitored through a load balancer.

For Cassandra-backed deployments, also add monitors for the Cassandra native transport port via a simple TCP check, or expose a Cassandra health endpoint via Cassandra's nodetool status wrapped in a sidecar.

Step 3: Heartbeat Monitoring for JanusGraph Graph Processing Jobs

Many JanusGraph deployments include periodic batch jobs: Apache Spark graph analytics runs (via SparkGraphComputer), nightly graph traversal pipelines that compute derived properties or materialise graph projections, or scheduled reindexing jobs that synchronise JanusGraph's mixed indices with Elasticsearch or Solr.

These jobs run asynchronously and don't expose HTTP endpoints. When they fail silently — Spark executor OOM, Gremlin traversal timeout mid-pipeline, or index backend write failure — derived data goes stale without any alert.

Vigilmon heartbeat monitors detect these silent failures: the job sends a ping to Vigilmon after each successful completion. If pings stop, Vigilmon fires an alert after the grace period expires.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: janusgraph-graph-analytics-job
3. Set the expected interval: 6 hours (adjust to your job schedule)
4. Set the grace period: 30 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Python Heartbeat Example: Gremlin Traversal Pipeline

# graph_pipeline.py
import os
import time
import requests
from gremlin_python.driver import client, serializer

GREMLIN_URL = os.environ["GREMLIN_URL"]          # e.g. ws://localhost:8182/gremlin
HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]
BATCH_SIZE = int(os.environ.get("PIPELINE_BATCH_SIZE", "5000"))


def run_pipeline():
    """
    Example: traverse all Person vertices and compute a derived property.
    In production this would be a larger analytics computation.
    """
    gremlin_client = client.Client(
        GREMLIN_URL,
        "g",
        message_serializer=serializer.GraphSONSerializersV2d0(),
    )

    try:
        # Paginate to avoid transaction timeouts on large graphs
        offset = 0
        processed = 0

        while True:
            results = gremlin_client.submit(
                f"g.V().hasLabel('person').range({offset}, {offset + BATCH_SIZE}).toList()"
            ).all().result()

            if not results:
                break

            for vertex in results:
                # ... derive and write computed properties ...
                pass

            processed += len(results)
            offset += BATCH_SIZE

            # Ping Vigilmon after each successful batch to prove liveness
            requests.get(HEARTBEAT_URL, timeout=10)

        return processed

    finally:
        gremlin_client.close()


def run_spark_graph_job():
    """
    For Spark graph analytics via SparkGraphComputer.
    JanusGraph's SparkGraphComputer submits the Gremlin OLAP job to Spark.
    Heartbeat after the job completes.
    """
    import subprocess

    result = subprocess.run(
        ["spark-submit", "--class", "com.example.GraphJob", "graph-job.jar"],
        capture_output=True,
        timeout=3600,
    )

    if result.returncode != 0:
        raise RuntimeError(
            f"Spark graph job failed (exit {result.returncode}):\n"
            f"{result.stderr.decode()}"
        )

    # Only ping Vigilmon on success — a failed job should let the heartbeat expire
    requests.get(HEARTBEAT_URL, timeout=10)


if __name__ == "__main__":
    start = time.time()
    try:
        count = run_pipeline()
        elapsed = time.time() - start
        print(f"Pipeline complete: {count} vertices processed in {elapsed:.1f}s")
        # Final heartbeat on successful completion
        requests.get(HEARTBEAT_URL, timeout=10)
    except Exception as exc:
        # Do NOT ping Vigilmon on failure — let the heartbeat expire and alert
        print(f"Pipeline failed: {exc}")
        raise

Reindex Job Heartbeat

For JanusGraph mixed index reindexing jobs (synchronising vertex data with Elasticsearch):

# reindex_job.py
import os
import requests
from gremlin_python.driver import client, serializer

GREMLIN_URL = os.environ["GREMLIN_URL"]
HEARTBEAT_URL = os.environ["VIGILMON_REINDEX_HEARTBEAT_URL"]


def run_reindex(index_name: str):
    gremlin_client = client.Client(
        GREMLIN_URL,
        "g",
        message_serializer=serializer.GraphSONSerializersV2d0(),
    )
    try:
        # Trigger index repair/reindex via JanusGraph management API
        gremlin_client.submit(
            f"""
            mgmt = graph.openManagement()
            idx = mgmt.getGraphIndex('{index_name}')
            mgmt.updateIndex(idx, SchemaAction.REINDEX).get()
            mgmt.commit()
            """
        ).all().result()

        # Ping Vigilmon only after successful commit
        requests.get(HEARTBEAT_URL, timeout=10)
        print(f"Reindex of '{index_name}' complete")
    finally:
        gremlin_client.close()

Step 4: Alert Routing and Response Time Thresholds

JanusGraph alert priority depends on the storage backend: a Cassandra backend failure in a distributed production cluster warrants immediate paging, while a BerkeleyDB failure in a development environment is lower priority. Response time thresholds also differ by backend — Cassandra and HBase introduce network round-trips that BerkeleyDB (in-process) does not.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | Response Time Threshold | |---|---|---|---| | JanusGraph node health (Cassandra backend) | Slack + PagerDuty | P1 | 8000ms | | JanusGraph node health (HBase backend) | Slack + PagerDuty | P1 | 10000ms | | JanusGraph node health (BerkeleyDB backend) | Slack + email | P2 | 3000ms | | Schema lock contention check | Slack + email | P2 | 5000ms | | HBase REST API / Cassandra health | Slack + PagerDuty | P1 | 5000ms | | Heartbeat: graph analytics Spark job | Slack + email | P2 | — | | Heartbeat: Gremlin traversal pipeline | Slack + email | P2 | — | | Heartbeat: mixed index reindex job | Email | P3 | — |

Response time threshold guidance by storage backend:

• Cassandra: The Gremlin probe traversal crosses the Cassandra native transport layer. Allow 6–10 seconds for the threshold; alert at 10+ seconds as a signal of Cassandra read latency pressure.
• HBase: HBase region server round-trips can add 20–50ms per operation. Allow 8–12 seconds for health probe thresholds; a sudden increase above baseline typically precedes region server failure.
• BerkeleyDB: In-process storage means probe latency should be under 500ms under normal conditions. A threshold breach above 2–3 seconds is a strong signal of lock table pressure or JVM GC pause.

For Gremlin Server thread pool exhaustion (all workers occupied by slow traversals), the health probe itself will queue behind the slow traversals and eventually time out — which means Vigilmon's response time alert will fire before the health check returns a 503. Set the response time threshold at roughly 50% of your Gremlin Server threadPoolWorker timeout to catch pool pressure early.

Summary

JanusGraph's distributed architecture means process liveness is not a useful health signal. You need external monitoring at three layers:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/janusgraph | Gremlin Server reachability, storage backend connectivity, traversal execution | | HTTP monitor on /health/janusgraph/schema | Schema lock contention from ghost JVM instances | | HTTP monitor on HBase/Cassandra health endpoint | Storage backend health independent of JanusGraph | | Heartbeat monitor on graph analytics jobs | Spark OLAP job and Gremlin traversal pipeline liveness | | Heartbeat monitor on reindex jobs | Mixed index synchronisation with Elasticsearch/Solr |

Get started free at vigilmon.online — your first JanusGraph monitor is running in under two minutes.