How to Monitor Datomic Uptime and Health with Vigilmon

Datomic takes an unconventional approach to database architecture: it separates the transactor (writes), peer library (reads), and storage backend (persistence) into independent components. That separation makes Datomic remarkably flexible, but it also means there are multiple failure points — the transactor, the peer server, the underlying storage (DynamoDB, Cassandra, SQL), and your application's connection pool.

Vigilmon gives you external visibility into Datomic-backed services through HTTP probe monitoring and heartbeat monitoring for background indexing and query workers. This tutorial walks through setting up both.

Why Datomic Monitoring Matters

Datomic's architecture makes failures non-obvious:

• Transactor crash — writes fail immediately, but reads from a cached database value continue to succeed, masking the outage from application health checks
• Peer connection exhaustion — Datomic peers maintain a connection pool to the transactor; when this pool is exhausted, new transactions block silently
• Storage backend degradation — slow DynamoDB reads or Cassandra timeouts propagate into query latency without generating a clear error
• Indexing lag — Datomic indexes asynchronously; long indexing queues inflate query time for new data
• Out-of-memory in the peer — large queries can exhaust heap in the peer process, causing OOM kills without a visible transactor error

External monitoring through Vigilmon tests the full write and query path from outside, catching failures that internal health checks miss.

Step 1: Build a Datomic Health Endpoint

Clojure (Ring / Compojure)

;; health.clj — Datomic health endpoint
(ns myapp.health
  (:require [datomic.api :as d]
            [ring.util.response :refer [response status]]))

(def conn (d/connect (System/getenv "DATOMIC_URI")))

(defn datomic-health []
  (try
    ;; Verify transactor connectivity with a lightweight read
    (let [db     (d/db conn)
          ;; Query the :db/ident of the first schema attribute — always present
          result (d/q '[:find ?e :where [?e :db/ident]] db)]
      (if (seq result)
        (-> (response {:status "ok" :facts (count result)})
            (status 200))
        (-> (response {:status "degraded" :reason "empty-schema"})
            (status 503))))
    (catch Exception e
      (-> (response {:status "down" :error (.getMessage e)})
          (status 503)))))

;; routes.clj — wire up the health endpoint
(ns myapp.routes
  (:require [compojure.core :refer [GET defroutes]]
            [myapp.health :refer [datomic-health]]))

(defroutes app-routes
  (GET "/health/datomic" [] (datomic-health)))

Clojure — Write Probe (More Thorough)

A read-only probe won't catch a crashed transactor if your peer has a cached db value. Add a write probe to verify the transactor is actually accepting transactions:

(defn datomic-write-health []
  (try
    (let [;; Transact a retractable probe entity
          result @(d/transact conn
                              [{:db/id    "datomic.temp/probe"
                                :db/doc   "__vigilmon_health_probe__"}])
          db-after (:db-after result)]
      ;; Retract the probe entity immediately
      (let [probe-eid (d/q '[:find ?e .
                              :where [?e :db/doc "__vigilmon_health_probe__"]]
                           db-after)]
        (when probe-eid
          @(d/transact conn [[:db/retractEntity probe-eid]])))
      {:status 200 :body {:status "ok" :t (:db/t db-after)}})
    (catch Exception e
      {:status 503 :body {:status "down" :error (.getMessage e)}})))

Peer Server (HTTP API for Datomic Cloud / On-Prem Peer Server)

If you run the Datomic peer server, it already exposes an HTTP API. Add a lightweight query endpoint to your application that proxies to it:

;; Use datomic.client.api for Datomic Cloud
(require '[datomic.client.api :as d])

(def client (d/client {:server-type :peer-server
                        :endpoint    (System/getenv "DATOMIC_PEER_SERVER_ENDPOINT")
                        :secret      (System/getenv "DATOMIC_SECRET")
                        :access-key  (System/getenv "DATOMIC_ACCESS_KEY")}))

(defn cloud-health []
  (try
    (let [conn (d/connect client {:db-name (System/getenv "DATOMIC_DB_NAME")})
          db   (d/db conn)
          res  (d/q {:query '[:find (count ?e) . :where [?e :db/ident]]
                     :args  [db]})]
      {:status 200 :body {:status "ok" :schema-count res}})
    (catch Exception e
      {:status 503 :body {:status "down" :error (.getMessage e)}})))

Verify the endpoint:

curl -i https://your-app.example.com/health/datomic
# HTTP/1.1 200 OK
# {"status":"ok","t":12345}

Step 2: Configure a Vigilmon HTTP Monitor for Datomic

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your Datomic health endpoint: https://your-app.example.com/health/datomic
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 5000ms (allow for indexing and query time)
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Vigilmon probes from multiple geographic regions and requires multi-region consensus before opening an incident. Transient network hiccups won't wake your on-call.

What This Catches

| Failure | Process monitoring | Vigilmon | |---|---|---| | Transactor crash (write-only failure) | ✗ | ✓ (with write probe) | | Storage backend degradation | ✗ | ✓ | | Peer connection exhaustion | ✗ | ✓ | | OOM in peer process | ✗ | ✓ | | Application crash | ✓ | ✓ |

Step 3: Heartbeat Monitoring for Datomic Background Workers

Datomic-backed applications commonly run:

• Scheduled query jobs that aggregate historical data (time travel queries are cheap in Datomic)
• Event sourcing consumers that transact external events into Datomic
• Background indexers that build derived indexes from Datomic as-of queries

These jobs run in separate JVM processes and can stall silently. Vigilmon's heartbeat monitors detect silent death.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: datomic-event-consumer
3. Set the expected interval: match your consumer's expected processing cycle
4. Set the grace period: 2× the interval
5. Save — copy the unique heartbeat URL

Wire It Into Your Consumer

Clojure core.async consumer:

(require '[clj-http.client :as http])

(defn start-event-consumer [conn event-ch]
  (a/go-loop []
    (when-let [event (a/<! event-ch)]
      ;; Transact the event into Datomic
      (d/transact conn [(event->tx-data event)])
      ;; Ping Vigilmon heartbeat
      (http/get (System/getenv "VIGILMON_HEARTBEAT_URL")
                {:throw-exceptions false})
      (recur))))

Java consumer (Datomic JNI):

public void runConsumer() throws Exception {
    while (!Thread.interrupted()) {
        Event event = eventQueue.poll(30, TimeUnit.SECONDS);
        if (event != null) {
            Connection conn = Peer.connect(datomicUri);
            conn.transact(eventToTxData(event)).get();
            HttpClient.newHttpClient().send(
                HttpRequest.newBuilder()
                    .uri(URI.create(System.getenv("VIGILMON_HEARTBEAT_URL")))
                    .GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        }
    }
}

Step 4: Monitor the Transactor Separately

The Datomic transactor is a standalone JVM process that can be monitored independently from your application. If you run the transactor with the built-in health API or JMX, expose it:

# Datomic transactor exposes a /health endpoint when configured
# Add to transactor.properties:
# health-endpoint-port=9999
curl -i http://transactor-host:9999/health

Create a separate Vigilmon monitor for the transactor health endpoint:

• URL: http://transactor-host:9999/health (internal monitor via VPN/private network if needed)
• Expected status: 200
• Interval: 1 minute

If the transactor is not directly reachable from Vigilmon's probes, route the health check through your application's network:

;; Expose transactor connectivity check via your app
(app.routes/GET "/health/transactor" []
  (try
    (let [info (d/transactor-info conn)]
      {:status 200 :body {:status "ok" :info info}})
    (catch Exception e
      {:status 503 :body {:status "down" :error (.getMessage e)}})))

Step 5: Alert Routing for Datomic Failures

Datomic transactor failures are critical — without the transactor, no writes succeed. Configure escalating alerts:

1. Write probe monitor → immediate Slack + PagerDuty page (P1 — transactor down)
2. Read-only health monitor → Slack (P2 — storage degraded, reads still serve stale data)
3. Event consumer heartbeat → Slack + email (P2 — background processing stalled)
4. Response time threshold → Slack (P3 — indexing lag early warning)

Group the transactor monitor, peer health monitor, and storage health check into a Datomic Status Page in Vigilmon.

Summary

Datomic's separation of write, read, and storage components means conventional single-process monitoring misses most failure modes. Vigilmon gives you:

| Monitor Type | What It Covers | |---|---| | HTTP write probe | Transactor connectivity, transaction acceptance | | HTTP read probe | Peer availability, storage backend health | | Heartbeat monitor | Event consumer and background query job liveness | | Response time threshold | Indexing lag and storage degradation early warning |

Get started free at vigilmon.online — your first Datomic monitor is running in under two minutes.