tutorial

Monitoring SpiceDB Permissions Database with Vigilmon: Health API, gRPC Port, Zanzibar Engine & SSL Certificate Alerts

How to monitor SpiceDB permissions database infrastructure with Vigilmon — health endpoint checks, gRPC port monitoring, permission check API availability, and SSL certificate alerts.

How to Monitor SpiceDB Permissions Database Infrastructure with Vigilmon

SpiceDB is the open-source implementation of Google's Zanzibar permissions system — a globally consistent, fine-grained permissions database that powers authorization for applications at Google-scale. Teams use SpiceDB to model complex permission relationships (user → group → role → resource) and evaluate permission checks in milliseconds with consistency guarantees. SpiceDB sits in the critical path of every authenticated action in your application.

When SpiceDB is down, permission checks fail and your application can't determine what users are allowed to do. External monitoring with Vigilmon ensures you know immediately when SpiceDB is unreachable or degraded — before cascading authorization failures reach your users.

Why SpiceDB needs external monitoring

SpiceDB's internal health endpoints report database and engine health, but miss the failure modes that applications encounter in the critical permission-check path:

  • SpiceDB loses connection to its backing database (PostgreSQL/CockroachDB) — the process is alive but all permission checks fail; the health endpoint may report NOT_SERVING only after a delay
  • gRPC port is bound but the permission check handler panics under load — the port accepts connections while individual permission check RPCs return errors
  • Consistency token (zookie) validation breaks after a database failover — permission checks requiring FullyConsistent start timing out while health checks with no consistency requirement pass
  • TLS certificate expires on the SpiceDB endpoint — all gRPC clients fail immediately; internal process checks using plain health gRPC pass
  • Schema migration failure leaves SpiceDB in an inconsistent state — the engine accepts new relationship writes but reads return stale results
  • SpiceDB dispatcher (for distributed deployments) loses cluster members — some permission checks route to dead nodes and time out while others succeed

External monitoring from Vigilmon catches authorization database failures that internal probes miss.

What you'll need

  • A running SpiceDB instance (standalone Docker, Kubernetes, or SpiceDB Cloud)
  • HTTP or gRPC endpoint accessible from outside the server
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Understand SpiceDB's health endpoints

SpiceDB exposes health via its gRPC health protocol and an optional HTTP metrics endpoint. The primary ports:

  • 50051 — gRPC API (permission checks, relationship writes, schema management)
  • 8080 — HTTP metrics (Prometheus-compatible)
  • 8443 — gRPC with TLS (if TLS is configured)
# gRPC health check (requires grpc_health_probe)
grpc_health_probe -addr=localhost:50051

# HTTP metrics endpoint (always accessible if --metrics-api-addr is set)
curl -s http://localhost:8080/metrics | grep spicedb_

# Test a permission check via gRPC using zed CLI
zed permission check document:readme reader user:alice \
  --endpoint=localhost:50051 \
  --insecure

# Test via SpiceDB's HTTP API (requires --http-enabled flag)
curl -s -X POST http://localhost:8080/v1/permissions/check \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_PRESHARED_KEY" \
  -d '{
    "resource": {"objectType": "document", "objectId": "readme"},
    "permission": "reader",
    "subject": {"object": {"objectType": "user", "objectId": "alice"}}
  }' | python3 -m json.tool

Starting SpiceDB with Docker

docker run -d \
  --name spicedb \
  -p 50051:50051 \
  -p 8080:8080 \
  authzed/spicedb serve \
    --grpc-preshared-key "your-preshared-key" \
    --datastore-engine postgres \
    --datastore-conn-uri "postgresql://spicedb:secret@postgres:5432/spicedb?sslmode=disable" \
    --http-enabled

Step 2: Add a TCP monitor for the gRPC port

SpiceDB's primary API is gRPC. The most direct availability check is a TCP port monitor:

  1. Click Monitors → New Monitor
  2. Set the Type to TCP
  3. Enter: spicedb.yourdomain.com:50051
  4. Set Check interval: 1 minute
  5. Set Alert after to 1 failure — permission database outages break authorization for every user
  6. Click Save

This is your primary monitor because SpiceDB's gRPC port is the critical path. TCP-level monitoring detects port binding failures and process crashes immediately.

What this monitor catches

| Failure | Vigilmon response | |---------|-----------------| | SpiceDB process crashed | Connection refused → alert | | Database backend unreachable | Port still bound but gRPC returns errors | | OOM kill of SpiceDB process | Connection refused → alert | | Kubernetes pod not scheduled | Connection refused → alert | | TLS handshake failure | TCP failure → alert | | Firewall rule blocking port | Connection refused → alert |

Step 3: Create an HTTP monitor for the metrics endpoint

If you've enabled SpiceDB's HTTP API or metrics endpoint, monitor it for HTTP-level availability:

  1. Click Monitors → New Monitor
  2. Set the Type to HTTP
  3. Enter: https://spicedb.yourdomain.com:8080/metrics
  4. Set Expected status: 200
  5. Set Keyword to find: spicedb_
  6. Set Check interval: 2 minutes
  7. Click Save

The keyword check (spicedb_) verifies that Prometheus metrics are actually being generated by the SpiceDB process — not just that the HTTP server is responding.

Step 4: Monitor the HTTP permission check API

If SpiceDB is configured with --http-enabled, you can monitor the permission check endpoint directly:

  1. Click Monitors → New Monitor
  2. Set the Type to HTTP
  3. Enter: https://spicedb.yourdomain.com/v1/permissions/check
  4. Set Method: POST
  5. Set Request headers:
    • Content-Type: application/json
    • Authorization: Bearer YOUR_PRESHARED_KEY
  6. Set Request body: 
    {
  "resource": {"objectType": "health", "objectId": "vigilmon"},
  "permission": "read",
  "subject": {"object": {"objectType": "user", "objectId": "monitor"}},
  "consistency": {"minimizeLatency": true}
}
  7. Set Expected status: 200
  8. Set Check interval: 5 minutes
  9. Click Save

Note: this check will return a PERMISSIONSHIP_NO_PERMISSION result (which is still HTTP 200) since the health-check relationship likely doesn't exist in your schema — that's expected. The monitor verifies the permission engine responds to requests, not the specific result.

Step 5: Configure alert channels

Navigate to Alerts → New Alert Channel:

Slack integration:

# In Slack, create an incoming webhook for #security or #permissions-alerts
# In Vigilmon, paste the webhook URL under Alerts → Slack

PagerDuty / Opsgenie:

POST https://events.pagerduty.com/v2/enqueue
{
  "routing_key": "YOUR_INTEGRATION_KEY",
  "event_action": "trigger",
  "payload": {
    "summary": "SpiceDB permissions database is DOWN — all authorization checks failing",
    "severity": "critical",
    "source": "vigilmon"
  }
}

Email: Configure alerts to your platform engineering and security on-call rotation.

Example Vigilmon webhook payload:

{
  "monitor_name": "SpiceDB gRPC port",
  "status": "down",
  "url": "spicedb.yourdomain.com:50051",
  "started_at": "2026-03-10T14:52:00Z",
  "duration_seconds": 120
}

Step 6: Correlate Vigilmon alerts with SpiceDB diagnostics

When an alert fires, run this triage sequence:

# 1. Is the SpiceDB process running?
docker ps | grep spicedb
# or for systemd:
systemctl status spicedb

# 2. Check SpiceDB logs for database or schema errors
docker logs spicedb --tail 100 | grep -E "ERROR|FAIL|panic|datastore" -i

# 3. Test the gRPC health check directly
grpc_health_probe -addr=localhost:50051
# or with TLS:
grpc_health_probe -addr=localhost:8443 -tls

# 4. Test with grpcurl
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

# 5. Verify database connectivity from SpiceDB
docker exec spicedb spicedb migrate head \
  --datastore-engine postgres \
  --datastore-conn-uri "postgresql://spicedb:secret@postgres:5432/spicedb?sslmode=disable"

# 6. Check Prometheus metrics for error rates
curl -s http://localhost:8080/metrics | grep -E "spicedb_check_errors|spicedb_datastore"

# 7. Test a real permission check
zed permission check document:test read user:test \
  --endpoint=localhost:50051 \
  --insecure \
  --log-level=debug

Triage decision tree:

  • Process not running → check host OOM (dmesg | grep killed), restart SpiceDB, verify database connection string
  • Process running, gRPC check fails → database backend unreachable; verify PostgreSQL/CockroachDB connectivity and credentials
  • gRPC health passes but permission checks time out → dispatcher cluster issue (for multi-node) or datastore query timeout; check slow query logs
  • Permission checks return errors on new schema → schema migration may have failed; run spicedb migrate head manually
  • Everything works locally but Vigilmon shows TCP down → firewall rules or Kubernetes NetworkPolicy blocking external access to port 50051

Step 7: Kubernetes deployment monitoring

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spicedb
  namespace: authz
spec:
  replicas: 3
  selector:
    matchLabels:
      app: spicedb
  template:
    metadata:
      labels:
        app: spicedb
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: spicedb
          image: authzed/spicedb:latest
          args:
            - serve
            - --grpc-preshared-key=$(PRESHARED_KEY)
            - --datastore-engine=cockroachdb
            - --datastore-conn-uri=$(DATABASE_URL)
            - --http-enabled
            - --dispatch-cluster-enabled
          env:
            - name: PRESHARED_KEY
              valueFrom:
                secretKeyRef:
                  name: spicedb-secrets
                  key: preshared-key
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: spicedb-secrets
                  key: database-url
          ports:
            - containerPort: 50051
              name: grpc
            - containerPort: 50053
              name: dispatch
            - containerPort: 8080
              name: metrics
          livenessProbe:
            exec:
              command:
                - grpc_health_probe
                - -addr=:50051
            initialDelaySeconds: 15
            periodSeconds: 20
          readinessProbe:
            exec:
              command:
                - grpc_health_probe
                - -addr=:50051
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "1000m"
---
apiVersion: v1
kind: Service
metadata:
  name: spicedb
  namespace: authz
spec:
  selector:
    app: spicedb
  ports:
    - name: grpc
      port: 50051
      targetPort: 50051
    - name: metrics
      port: 8080
      targetPort: 8080
  type: LoadBalancer

Expose SpiceDB via a LoadBalancer or Ingress and monitor the external endpoint with Vigilmon. Kubernetes liveness probes tell you what the pod scheduler sees; Vigilmon tells you what your application services and external clients actually reach.

Step 8: Create a status page for authorization infrastructure

  1. Go to Status Pages → New Status Page
  2. Name it: "Permissions Infrastructure"
  3. Add your monitors grouped by component:
    • SpiceDB Core: gRPC TCP port, HTTP metrics endpoint
    • Permission API: HTTP permission check (if enabled)
  4. Publish and share with:
    • Application teams whose services depend on SpiceDB for every authenticated action
    • Security teams tracking authorization infrastructure uptime
    • On-call engineers who need to quickly determine if auth failures are infrastructure vs. application

Putting it all together

| Monitor | Type | What it catches | |---------|------|----------------| | spicedb.yourdomain.com:50051 | TCP | gRPC port availability, process crashes | | https://spicedb.yourdomain.com:8080/metrics | HTTP keyword | SpiceDB metrics, process health | | https://spicedb.yourdomain.com/v1/permissions/check | HTTP | End-to-end permission evaluation path |

What's next

  • Prometheus + Grafana — scrape SpiceDB's metrics endpoint for check latency, datastore query times, and dispatch cluster health; use Vigilmon for the external availability signal Prometheus can't provide
  • SSL certificate monitoring — Vigilmon tracks TLS cert expiry for your SpiceDB endpoints automatically
  • Multi-region monitoring — if you run SpiceDB in multiple regions for latency, add a Vigilmon monitor for each regional gRPC endpoint

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles