tutorial

Monitoring OpenFGA with Vigilmon

OpenFGA is the open-source fine-grained authorization engine from Auth0/Okta — here's how to monitor authorization check latency, store health, relationship tuple ingestion, and consistency lag with Vigilmon.

OpenFGA is an open-source, scalable Fine-Grained Authorization (FGA) system originally developed by Auth0/Okta and now a CNCF sandbox project. It implements Google Zanzibar-style relationship-based access control (ReBAC), letting you express complex authorization policies like "can this user edit this document?" in a simple, auditable model. When your application's entire permission system runs through OpenFGA, its availability and latency directly impact your users. Vigilmon gives you the uptime monitoring and alerting to know the moment your authorization layer degrades.

What You'll Set Up

  • Vigilmon HTTP checks on the OpenFGA REST and gRPC health endpoints
  • Authorization check latency tracking via Prometheus metrics
  • Store health and relationship tuple ingestion rate monitoring
  • Consistency lag detection for replicated deployments
  • Heartbeat monitoring for authorization model update pipelines

Prerequisites

  • OpenFGA server (v1.x) running via Docker or Kubernetes
  • A PostgreSQL or MySQL backend for tuple storage
  • A free Vigilmon account
  • Optional: Prometheus and Grafana for metrics dashboards

Step 1: Deploy OpenFGA with Metrics Enabled

Run OpenFGA with Prometheus metrics and health checks enabled:

docker run -d \
  --name openfga \
  -p 8080:8080 \
  -p 8081:8081 \
  -p 3000:3000 \
  -e OPENFGA_DATASTORE_ENGINE=postgres \
  -e OPENFGA_DATASTORE_URI="postgresql://user:pass@postgres:5432/openfga" \
  -e OPENFGA_METRICS_ENABLED=true \
  -e OPENFGA_METRICS_ADDR=:3000 \
  -e OPENFGA_PLAYGROUND_ENABLED=false \
  openfga/openfga:latest run

For Kubernetes, use the official Helm chart:

helm repo add openfga https://openfga.github.io/helm-charts
helm repo update

helm upgrade --install openfga openfga/openfga \
  --namespace openfga \
  --create-namespace \
  --set datastore.engine=postgres \
  --set datastore.uri="postgresql://user:pass@postgres:5432/openfga" \
  --set metrics.enabled=true \
  --set metrics.serviceMonitor.enabled=true

Verify the server is up:

curl http://localhost:8080/healthz
# {"status":"SERVING"}

Step 2: Add Vigilmon Health Endpoint Checks

2a. HTTP Health Check

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the URL: https://openfga.yourdomain.com/healthz
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Set Expected response body contains to SERVING.
  7. Click Save.

2b. Store List Endpoint

OpenFGA exposes a /stores endpoint that confirms the API is fully operational (not just alive):

curl -H "Authorization: Bearer <token>" \
  https://openfga.yourdomain.com/stores
# {"stores":[...],"continuation_token":""}

Add a second Vigilmon monitor:

  • URL: https://openfga.yourdomain.com/stores
  • Expected HTTP status: 200
  • Check interval: 2 minutes

If this fails while /healthz passes, your authorization server is alive but not serving requests correctly — a more actionable signal.

Step 3: Scrape Prometheus Metrics

OpenFGA exposes detailed Prometheus metrics on port 3000. Configure scraping:

# prometheus.yml
scrape_configs:
  - job_name: 'openfga'
    static_configs:
      - targets: ['openfga.openfga.svc.cluster.local:3000']
    scrape_interval: 15s

Key metrics to track:

| Metric | Description | Alert threshold | |--------|-------------|-----------------| | openfga_check_request_duration_seconds | Authorization check latency (p99) | > 200ms | | openfga_datastore_query_count_total | Database queries per second | sudden spike | | openfga_request_duration_seconds | Overall API request latency | > 500ms p99 | | openfga_list_objects_request_duration_seconds | ListObjects latency | > 1s p99 | | openfga_datastore_read_request_count_total | Tuple read operations | trend baseline | | openfga_dispatch_count_total | Internal dispatch calls (Zanzibar recursion depth proxy) | > 1000/s |

The most critical: openfga_check_request_duration_seconds bucket distribution. A spike in p99 above 200ms often precedes user-visible authorization failures in high-traffic applications.

Step 4: Monitor Authorization Check Latency

Write a latency probe that performs a real authorization check and reports to Vigilmon:

#!/usr/bin/env python3
# openfga_latency_check.py
import requests
import time
import sys

OPENFGA_URL = "https://openfga.yourdomain.com"
STORE_ID = "<YOUR_STORE_ID>"
VIGILMON_HEARTBEAT = "https://vigilmon.online/api/v1/heartbeat/<YOUR_TOKEN>"
LATENCY_THRESHOLD_MS = 200

def main():
    headers = {
        "Authorization": "Bearer <YOUR_API_TOKEN>",
        "Content-Type": "application/json"
    }

    payload = {
        "tuple_key": {
            "user": "user:monitor-probe",
            "relation": "viewer",
            "object": "document:probe-doc"
        }
    }

    start = time.monotonic()
    try:
        resp = requests.post(
            f"{OPENFGA_URL}/stores/{STORE_ID}/check",
            json=payload,
            headers=headers,
            timeout=5
        )
        elapsed_ms = (time.monotonic() - start) * 1000

        if resp.status_code != 200:
            print(f"ERROR: check returned {resp.status_code}")
            sys.exit(1)

        if elapsed_ms > LATENCY_THRESHOLD_MS:
            print(f"SLOW: check latency {elapsed_ms:.0f}ms > {LATENCY_THRESHOLD_MS}ms threshold")
            sys.exit(1)

        # All good — ping heartbeat
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
        print(f"OK: check latency {elapsed_ms:.0f}ms")

    except requests.RequestException as e:
        print(f"ERROR: {e}")
        sys.exit(1)

if __name__ == "__main__":
    main()

Set up a prerequisite probe document tuple before running:

curl -X POST "https://openfga.yourdomain.com/stores/${STORE_ID}/write" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "writes": {
      "tuple_keys": [{
        "user": "user:monitor-probe",
        "relation": "viewer",
        "object": "document:probe-doc"
      }]
    }
  }'

Schedule the check as a CronJob or systemd timer every minute.

Step 5: Relationship Tuple Ingestion Rate Monitoring

High-throughput applications write thousands of relationship tuples per second. Monitor ingestion health:

# Check the current tuple count in a store
curl "https://openfga.yourdomain.com/stores/${STORE_ID}/read" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"page_size": 1}'
# Returns continuation_token and total count in headers

Write an ingestion health check that measures write throughput:

#!/usr/bin/env python3
# openfga_ingestion_check.py
import requests
import time
import sys

OPENFGA_URL = "https://openfga.yourdomain.com"
STORE_ID = "<YOUR_STORE_ID>"
WRITE_LATENCY_THRESHOLD_MS = 100

headers = {
    "Authorization": "Bearer <YOUR_API_TOKEN>",
    "Content-Type": "application/json"
}

test_tuple = {
    "writes": {
        "tuple_keys": [{
            "user": f"user:ingestion-probe-{int(time.time())}",
            "relation": "viewer",
            "object": "document:probe-doc"
        }]
    }
}

start = time.monotonic()
resp = requests.post(
    f"{OPENFGA_URL}/stores/{STORE_ID}/write",
    json=test_tuple,
    headers=headers,
    timeout=5
)
elapsed_ms = (time.monotonic() - start) * 1000

if resp.status_code != 200:
    print(f"WRITE ERROR: {resp.status_code} {resp.text}")
    sys.exit(1)

if elapsed_ms > WRITE_LATENCY_THRESHOLD_MS:
    print(f"SLOW WRITE: {elapsed_ms:.0f}ms (threshold: {WRITE_LATENCY_THRESHOLD_MS}ms)")
    sys.exit(1)

print(f"OK: write latency {elapsed_ms:.0f}ms")

Add this as a Vigilmon heartbeat monitor — if tuple writes start failing or slowing, you'll know before your application users notice authorization data going stale.

Step 6: Consistency Lag Monitoring for Replicated Deployments

If you run multiple OpenFGA replicas backed by a replicated database, read-after-write consistency becomes critical. A tuple written on one replica must be visible on others within your consistency SLA.

Configure OpenFGA's consistency settings:

# openfga-values.yaml for Helm
consistency:
  enabled: true
  slower_request_threshold_in_milliseconds: 100

Monitor consistency lag via custom probe:

#!/bin/bash
# consistency_probe.sh
# Write a tuple to replica A, immediately read from replica B
STORE_ID="<YOUR_STORE_ID>"
REPLICA_A="https://openfga-0.openfga.svc.cluster.local:8080"
REPLICA_B="https://openfga-1.openfga.svc.cluster.local:8080"
PROBE_USER="user:consistency-probe-$(date +%s)"

# Write to replica A
curl -s -X POST "$REPLICA_A/stores/$STORE_ID/write" \
  -H "Authorization: Bearer $TOKEN" \
  -d "{\"writes\":{\"tuple_keys\":[{\"user\":\"$PROBE_USER\",\"relation\":\"viewer\",\"object\":\"document:probe\"}]}}"

# Read from replica B immediately
RESULT=$(curl -s -X POST "$REPLICA_B/stores/$STORE_ID/check" \
  -H "Authorization: Bearer $TOKEN" \
  -d "{\"tuple_key\":{\"user\":\"$PROBE_USER\",\"relation\":\"viewer\",\"object\":\"document:probe\"}}")

ALLOWED=$(echo "$RESULT" | jq -r '.allowed')
if [ "$ALLOWED" != "true" ]; then
  echo "CONSISTENCY LAG DETECTED: write on A not visible on B"
  exit 1
fi
echo "OK: cross-replica consistency confirmed"

If this fails consistently, investigate your database replication lag or increase OpenFGA's OPENFGA_CHECK_QUERY_CACHE_TTL.

Step 7: Store Health and Multi-Store Monitoring

For multi-tenant deployments with one store per tenant, monitor store health in aggregate:

#!/usr/bin/env python3
# openfga_stores_health.py
import requests

OPENFGA_URL = "https://openfga.yourdomain.com"
headers = {"Authorization": "Bearer <token>"}

stores = []
continuation_token = ""

while True:
    params = {"page_size": 50}
    if continuation_token:
        params["continuation_token"] = continuation_token

    resp = requests.get(f"{OPENFGA_URL}/stores", headers=headers, params=params)
    data = resp.json()
    stores.extend(data.get("stores", []))

    continuation_token = data.get("continuation_token", "")
    if not continuation_token:
        break

print(f"Total stores: {len(stores)}")
for store in stores:
    print(f"  {store['id']}: {store['name']} (created: {store['created_at']})")

Add a Vigilmon monitor that alerts if the store count drops unexpectedly (indicating accidental store deletion):

  1. Add a monitor with URL: https://openfga.yourdomain.com/stores
  2. Set Expected response body contains to a known store ID from your primary tenant.

Alerting Configuration

Configure Vigilmon alert channels in Settings → Notifications:

  • PagerDuty: immediately page on-call when /healthz returns non-200 (authorization outage = all users locked out)
  • Slack: post to #authz-alerts on latency threshold breaches
  • Email: send to security team on ingestion write failures
  • Webhooks: trigger circuit-breaker logic in your API gateway to fall back to cached permissions

Set multi-step escalation: warn at p99 > 200ms, page at p99 > 500ms, escalate to engineering lead if down for 5 minutes.

Troubleshooting

Check requests returning 403 Verify your authorization model and tuple setup:

curl -X POST "https://openfga.yourdomain.com/stores/${STORE_ID}/check" \
  -H "Authorization: Bearer <token>" \
  -d '{"tuple_key":{"user":"user:alice","relation":"owner","object":"doc:readme"}}'

A 403 from the FGA server itself means your admin token is wrong; a {"allowed":false} response means the tuple doesn't exist — expected for the probe setup.

High dispatch count openfga_dispatch_count_total growing rapidly indicates deep relationship graph traversal. Review your authorization model for overly broad union or intersection rules that cause O(n) lookups.

Database connection pool exhausted

kubectl logs -n openfga deployment/openfga | grep "connection pool"

Increase OPENFGA_DATASTORE_MAX_OPEN_CONNS and review Postgres connection limits.

Summary

You now have full-stack OpenFGA monitoring with Vigilmon:

  • API health checks every minute via /healthz and /stores
  • Authorization check latency probes with 200ms alerting threshold
  • Relationship tuple ingestion monitoring measuring write throughput
  • Cross-replica consistency probes to catch replication lag
  • Multi-store health checks to detect accidental store deletion

OpenFGA is the foundation of your application's access control — Vigilmon ensures that foundation stays solid.

Ready to monitor your OpenFGA deployment? Create a free Vigilmon account and add your first authorization health check in minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles