tutorial

Monitoring Ory Oathkeeper with Vigilmon

Ory Oathkeeper is an Identity and Access Proxy that enforces authentication and authorization rules on every request — making its health critical to your entire service mesh. Here's how to monitor it with Vigilmon.

Ory Oathkeeper sits in front of your upstream services and enforces configurable access rules — deciding who can reach what based on JWT validation, OAuth2 introspection, cookie sessions, and OPA/Keto policy checks. Every proxied request goes through Oathkeeper's rule evaluation engine, which means downtime or misconfiguration has immediate, service-wide impact. Vigilmon gives you the health monitoring, endpoint checks, and alert routing to detect Oathkeeper failures before they affect end users.

What You'll Set Up

  • HTTP health check on the Oathkeeper proxy (port 4455)
  • HTTP health check on the Oathkeeper decisions API (port 4456)
  • Upstream authenticator connectivity monitors (JWKS, OPA, Keto)
  • TLS certificate expiry alert
  • Prometheus metrics scrape health
  • Alert channels for IAP failures

Prerequisites

  • Ory Oathkeeper running (proxy on 4455, management/decisions API on 4456)
  • Access rules configured and loaded
  • A free Vigilmon account

Why Monitor Ory Oathkeeper?

Oathkeeper is the enforcement point for your entire access control policy. When it fails:

  • All proxied traffic is blocked — the proxy on port 4455 is the gateway; if it's down, no requests reach upstream services
  • Decision API failures break sidecar deployments — if using Nginx auth_request against port 4456, Oathkeeper downtime blocks all auth_request-gated resources
  • Stale access rules let unauthorized requests through — if rule sync from file or API fails silently, Oathkeeper evaluates outdated rules
  • Authenticator endpoint failures cause partial outages — if the JWKS endpoint is unreachable, JWT validation fails; if OPA is down, OPA-authorized rules reject all requests

Because Oathkeeper composes multiple security layers (authenticator + authorizer + mutator), each component needs independent monitoring.

Key Metrics to Monitor

| Metric | Why It Matters | |---|---| | Proxy port 4455 liveness | Downtime blocks all proxied traffic | | Decisions API port 4456 liveness | Downtime blocks all auth_request-based deployments | | JWKS endpoint reachability | JWT authenticator fails if JWKS is unreachable | | OPA server connectivity | OPA authorizer fails for all OPA-backed access rules | | Keto service connectivity | Keto authorizer fails for all permission-based rules | | Access rule sync freshness | Stale rules cause incorrect allow/deny decisions | | Prometheus /metrics scrape | Reveals request_duration and access_decision counters |

Step 1: Monitor the Oathkeeper Proxy

The Oathkeeper reverse proxy listens on port 4455 and forwards authenticated requests to upstream services. Monitor its liveness endpoint:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter: http://oathkeeper:4455/.well-known/ory/webauthn.js
    • Or use the /health/alive endpoint if your version exposes it: http://oathkeeper:4455/health/alive
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

This is your highest-priority alert. If port 4455 is down, all proxied traffic is blocked.

Step 2: Monitor the Oathkeeper Decisions API

When using Oathkeeper in sidecar mode with Nginx auth_request, the decisions API on port 4456 is what Nginx consults. Monitor it separately:

  1. Click Add Monitor in Vigilmon.
  2. Set Type to HTTP / HTTPS.
  3. Enter: http://oathkeeper:4456/health/alive
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

If both ports are down, Oathkeeper itself has crashed. If only 4456 is down, the process may have partially started or the management API is misconfigured.

Step 3: Monitor Authenticator Endpoints

Oathkeeper's authenticators call external endpoints to validate credentials. Each authenticator type has a different dependency:

JWT Authenticator — JWKS Endpoint

If you use JWT authentication, Oathkeeper fetches the public key set from a JWKS URL to verify token signatures.

  1. Click Add Monitor.
  2. Enter your JWKS URL (from Ory Hydra, Keycloak, or your identity provider):
    • Ory Hydra: https://hydra.yourdomain.com/.well-known/jwks.json
    • Keycloak: https://keycloak.yourdomain.com/realms/yourrealm/protocol/openid-connect/certs
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 5 minutes.
  5. Click Save.

OAuth2 Introspection Endpoint

If using the oauth2_introspection authenticator:

  1. Click Add Monitor.
  2. Enter the introspection endpoint (from your OAuth2 server).
  3. Set Expected HTTP status to 401 — a GET without credentials returns 401, confirming the endpoint is alive.
  4. Click Save.

Step 4: Monitor OPA Connectivity

If using the remote_json or OPA authorizer:

  1. Click Add Monitor in Vigilmon.
  2. Enter the OPA health endpoint: http://opa:8181/health
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Click Save.

OPA downtime causes Oathkeeper to deny all requests that go through OPA-backed authorizer rules, even if the user is correctly authenticated. A separate OPA health monitor isolates the root cause quickly.

Step 5: Monitor Keto Connectivity

If using the Keto authorizer for relationship-based access control:

  1. Click Add Monitor in Vigilmon.
  2. Enter Keto's health endpoint: http://keto:4466/health/alive
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Click Save.

Step 6: Monitor the Prometheus Metrics Endpoint

Oathkeeper exposes Prometheus metrics on /metrics that include access_decision_total (labeled by decision: allow/deny/error) and request_duration_seconds. Monitoring metrics scrape availability ensures you're collecting these signals.

  1. Click Add Monitor in Vigilmon.
  2. Enter: http://oathkeeper:9000/metrics (check your config for the actual metrics port; some deployments use 4456 or a dedicated 9000)
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 2 minutes.
  5. Click Save.

If you run Prometheus already, configure it to scrape Oathkeeper and alert on:

  • access_decision_total{decision="error"} > 0 — rule evaluation errors
  • rate(request_duration_seconds_count[5m]) == 0 — no traffic being processed

Step 7: TLS Certificate Expiry Alert

If Oathkeeper terminates TLS for the proxy endpoint, or if an upstream TLS dependency (JWKS, OPA) has a certificate:

  1. Edit the relevant monitor in Vigilmon.
  2. Enable SSL certificate expiry check.
  3. Set Alert threshold to 30 days.
  4. Click Save.

Step 8: Configure Alert Channels

Oathkeeper failures affect all traffic it proxies. Route alerts appropriately:

  1. Go to Settings → Alert Channels in Vigilmon.
  2. Add a Slack or PagerDuty integration for immediate on-call notification.
  3. Assign the proxy health monitor (port 4455) and decisions API monitor (port 4456) to the high-urgency on-call channel.
  4. Assign authenticator/OPA/Keto monitors to a secondary channel — these cause partial failures rather than total outages.

Set sensitivities:

  • Proxy monitor: alert after 1 failed check
  • OPA / Keto monitors: alert after 2 consecutive failures — brief restarts are normal
  • JWKS endpoint: alert after 3 consecutive failures — cached keys extend resilience

Step 9: Validate End-to-End

  1. Stop Oathkeeper: docker stop oathkeeper
  2. Verify the proxy monitor goes red in Vigilmon within 2 minutes.
  3. Confirm the Slack/PagerDuty alert fires.
  4. Restart Oathkeeper: docker start oathkeeper
  5. Verify recovery notification fires.

For OPA monitoring validation:

  1. Stop OPA: docker stop opa
  2. Confirm the OPA monitor alerts within 2 minutes.
  3. Restart OPA and verify recovery.

Troubleshooting Common Oathkeeper Issues

All proxied requests returning 500 or failing silently Check Oathkeeper logs: docker logs oathkeeper. Common causes: malformed access rule JSON, unreachable authenticator endpoint, or a misconfigured upstream URL. Use the rule validation endpoint to check rule syntax: GET http://oathkeeper:4456/rules.

JWT validation failures for all requests The JWKS endpoint is either unreachable or returning a key set that doesn't match the signing key used to issue tokens. Verify JWKS URL in your Oathkeeper config, confirm the JWT kid claim matches a key in the JWKS response, and check for clock skew between token issuer and Oathkeeper.

OPA authorizer denying all requests Check OPA is running and the policy bundle is loaded: curl http://opa:8181/v1/data. If the policy returns an empty result (not an explicit allow: true), Oathkeeper interprets it as deny. Verify your Rego policy returns allow = true for valid requests.

Access rules not updating after changes If using file-based rule loading, Oathkeeper only reloads rules on startup (by default). If using the API to manage rules, verify the management API is returning your updated rules via GET http://oathkeeper:4456/rules. Enable file watching in config if you need hot-reload from disk.

Conclusion

Ory Oathkeeper is a sophisticated multi-layer security gateway — but that sophistication means more potential failure points than a simple reverse proxy. Monitoring the proxy port, decisions API, and each authenticator/authorizer dependency gives you granular visibility into which layer failed when something goes wrong.

Start with the port 4455 proxy monitor and port 4456 decisions API monitor — those cover the most impactful failures. Add JWKS and OPA monitors to cover the most common authentication and authorization failure modes. The Prometheus metrics scrape check gives you access decision counters for trend analysis.

Sign up for Vigilmon and add the Oathkeeper proxy health check as your first monitor.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles