tutorial

How to Monitor Apache Ranger Health and Policy Engine Availability with Vigilmon

Ranger policy engine outages silently open or lock down your Hadoop security perimeter. Learn how to monitor Ranger admin server health, policy sync status, and client plugin liveness with Vigilmon HTTP probes and heartbeat monitors.

Apache Ranger is the centralized security framework that enforces fine-grained access control policies across HDFS, Hive, HBase, Kafka, and dozens of other Hadoop ecosystem components — but when the Ranger admin server crashes, the policy sync service falls behind, or a plugin loses its connection to the admin server, your security posture either locks all users out silently or fails open, depending on your plugin's fallback configuration. Neither failure mode is acceptable, and neither one fires an obvious alert.

Vigilmon gives you external visibility into Ranger cluster health through HTTP probe monitoring (via Ranger's admin REST API or a custom health sidecar) and heartbeat monitoring for your Ranger policy sync and audit processes. This tutorial covers both.

Why Ranger Needs External Monitoring

Ranger's built-in tooling (Ranger Admin UI, audit logs, plugin sync status dashboards) provides rich operational data — but only when someone is actively checking. External monitoring with Vigilmon adds:

  • Proactive alerting when the Ranger Admin REST API becomes unreachable (early sign of admin server crash, DB connection loss, or GC storm)
  • Policy sync detection via Ranger's policy download endpoint that confirms plugins are receiving fresh policy updates
  • Heartbeat monitoring so you know immediately when a Ranger audit writer or policy evaluator stops functioning correctly
  • Availability checking from outside the cluster — confirming Ranger is reachable from the same network segment as your policy-enforcing components

These layers work together: the Ranger Admin UI can load while the policy distribution REST endpoint returns stale data, and the admin server can be healthy while a specific plugin's policy cache has grown hours out of date due to a network partition.

Step 1: Build a Ranger Health Endpoint

Ranger Admin exposes a REST API on port 6080 (HTTP) or 6182 (HTTPS). Use it directly or wrap it in a custom health sidecar.

Using the Ranger Admin REST API Directly

# Check Ranger Admin server health
curl -i -u admin:admin http://ranger-admin.example.com:6080/service/public/api/repository/count

# List all registered services (confirms DB connectivity and policy store is accessible)
curl -i -u admin:admin http://ranger-admin.example.com:6080/service/public/v2/api/service

# Check policy download for a specific service (confirms plugin sync is working)
curl -i -u admin:admin \
  "http://ranger-admin.example.com:6080/service/plugins/policies/download/hadoop-service"

Point a Vigilmon monitor at the repository count endpoint for a quick liveness check. Note that basic auth credentials should be passed via a proxy or sidecar — do not embed credentials in Vigilmon monitor URLs.

Custom Python Health Sidecar

# ranger_health.py
from flask import Flask, jsonify
import requests
import os
import time

app = Flask(__name__)

RANGER_URL = os.environ.get('RANGER_URL', 'http://localhost:6080')
RANGER_USER = os.environ.get('RANGER_USER', 'admin')
RANGER_PASS = os.environ.get('RANGER_PASS', '')
POLICY_SERVICE = os.environ.get('RANGER_SERVICE_NAME', 'hadoop-service')
MAX_POLICY_AGE_SECONDS = int(os.environ.get('RANGER_MAX_POLICY_AGE', '300'))

auth = (RANGER_USER, RANGER_PASS)

@app.route('/health/ranger')
def ranger_health():
    try:
        # Check admin server is reachable and DB is accessible
        count_resp = requests.get(
            f'{RANGER_URL}/service/public/api/repository/count',
            auth=auth, timeout=5
        )
        if count_resp.status_code != 200:
            return jsonify({'status': 'down',
                            'reason': 'admin_api_error',
                            'http_status': count_resp.status_code}), 503

        # Check policy download timestamp to detect stale sync
        policy_resp = requests.get(
            f'{RANGER_URL}/service/plugins/policies/download/{POLICY_SERVICE}',
            auth=auth, timeout=10
        )
        if policy_resp.status_code == 200:
            policy_data = policy_resp.json()
            # Ranger returns policyUpdateTime in milliseconds epoch
            update_time_ms = policy_data.get('policyUpdateTime', 0)
            age_seconds = time.time() - (update_time_ms / 1000)

            if age_seconds > MAX_POLICY_AGE_SECONDS:
                return jsonify({
                    'status': 'degraded',
                    'reason': 'stale_policy_sync',
                    'policy_age_seconds': int(age_seconds),
                }), 503

        return jsonify({
            'status': 'ok',
            'service_count': count_resp.json(),
        }), 200

    except requests.RequestException as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8094)

Java Health Sidecar (Ranger REST Client)

// RangerHealthController.java
@RestController
public class RangerHealthController {

    @Value("${ranger.admin.url:http://localhost:6080}")
    private String rangerAdminUrl;

    @Value("${ranger.admin.user:admin}")
    private String rangerUser;

    @Value("${ranger.admin.password}")
    private String rangerPassword;

    @Value("${ranger.service.name}")
    private String serviceName;

    private final RestTemplate restTemplate;

    public RangerHealthController() {
        // Configure RestTemplate with basic auth
        this.restTemplate = new RestTemplate();
    }

    @GetMapping("/health/ranger")
    public ResponseEntity<Map<String, Object>> checkHealth() {
        Map<String, Object> status = new LinkedHashMap<>();
        try {
            HttpHeaders headers = new HttpHeaders();
            String credentials = Base64.getEncoder().encodeToString(
                (rangerUser + ":" + rangerPassword).getBytes());
            headers.set("Authorization", "Basic " + credentials);
            HttpEntity<Void> entity = new HttpEntity<>(headers);

            // Check admin server
            ResponseEntity<Map> countResponse = restTemplate.exchange(
                rangerAdminUrl + "/service/public/api/repository/count",
                HttpMethod.GET, entity, Map.class);

            if (!countResponse.getStatusCode().is2xxSuccessful()) {
                status.put("status", "down");
                status.put("reason", "admin_api_unavailable");
                return ResponseEntity.status(503).body(status);
            }

            // Check policy download endpoint
            ResponseEntity<Map> policyResponse = restTemplate.exchange(
                rangerAdminUrl + "/service/plugins/policies/download/" + serviceName,
                HttpMethod.GET, entity, Map.class);

            if (!policyResponse.getStatusCode().is2xxSuccessful()) {
                status.put("status", "degraded");
                status.put("reason", "policy_download_failed");
                return ResponseEntity.status(503).body(status);
            }

            Long updateTimeMs = (Long) policyResponse.getBody().get("policyUpdateTime");
            long ageSeconds = (System.currentTimeMillis() - updateTimeMs) / 1000;
            if (ageSeconds > 300) {
                status.put("status", "degraded");
                status.put("reason", "stale_policy_sync");
                status.put("policy_age_seconds", ageSeconds);
                return ResponseEntity.status(503).body(status);
            }

            status.put("status", "ok");
            status.put("policy_age_seconds", ageSeconds);
            return ResponseEntity.ok(status);

        } catch (Exception e) {
            status.put("status", "down");
            status.put("error", e.getMessage());
            return ResponseEntity.status(503).body(status);
        }
    }
}

Step 2: Configure Vigilmon HTTP Monitor for Ranger

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Ranger health sidecar: https://your-sidecar.example.com/health/ranger
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms (Ranger policy download calls can involve DB reads across many policies)
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the Ranger Admin server directly (via your sidecar proxy, not embedding credentials in Vigilmon):

  • URL: https://your-sidecar.example.com/health/ranger/admin
  • Expected: 200
  • Interval: 2 minutes
  • Alert channel: on-call channel

Vigilmon's multi-region probe consensus prevents false positives from transient network blips between probe nodes and your Ranger admin cluster.

Step 3: Heartbeat Monitoring for Ranger Audit Writers and Policy Sync Jobs

Admin server health checks are necessary — but not sufficient. A Ranger plugin can lose its policy cache silently if the cache TTL expires during a network partition, and audit writers can fail to deliver audit events to Solr or HDFS while the admin server appears healthy.

Vigilmon heartbeat monitors detect silent sync and audit stalls: your policy sync job and audit writer ping Vigilmon after each successful operation cycle. If pings stop, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: ranger-policy-sync-job
  3. Set the expected interval: 5 minutes (adjust to your policy refresh interval)
  4. Set the grace period: 15 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Ranger Policy Sync Job (Python)

# ranger_policy_sync.py
import requests
import os
import time
import logging

logger = logging.getLogger(__name__)

RANGER_URL = os.environ['RANGER_URL']
RANGER_AUTH = (os.environ['RANGER_USER'], os.environ['RANGER_PASS'])
SERVICE_NAME = os.environ['RANGER_SERVICE_NAME']
HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']
SYNC_INTERVAL = int(os.environ.get('SYNC_INTERVAL_SECONDS', '240'))

def sync_policies():
    while True:
        try:
            resp = requests.get(
                f'{RANGER_URL}/service/plugins/policies/download/{SERVICE_NAME}',
                auth=RANGER_AUTH,
                timeout=30
            )
            resp.raise_for_status()
            policies = resp.json()
            apply_policies_to_local_cache(policies)
            logger.info(f"Synced {len(policies.get('policies', []))} policies")

            # Ping Vigilmon after successful sync
            try:
                requests.get(HEARTBEAT_URL, timeout=5)
            except Exception:
                pass

        except Exception as e:
            logger.error(f"Policy sync failed: {e}")
            # Do NOT ping Vigilmon on failure — Vigilmon will alert on missing heartbeat

        time.sleep(SYNC_INTERVAL)

if __name__ == '__main__':
    sync_policies()

Java Ranger Audit Writer with Heartbeat

// RangerAuditWriter.java
@Component
public class RangerAuditWriter {

    @Value("${vigilmon.heartbeat.url}")
    private String heartbeatUrl;

    private final HttpClient httpClient = HttpClient.newHttpClient();
    private int auditsSinceHeartbeat = 0;
    private static final int HEARTBEAT_EVERY = 500;

    @Autowired
    private AuditDestination auditDestination; // Solr, HDFS, or Log4j

    public void writeAuditEvent(AuthzAuditEvent event) {
        try {
            auditDestination.logEvent(event);
            auditsSinceHeartbeat++;

            if (auditsSinceHeartbeat >= HEARTBEAT_EVERY) {
                pingVigilmon();
                auditsSinceHeartbeat = 0;
            }
        } catch (Exception e) {
            // Log but don't suppress the original exception
            throw new RuntimeException("Audit write failed", e);
        }
    }

    private void pingVigilmon() {
        try {
            HttpRequest req = HttpRequest.newBuilder()
                .uri(URI.create(heartbeatUrl))
                .GET().build();
            httpClient.sendAsync(req, HttpResponse.BodyHandlers.discarding());
        } catch (Exception e) {
            // Non-fatal: don't let heartbeat failure interrupt audit writes
        }
    }
}

Scheduled Ranger Health Check Script with Heartbeat

#!/bin/bash
# ranger_heartbeat.sh — run via cron every 4 minutes
# Crontab: */4 * * * * /opt/ranger/ranger_heartbeat.sh

RANGER_URL="${RANGER_URL:-http://localhost:6080}"
RANGER_USER="${RANGER_USER:-admin}"
RANGER_PASS="${RANGER_PASS}"
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"
SERVICE="${RANGER_SERVICE_NAME:-hadoop-service}"

# Try policy download — this exercises DB + policy store
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -u "${RANGER_USER}:${RANGER_PASS}" \
  "${RANGER_URL}/service/plugins/policies/download/${SERVICE}" \
  --max-time 10)

if [ "${STATUS}" = "200" ]; then
  curl -s --max-time 5 "${HEARTBEAT_URL}" > /dev/null
  echo "Ranger healthy — heartbeat sent"
else
  echo "Ranger check failed with HTTP ${STATUS} — heartbeat NOT sent"
  exit 1
fi

Step 4: Alert Routing for Ranger Failures

Ranger failures cascade through the security layer: an admin server crash causes plugins to fall back to their cached policy snapshot, which has a TTL (typically 30 seconds to 30 minutes depending on plugin config). After the TTL, plugins either fail open (allowing all access) or fail closed (denying all access) — both catastrophic. Alert routing should reflect the time-sensitivity of the TTL window.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Ranger health /health/ranger | Slack + PagerDuty | P1 | | Ranger Admin API direct check | Slack | P1 | | Heartbeat: policy sync job | Slack + email | P1 | | Heartbeat: Ranger audit writer | Slack + email | P2 |

Set response time thresholds for early warning:

  • Alert at 5000ms for the health endpoint (slow policy download signals DB pressure or slow policy store queries)
  • Alert at 15000ms for the admin API (slow admin responses signal JVM GC pressure)

For security-critical environments, enable one consecutive failure before alerting (not the default two) — a single missed Ranger check means your security perimeter may already be compromised by the plugin TTL window.

Summary

Ranger failures are invisible to end users until access policies either lock everyone out or fail open entirely. External monitoring catches admin server crashes, policy sync staleness, and audit writer stalls before they become security incidents:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/ranger | Admin server liveness, DB connectivity, policy sync freshness | | HTTP monitor on Ranger Admin API | REST API availability, repository accessibility | | Heartbeat monitor | Policy sync job liveness, audit writer throughput |

Get started free at vigilmon.online — your first Ranger monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles