tutorial

How to Monitor Apache Knox Gateway Health and API Availability with Vigilmon

Knox gateway failures silently block all Hadoop REST API access. Learn how to monitor Knox topology endpoints, token service health, and gateway process liveness with Vigilmon HTTP probes and heartbeat monitors.

Apache Knox is the security gateway that proxies every REST API request to your Hadoop cluster — HDFS, YARN, Hive, HBase, Oozie, and more — enforcing authentication and authorization at a single entry point. When Knox goes down, every downstream REST client loses access instantly. When Knox is degraded — slow TLS handshakes, expired token caches, misconfigured topology descriptors — failures are often silent, appearing as timeouts or opaque 401s to upstream services rather than clear gateway errors.

Vigilmon gives you external visibility into Knox gateway health through HTTP probe monitoring (via Knox's built-in admin API and topology health endpoints) and heartbeat monitoring for your Knox token renewal and background processes. This tutorial covers both.

Why Knox Needs External Monitoring

Knox's internal health pages are useful for operators logged into the cluster, but they don't alert you proactively. External monitoring with Vigilmon adds:

  • Proactive alerting when the Knox gateway process becomes unreachable — the fastest possible signal before users start reporting API failures
  • Topology health checks that confirm individual service proxies (HDFS, YARN, Hive) are correctly configured and responding
  • Heartbeat monitoring for Knox token renewal jobs and background authentication cache refresh processes
  • Availability checking from outside the cluster — confirming Knox is reachable from the same network path as your REST API clients

These layers work together: the Knox process can appear healthy while a specific topology descriptor has a misconfigured backend URL, causing all requests to that service to silently return 502.

Step 1: Build a Knox Health Endpoint

Knox exposes an admin API on port 8443 (HTTPS) and individual topology endpoints for each configured service cluster. Use these directly or wrap them in a lightweight health sidecar.

Using the Knox Admin API Directly

# Knox version endpoint — basic liveness check (no auth required by default)
curl -ik https://knox-gateway.example.com:8443/gateway/admin/api/v1/version

# List all configured topologies
curl -ik -u admin:admin-password \
  https://knox-gateway.example.com:8443/gateway/admin/api/v1/topologies

# Check a specific topology is accessible
curl -ik https://knox-gateway.example.com:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS \
  --user hdfs-user:password

The /version endpoint is ideal for Vigilmon monitoring — it responds without heavy backend dependencies and confirms the Knox process is alive and serving HTTPS traffic.

Python Health Sidecar

# knox_health.py
from flask import Flask, jsonify
import requests
import os
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

app = Flask(__name__)

KNOX_URL = os.environ.get('KNOX_URL', 'https://localhost:8443')
KNOX_ADMIN_USER = os.environ.get('KNOX_ADMIN_USER', 'admin')
KNOX_ADMIN_PASS = os.environ.get('KNOX_ADMIN_PASS', '')
TOPOLOGY_NAME = os.environ.get('KNOX_TOPOLOGY', 'sandbox')
VERIFY_SSL = os.environ.get('KNOX_VERIFY_SSL', 'false').lower() == 'true'

@app.route('/health/knox')
def knox_health():
    try:
        # Check Knox is alive via version endpoint
        version_resp = requests.get(
            f'{KNOX_URL}/gateway/admin/api/v1/version',
            verify=VERIFY_SSL,
            timeout=5
        )
        if version_resp.status_code != 200:
            return jsonify({
                'status': 'down',
                'reason': 'version_endpoint_error',
                'http_status': version_resp.status_code,
            }), 503

        # Check topology list is accessible (validates DB/descriptor store)
        topo_resp = requests.get(
            f'{KNOX_URL}/gateway/admin/api/v1/topologies',
            auth=(KNOX_ADMIN_USER, KNOX_ADMIN_PASS),
            verify=VERIFY_SSL,
            timeout=8
        )
        if topo_resp.status_code != 200:
            return jsonify({
                'status': 'degraded',
                'reason': 'topology_api_error',
                'http_status': topo_resp.status_code,
            }), 503

        topo_data = topo_resp.json()
        topo_count = len(topo_data.get('topologies', {}).get('topology', []))

        return jsonify({
            'status': 'ok',
            'topology_count': topo_count,
            'version': version_resp.json().get('ServerVersion', {}).get('version', 'unknown'),
        }), 200

    except requests.RequestException as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8090)

Bash Health Check Script

#!/bin/bash
# knox_health_check.sh
KNOX_URL="${KNOX_URL:-https://localhost:8443}"
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"

STATUS=$(curl -sk -o /dev/null -w "%{http_code}" \
  "${KNOX_URL}/gateway/admin/api/v1/version" \
  --max-time 10)

if [ "${STATUS}" = "200" ]; then
  curl -s --max-time 5 "${HEARTBEAT_URL}" > /dev/null
  echo "Knox healthy — heartbeat sent"
else
  echo "Knox check failed with HTTP ${STATUS} — heartbeat NOT sent"
  exit 1
fi

Step 2: Configure Vigilmon HTTP Monitor for Knox

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Knox health sidecar: https://your-sidecar.example.com/health/knox
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the Knox version endpoint directly (no credentials required):

  • URL: https://knox-gateway.example.com:8443/gateway/admin/api/v1/version
  • Expected: 200
  • Interval: 2 minutes
  • Alert channel: on-call channel

Vigilmon probes Knox from multiple geographic regions, so a single failed check doesn't trigger a false alarm — you need two consecutive failures before the alert fires.

Step 3: Heartbeat Monitoring for Knox Token and Session Jobs

Knox token service maintains short-lived tokens for stateless REST clients. If the token refresh background job stalls, existing tokens expire and clients begin seeing 401s across all services simultaneously. Vigilmon heartbeat monitors detect this class of silent failure.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: knox-token-refresh-job
  3. Set the expected interval: 10 minutes (adjust to your token TTL)
  4. Set the grace period: 20 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Token Refresh Job

# knox_token_refresh.py
import requests
import os
import time
import logging

logger = logging.getLogger(__name__)

KNOX_URL = os.environ['KNOX_URL']
TOKEN_ENDPOINT = f"{KNOX_URL}/gateway/knoxtoken/api/v1/token"
HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']
REFRESH_INTERVAL = int(os.environ.get('REFRESH_INTERVAL_SECONDS', '600'))

def refresh_tokens():
    while True:
        try:
            resp = requests.get(
                TOKEN_ENDPOINT,
                auth=(os.environ['KNOX_USER'], os.environ['KNOX_PASS']),
                verify=False,
                timeout=15
            )
            resp.raise_for_status()
            token_data = resp.json()
            logger.info(f"Token refreshed, expires: {token_data.get('expires_in')}")

            # Ping Vigilmon only on success
            try:
                requests.get(HEARTBEAT_URL, timeout=5)
            except Exception:
                pass

        except Exception as e:
            logger.error(f"Token refresh failed: {e}")
            # Do NOT ping on failure — Vigilmon alerts on missing heartbeat

        time.sleep(REFRESH_INTERVAL)

if __name__ == '__main__':
    refresh_tokens()

Step 4: Alert Routing for Knox Failures

Knox is a single point of failure for all Hadoop REST API access. Treat every Knox alert as P1.

| Monitor | Alert Channel | Priority | |---|---|---| | Knox health /health/knox | Slack + PagerDuty | P1 | | Knox version endpoint direct | Slack | P1 | | Heartbeat: token refresh job | Slack + email | P1 |

Set response time thresholds:

  • Alert at 3000ms (slow Knox TLS handshakes often precede full gateway failure)
  • Alert at 8000ms for topology API checks (slow topology queries signal descriptor store pressure)

For Knox, enable one consecutive failure before alerting — Knox outages cascade immediately to all REST API clients.

Summary

Knox is the single security entry point for your entire Hadoop REST API surface. External monitoring catches process crashes, TLS failures, topology misconfigurations, and token service stalls before they become cluster-wide outages:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/knox | Gateway liveness, topology count, admin API health | | HTTP monitor on Knox version endpoint | Process availability, TLS termination | | Heartbeat monitor | Token refresh job liveness, session cache freshness |

Get started free at vigilmon.online — your first Knox monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles