tutorial

How to Monitor Apache Eagle Uptime and Health with Vigilmon

Apache Eagle is a distributed monitoring and alerting framework for big data platforms — but Eagle itself can go down, leaving your Hadoop and Spark clusters unmonitored without any alert. Here's how to monitor Eagle with Vigilmon.

Apache Eagle is a distributed real-time monitoring and alerting framework designed to protect big data platforms such as Hadoop, HBase, and Spark from security threats, performance anomalies, and compliance violations. Eagle ingests audit logs, JMX metrics, and topology events from these systems and runs policy-based alert rules in near real-time using a stream processing engine. The critical problem: if Eagle itself goes down, your entire big data monitoring plane goes dark — Hadoop clusters run without security alerting, anomalous access patterns go undetected, and compliance audit trails stop recording — all silently.

Vigilmon gives you external visibility into Eagle availability so that your monitoring infrastructure is itself monitored. This tutorial walks through HTTP probe and heartbeat monitoring for Apache Eagle.

Why Apache Eagle Monitoring Matters

Eagle acts as the security and observability layer for your data platform — losing Eagle means losing all of the following simultaneously:

  • Real-time security alerting stops — HDFS privilege escalation, unauthorized HBase access, and Hive data exfiltration attempts go undetected
  • Compliance audit trails are interrupted — GDPR and HIPAA audit log ingestion stops recording, creating compliance gaps
  • JMX metric collection from Hadoop NameNode, DataNode, HBase RegionServer, and Spark masters ceases — you lose cluster health visibility
  • Alert policy enforcement is suspended — even if the underlying platforms continue to run, no alerts fire on policy violations
  • Eagle's web UI and REST API may become unreachable while backend processors continue partially — users cannot create or modify policies, and downstream consumers of Eagle's alert API receive no events

External monitoring through Vigilmon verifies whether Eagle's API and alert pipeline are operational — independently of the systems Eagle itself monitors.

Step 1: Build an Eagle Health Endpoint

Apache Eagle exposes a REST API on port 9090 by default. Test the native health endpoint:

curl -i http://your-eagle-host:9090/eagle-service/rest/api/health
# HTTP/1.1 200 OK

For a deeper check that verifies alert processing health, build a sidecar that queries Eagle's service status and recent alert activity:

Node.js Sidecar Health Endpoint

// eagle-health.js — checks Eagle API and alert pipeline activity
const express = require('express');
const axios = require('axios');

const app = express();
const EAGLE_API = process.env.EAGLE_API_URL || 'http://localhost:9090/eagle-service/rest/api';
const EAGLE_USER = process.env.EAGLE_USER || 'admin';
const EAGLE_PASS = process.env.EAGLE_PASSWORD || 'secret';

const AUTH = Buffer.from(`${EAGLE_USER}:${EAGLE_PASS}`).toString('base64');

app.get('/health/eagle', async (req, res) => {
  try {
    // Check service list — a healthy Eagle instance reports its registered services
    const servicesRes = await axios.get(`${EAGLE_API}/services`, {
      headers: { Authorization: `Basic ${AUTH}` },
      timeout: 10000,
    });

    const services = servicesRes.data || [];
    const unhealthy = services.filter(s => s.status !== 'GREEN');

    if (unhealthy.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'unhealthy_services',
        unhealthy: unhealthy.map(s => ({ name: s.name, status: s.status })),
      });
    }

    return res.status(200).json({
      status: 'ok',
      services: services.length,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Python Sidecar Health Endpoint

# eagle_health.py — FastAPI sidecar for Apache Eagle health
import os
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from base64 import b64encode

app = FastAPI()
EAGLE_API = os.environ.get("EAGLE_API_URL", "http://localhost:9090/eagle-service/rest/api")
EAGLE_USER = os.environ.get("EAGLE_USER", "admin")
EAGLE_PASS = os.environ.get("EAGLE_PASSWORD", "secret")
AUTH_HEADER = "Basic " + b64encode(f"{EAGLE_USER}:{EAGLE_PASS}".encode()).decode()

@app.get("/health/eagle")
async def eagle_health():
    async with httpx.AsyncClient(timeout=10.0) as client:
        # Check services status
        try:
            r = await client.get(
                f"{EAGLE_API}/services",
                headers={"Authorization": AUTH_HEADER},
            )
            r.raise_for_status()
            services = r.json()
        except Exception as e:
            return JSONResponse(status_code=503,
                                content={"status": "down", "check": "services", "error": str(e)})

        unhealthy = [s for s in services if s.get("status") != "GREEN"]
        if unhealthy:
            return JSONResponse(status_code=503, content={
                "status": "degraded",
                "reason": "unhealthy_services",
                "unhealthy": [{"name": s["name"], "status": s["status"]} for s in unhealthy],
            })

        # Check recent alert events — verify the alert pipeline is processing
        try:
            r = await client.get(
                f"{EAGLE_API}/alerts",
                headers={"Authorization": AUTH_HEADER},
                params={"pageSize": 1, "pageIndex": 0},
            )
            r.raise_for_status()
        except Exception as e:
            return JSONResponse(status_code=503,
                                content={"status": "degraded", "check": "alerts_api", "error": str(e)})

    return {"status": "ok", "services": len(services)}

Deploy and verify:

curl -i https://your-app.example.com/health/eagle
# HTTP/1.1 200 OK
# {"status":"ok","services":4}

Step 2: Configure a Vigilmon HTTP Monitor for Eagle

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-app.example.com/health/eagle
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms
  6. Under Alert channels, assign your Slack or email channel
  7. Save the monitor

For a lighter-weight check against the native Eagle REST API:

  1. Create a second monitor pointing to http://your-eagle-host:9090/eagle-service/rest/api/health
  2. Set the expected status code to 200
  3. Set the response time threshold to 3000ms

Failure Coverage

| Failure Mode | Big Data Platform Monitor | Vigilmon | |---|---|---| | Eagle JVM crash | ✗ | ✓ | | Eagle service degraded (unhealthy component) | ✗ | ✓ | | Alert pipeline stalled (no events processed) | ✗ | ✓ (via heartbeat) | | REST API unreachable | ✗ | ✓ | | Policy evaluation paused | ✗ | ✓ (via heartbeat) |

Step 3: Heartbeat Monitoring for Eagle Alert Pipeline Liveness

Eagle's alert pipeline — the stream processor that evaluates security policies against audit log events — can stall independently of the API server being up. The REST API may respond 200 while the underlying Kafka consumer group has fallen behind or the alert evaluation engine is stuck. Vigilmon heartbeat monitoring detects this silent stall.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: eagle-alert-pipeline
  3. Set the expected interval: 5 minutes
  4. Set the grace period: 15 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Eagle's Alert Sink

Create a custom Eagle alert sink that pings Vigilmon whenever Eagle successfully evaluates and routes an alert:

// VigilmonAlertSink.java — Eagle AlertSink that pings Vigilmon
package org.apache.eagle.alert.engine.sink;

import org.apache.eagle.alert.engine.coordinator.StreamDefinition;
import org.apache.eagle.alert.engine.model.AlertStreamEvent;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.atomic.AtomicLong;

public class VigilmonAlertSink {
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final AtomicLong evaluationCount = new AtomicLong(0);

    public void onAlert(AlertStreamEvent event) {
        // Ping every 50 evaluations (alert or no-alert — we care about evaluation activity)
        long count = evaluationCount.incrementAndGet();
        if (count % 50 == 0) {
            pingVigilmon();
        }
    }

    private void pingVigilmon() {
        if (heartbeatUrl == null || heartbeatUrl.isEmpty()) return;
        try {
            HttpURLConnection conn = (HttpURLConnection) new URL(heartbeatUrl).openConnection();
            conn.setRequestMethod("GET");
            conn.setConnectTimeout(3000);
            conn.setReadTimeout(3000);
            conn.getResponseCode();
            conn.disconnect();
        } catch (Exception ignored) {}
    }
}

Python Integration via Eagle REST Polling

For environments where you cannot modify Eagle's alert sink, poll the alerts API from an external process:

# eagle_heartbeat_poller.py — polls Eagle alerts API and pings Vigilmon
import os, time, requests
from base64 import b64encode

EAGLE_API = os.environ["EAGLE_API_URL"]
AUTH_HEADER = "Basic " + b64encode(
    f"{os.environ['EAGLE_USER']}:{os.environ['EAGLE_PASSWORD']}".encode()
).decode()
HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]
LAST_SEEN_COUNT = 0

def poll_and_ping():
    global LAST_SEEN_COUNT
    r = requests.get(
        f"{EAGLE_API}/alerts",
        headers={"Authorization": AUTH_HEADER},
        params={"pageSize": 1, "pageIndex": 0},
        timeout=10,
    )
    r.raise_for_status()
    # If Eagle is alive and API is responsive, ping Vigilmon
    requests.get(HEARTBEAT_URL, timeout=3)

if __name__ == "__main__":
    while True:
        try:
            poll_and_ping()
        except Exception as e:
            print(f"Poll failed: {e}")
        time.sleep(240)  # ping every 4 minutes; heartbeat interval is 5 minutes

Step 4: Monitor Multiple Eagle Components

A production Eagle deployment includes multiple services. Monitor each independently:

[eagle-api]          :9090/eagle-service/rest/api/health
[eagle-health]       /health/eagle (sidecar)
[eagle-kafka]        :9092 (TCP monitor)
[eagle-zookeeper]    :2181 (TCP monitor)
[eagle-pipeline]     (heartbeat monitor)

Name monitors using the bracket convention so alert notifications immediately identify the failing component.

Group all Eagle monitors into a Vigilmon Status Page and share it with your data platform and security operations teams.

Step 5: Alert Routing for Eagle Outages

Eagle outages are second-order threats — when Eagle goes down, your data platform becomes an unmonitored environment where security incidents, compliance violations, and performance anomalies can occur without detection.

Configure alert channels in Vigilmon:

  1. Eagle API monitor → Slack + PagerDuty (P1 — monitoring plane offline)
  2. Eagle health sidecar → Slack + PagerDuty (P1 — service degraded)
  3. Alert pipeline heartbeat → Slack + PagerDuty (P1 — evaluation stalled)
  4. Kafka/ZooKeeper TCP monitors → Slack (P2 — infrastructure degraded)

Set response time thresholds carefully:

  • Alert at 5000ms on the health sidecar — a slow response indicates the Eagle JVM is GC-paused or the Kafka consumer is overwhelmed
  • Alert at 2000ms on the native health endpoint — the health check should be near-instant; slowness here signals the web tier is struggling

Summary

Apache Eagle is your data platform's security and observability layer — when Eagle goes down, you are flying blind across Hadoop, HBase, and Spark. Vigilmon monitors Eagle itself so you know the moment your monitoring infrastructure stops working.

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /rest/api/health | Eagle API reachability | | HTTP monitor on /health/eagle | Service health, degraded components | | Heartbeat monitor | Alert pipeline evaluation liveness | | TCP monitors on Kafka/ZooKeeper | Infrastructure connectivity |

Get started free at vigilmon.online — your first Eagle monitor takes under two minutes to configure.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles