How to Monitor Apache Ambari Cluster Management Health with Vigilmon

Apache Ambari is the management plane for your entire Hadoop cluster — it provisions services, pushes configuration changes, monitors host health, and runs service checks that confirm HDFS, YARN, Hive, and HBase are functioning correctly. When the Ambari server goes down, you lose the ability to deploy configs, scale services, or respond to cluster degradation. When the Ambari agents on individual nodes stop heartbeating, those hosts fall off the management radar entirely — and any configuration drift or service failure on those nodes goes undetected.

Vigilmon gives you external visibility into Ambari management plane health through HTTP probe monitoring (via Ambari's REST API) and heartbeat monitoring for Ambari agent check-in and service check jobs. This tutorial covers both.

Why Ambari Needs External Monitoring

Ambari's own dashboard shows cluster health — but only when the Ambari server itself is accessible. External monitoring with Vigilmon adds a layer that works even when your cluster's internal tooling is broken:

• Proactive alerting when the Ambari server REST API becomes unreachable, signaling a server crash, database connection failure, or network partition
• Cluster summary checks that confirm all expected hosts and services are registered and reporting
• Heartbeat monitoring for Ambari's internal service check jobs, which run periodic smoke tests against HDFS, YARN, and other services
• External availability — confirming the Ambari API is reachable from the same network segment as your automation scripts and CI pipelines

Step 1: Build an Ambari Health Endpoint

Ambari exposes a full REST API on port 8080 (HTTP) or 8443 (HTTPS). The API covers cluster state, host health, and service status in a single unified interface.

Using the Ambari REST API Directly

# Ambari server basic health check
curl -i -u admin:admin http://ambari-server.example.com:8080/api/v1/clusters

# List all cluster hosts and their states
curl -u admin:admin \
  "http://ambari-server.example.com:8080/api/v1/clusters/MyCluster/hosts" \
  | python3 -m json.tool

# Check host heartbeat state — HEALTHY means agent is checking in
curl -u admin:admin \
  "http://ambari-server.example.com:8080/api/v1/clusters/MyCluster/hosts?fields=Hosts/host_state"

# Check service HDFS is in STARTED state
curl -u admin:admin \
  "http://ambari-server.example.com:8080/api/v1/clusters/MyCluster/services/HDFS?fields=ServiceInfo/state"

A healthy Ambari response to /api/v1/clusters returns 200 with a JSON array of cluster objects. An unreachable server returns a connection timeout or 5xx.

Python Health Sidecar

# ambari_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

AMBARI_URL = os.environ.get('AMBARI_URL', 'http://localhost:8080')
AMBARI_USER = os.environ.get('AMBARI_USER', 'admin')
AMBARI_PASS = os.environ.get('AMBARI_PASS', '')
CLUSTER_NAME = os.environ.get('AMBARI_CLUSTER', 'MyCluster')
CRITICAL_SERVICES = os.environ.get('CRITICAL_SERVICES', 'HDFS,YARN,ZOOKEEPER').split(',')

auth = (AMBARI_USER, AMBARI_PASS)

@app.route('/health/ambari')
def ambari_health():
    try:
        # Check Ambari server is reachable and DB is connected
        clusters_resp = requests.get(
            f'{AMBARI_URL}/api/v1/clusters',
            auth=auth, timeout=5
        )
        if clusters_resp.status_code != 200:
            return jsonify({
                'status': 'down',
                'reason': 'clusters_api_error',
                'http_status': clusters_resp.status_code,
            }), 503

        # Check for unhealthy hosts (HEARTBEAT_LOST means agent stopped checking in)
        hosts_resp = requests.get(
            f'{AMBARI_URL}/api/v1/clusters/{CLUSTER_NAME}/hosts'
            '?fields=Hosts/host_state',
            auth=auth, timeout=8
        )
        if hosts_resp.status_code == 200:
            hosts = hosts_resp.json().get('items', [])
            lost = [
                h['Hosts']['host_name']
                for h in hosts
                if h.get('Hosts', {}).get('host_state') == 'HEARTBEAT_LOST'
            ]
            if lost:
                return jsonify({
                    'status': 'degraded',
                    'reason': 'hosts_heartbeat_lost',
                    'lost_hosts': lost,
                }), 503

        # Check critical services are STARTED
        degraded_services = []
        for svc in CRITICAL_SERVICES:
            svc_resp = requests.get(
                f'{AMBARI_URL}/api/v1/clusters/{CLUSTER_NAME}/services/{svc}'
                '?fields=ServiceInfo/state',
                auth=auth, timeout=5
            )
            if svc_resp.status_code == 200:
                state = svc_resp.json().get('ServiceInfo', {}).get('state')
                if state != 'STARTED':
                    degraded_services.append({'service': svc, 'state': state})

        if degraded_services:
            return jsonify({
                'status': 'degraded',
                'reason': 'services_not_started',
                'services': degraded_services,
            }), 503

        return jsonify({
            'status': 'ok',
            'cluster': CLUSTER_NAME,
            'host_count': len(hosts_resp.json().get('items', [])),
        }), 200

    except requests.RequestException as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8091)

Java Health Sidecar

// AmbariHealthController.java
@RestController
public class AmbariHealthController {

    @Value("${ambari.url:http://localhost:8080}")
    private String ambariUrl;

    @Value("${ambari.user:admin}")
    private String ambariUser;

    @Value("${ambari.password}")
    private String ambariPassword;

    @Value("${ambari.cluster}")
    private String clusterName;

    private final RestTemplate restTemplate = new RestTemplate();

    @GetMapping("/health/ambari")
    public ResponseEntity<Map<String, Object>> checkHealth() {
        Map<String, Object> result = new LinkedHashMap<>();
        try {
            HttpHeaders headers = new HttpHeaders();
            String creds = Base64.getEncoder().encodeToString(
                (ambariUser + ":" + ambariPassword).getBytes());
            headers.set("Authorization", "Basic " + creds);
            HttpEntity<Void> entity = new HttpEntity<>(headers);

            // Check clusters API
            ResponseEntity<Map> clustersResp = restTemplate.exchange(
                ambariUrl + "/api/v1/clusters",
                HttpMethod.GET, entity, Map.class);

            if (!clustersResp.getStatusCode().is2xxSuccessful()) {
                result.put("status", "down");
                result.put("reason", "clusters_api_unavailable");
                return ResponseEntity.status(503).body(result);
            }

            // Check for HEARTBEAT_LOST hosts
            ResponseEntity<Map> hostsResp = restTemplate.exchange(
                ambariUrl + "/api/v1/clusters/" + clusterName
                    + "/hosts?fields=Hosts/host_state",
                HttpMethod.GET, entity, Map.class);

            List<Map<?, ?>> items = (List<Map<?, ?>>) hostsResp.getBody().get("items");
            long lostCount = items.stream()
                .map(h -> ((Map<?, ?>) h.get("Hosts")).get("host_state"))
                .filter("HEARTBEAT_LOST"::equals)
                .count();

            if (lostCount > 0) {
                result.put("status", "degraded");
                result.put("lost_hosts", lostCount);
                return ResponseEntity.status(503).body(result);
            }

            result.put("status", "ok");
            result.put("host_count", items.size());
            return ResponseEntity.ok(result);

        } catch (Exception e) {
            result.put("status", "down");
            result.put("error", e.getMessage());
            return ResponseEntity.status(503).body(result);
        }
    }
}

Step 2: Configure Vigilmon HTTP Monitor for Ambari

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your Ambari health sidecar: https://your-sidecar.example.com/health/ambari
4. Set the check interval to 2 minutes
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 10000ms (Ambari API calls can span multiple DB queries)
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Add a second lightweight monitor for the Ambari server directly:

• URL: http://ambari-server.example.com:8080/api/v1/clusters (via sidecar proxy with credentials)
• Expected: 200
• Interval: 3 minutes

Step 3: Heartbeat Monitoring for Ambari Agent Check-Ins and Service Jobs

Ambari agents on each cluster host check in with the Ambari server every 10 seconds by default. If an agent loses connectivity, its host moves to HEARTBEAT_LOST state — a condition the sidecar above catches. But you also want to monitor Ambari's own scheduled service check jobs, which run smoke tests against each Hadoop service.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: ambari-service-checks
3. Set the expected interval: 15 minutes
4. Set the grace period: 30 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Ambari Service Check Script

# ambari_service_check.py
import requests
import os
import time
import logging

logger = logging.getLogger(__name__)

AMBARI_URL = os.environ['AMBARI_URL']
AUTH = (os.environ['AMBARI_USER'], os.environ['AMBARI_PASS'])
CLUSTER = os.environ['AMBARI_CLUSTER']
HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']

def run_service_check():
    """Trigger an Ambari service check and wait for completion."""
    body = {
        "RequestInfo": {
            "command": "HDFS_SERVICE_CHECK",
            "context": "Service Check - Vigilmon"
        },
        "Requests/resource_filters": [{"service_name": "HDFS"}]
    }
    try:
        resp = requests.post(
            f'{AMBARI_URL}/api/v1/clusters/{CLUSTER}/requests',
            json=body,
            auth=AUTH,
            timeout=10
        )
        resp.raise_for_status()
        request_id = resp.json()['Requests']['id']
        logger.info(f"Service check started, request_id={request_id}")

        # Poll until completed
        for _ in range(30):
            time.sleep(10)
            status_resp = requests.get(
                f'{AMBARI_URL}/api/v1/clusters/{CLUSTER}/requests/{request_id}'
                '?fields=Requests/request_status',
                auth=AUTH, timeout=5
            )
            state = status_resp.json()['Requests']['request_status']
            if state == 'Completed':
                # Send heartbeat — service check succeeded
                requests.get(HEARTBEAT_URL, timeout=5)
                logger.info("Service check complete — heartbeat sent")
                return
            elif state in ('Failed', 'Aborted'):
                logger.error(f"Service check failed with state: {state}")
                return

    except Exception as e:
        logger.error(f"Ambari service check error: {e}")

if __name__ == '__main__':
    while True:
        run_service_check()
        time.sleep(900)  # Run every 15 minutes

Step 4: Alert Routing for Ambari Failures

| Monitor | Alert Channel | Priority | |---|---|---| | Ambari health /health/ambari | Slack + PagerDuty | P1 | | Ambari clusters API direct | Slack | P1 | | Heartbeat: service check jobs | Slack + email | P2 |

Set response time thresholds:

• Alert at 8000ms for the health endpoint (slow Ambari API responses signal DB or JVM pressure)
• Alert at 20000ms for service check polling

Ambari server failures mean you cannot deploy config changes, run maintenance operations, or detect service degradation across the cluster. Treat Ambari server availability as P1.

Summary

Ambari is your cluster management control plane. When it goes dark, you lose operational visibility and the ability to respond to cluster events:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/ambari | Server liveness, host heartbeat state, service state | | HTTP monitor on Ambari clusters API | API availability, DB connectivity | | Heartbeat monitor | Service check job liveness, smoke test execution |

Get started free at vigilmon.online — your first Ambari monitor is running in under two minutes.