tutorial

How to Monitor Apache Mesos Cluster Health with Vigilmon

Mesos master failovers and agent disconnections kill scheduled tasks silently. Learn how to monitor Mesos master health, agent availability, and framework registration with Vigilmon HTTP probes and heartbeat monitors.

Apache Mesos abstracts cluster resources across a pool of machines — Mesos masters schedule tasks, agents execute them, and frameworks (Marathon, Chronos, Aurora) submit workloads. When a Mesos master fails over, frameworks briefly lose their resource offers. When an agent disconnects, running tasks are silently killed or left in an orphaned state. When a framework deregisters unexpectedly, scheduled jobs stop running entirely.

Vigilmon gives you external visibility into Mesos cluster health through HTTP probe monitoring on the Mesos REST API and heartbeat monitoring for the frameworks and tasks running on the cluster. This tutorial covers both.

Why Mesos Needs External Monitoring

Mesos exposes a REST API at port 5050 (masters) and 5051 (agents). These endpoints provide rich internal metrics — but only if you're actively polling them. External monitoring with Vigilmon adds:

  • Proactive alerting when Mesos masters fail over or lose quorum
  • Agent availability monitoring to catch disconnected or unhealthy agents before task scheduling fails
  • Framework health checks to detect frameworks that have deregistered or stopped submitting work
  • Heartbeat monitoring for long-running Mesos tasks that should never stop

These layers work together: a Mesos master can be healthy while agents disconnect one by one, shrinking the cluster capacity below the threshold needed to schedule critical workloads.

Step 1: Build Mesos Health Endpoints

Mesos masters expose a REST API you can probe directly. You need a lightweight HTTP sidecar that interprets these responses for Vigilmon.

Node.js Mesos Health Sidecar

// health/mesos.js
const express = require('express');
const axios = require('axios');

const app = express();
const MESOS_MASTER = process.env.MESOS_MASTER || 'http://localhost:5050';
const MIN_ACTIVE_AGENTS = parseInt(process.env.MESOS_MIN_AGENTS || '3');

app.get('/health/mesos', async (req, res) => {
  try {
    // Mesos master /health endpoint returns 200 if master is healthy
    await axios.get(`${MESOS_MASTER}/health`, { timeout: 5000 });

    // Get master state for quorum and agent info
    const state = await axios.get(`${MESOS_MASTER}/state`, { timeout: 10000 });
    const { leader, activated_slaves, deactivated_slaves, frameworks } = state.data;

    if (!leader) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'no_leader_elected',
      });
    }

    if (activated_slaves < MIN_ACTIVE_AGENTS) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'insufficient_active_agents',
        active_agents: activated_slaves,
        min_required: MIN_ACTIVE_AGENTS,
        deactivated_agents: deactivated_slaves,
      });
    }

    return res.status(200).json({
      status: 'ok',
      leader,
      active_agents: activated_slaves,
      deactivated_agents: deactivated_slaves,
      framework_count: frameworks ? frameworks.length : 0,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/mesos/agents', async (req, res) => {
  try {
    const slaves = await axios.get(`${MESOS_MASTER}/slaves`, { timeout: 10000 });
    const agents = slaves.data.slaves || [];

    const active = agents.filter(a => a.active);
    const inactive = agents.filter(a => !a.active);

    if (active.length < MIN_ACTIVE_AGENTS) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'too_many_inactive_agents',
        active_count: active.length,
        inactive_count: inactive.length,
        inactive_agents: inactive.map(a => ({ id: a.id, hostname: a.hostname })),
      });
    }

    return res.status(200).json({
      status: 'ok',
      active_agents: active.length,
      inactive_agents: inactive.length,
      total_cpus: active.reduce((sum, a) => sum + a.resources.cpus, 0),
      total_mem_mb: active.reduce((sum, a) => sum + a.resources.mem, 0),
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/mesos/frameworks', async (req, res) => {
  const requiredFrameworks = (process.env.MESOS_REQUIRED_FRAMEWORKS || '').split(',').filter(Boolean);

  try {
    const state = await axios.get(`${MESOS_MASTER}/state`, { timeout: 10000 });
    const activeFrameworks = (state.data.frameworks || []).map(f => f.name);
    const unregistered = requiredFrameworks.filter(f => !activeFrameworks.includes(f));

    if (unregistered.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'required_frameworks_missing',
        missing: unregistered,
        registered: activeFrameworks,
      });
    }

    return res.status(200).json({
      status: 'ok',
      registered_frameworks: activeFrameworks,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3008);

Python Health Sidecar

# health_mesos.py
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)
MESOS_MASTER = os.environ.get('MESOS_MASTER', 'http://localhost:5050')
MIN_ACTIVE_AGENTS = int(os.environ.get('MESOS_MIN_AGENTS', '3'))

@app.route('/health/mesos')
def mesos_health():
    try:
        requests.get(f'{MESOS_MASTER}/health', timeout=5).raise_for_status()
        state = requests.get(f'{MESOS_MASTER}/state', timeout=10).json()

        if not state.get('leader'):
            return jsonify({'status': 'degraded', 'reason': 'no_leader_elected'}), 503

        active = state.get('activated_slaves', 0)
        if active < MIN_ACTIVE_AGENTS:
            return jsonify({
                'status': 'degraded',
                'reason': 'insufficient_active_agents',
                'active_agents': active,
                'min_required': MIN_ACTIVE_AGENTS,
            }), 503

        return jsonify({
            'status': 'ok',
            'leader': state.get('leader'),
            'active_agents': active,
            'framework_count': len(state.get('frameworks', [])),
        })
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3008)

Direct Mesos API Probing

The Mesos master REST API can be probed directly without a sidecar:

# Master health check (returns 200 if healthy)
curl http://mesos-master:5050/health

# Master state (includes leader, agents, frameworks)
curl http://mesos-master:5050/state | jq '{leader, activated_slaves, deactivated_slaves}'

# Agent health (individual agent port 5051)
curl http://mesos-agent-01:5051/health

Point Vigilmon directly at http://mesos-master:5050/health for a simple liveness check, and add the sidecar for richer semantic checks.

Step 2: Configure Vigilmon HTTP Monitor for Mesos

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Mesos health endpoint: https://your-cluster.example.com/health/mesos
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add additional monitors for agent and framework health:

  • URL: https://your-cluster.example.com/health/mesos/agents

  • Expected: 200, body contains "status":"ok"

  • Interval: 2 minutes

  • Alert channel: infrastructure Slack channel

  • URL: https://your-cluster.example.com/health/mesos/frameworks

  • Expected: 200, body contains "status":"ok"

  • Interval: 2 minutes

  • Alert channel: Slack + email

Vigilmon's multi-region probe consensus prevents transient Mesos master failover events (which typically resolve in seconds) from triggering false-positive P1 alerts.

Step 3: Heartbeat Monitoring for Mesos Tasks and Frameworks

Mesos master health doesn't tell you whether your long-running tasks are actually doing work. A Marathon application can be running (from Marathon's perspective) while the underlying process is stuck, deadlocked, or silently throwing exceptions.

Vigilmon heartbeat monitors detect these stalls: your Mesos task sends a periodic ping to Vigilmon. If pings stop, Vigilmon alerts.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: mesos-worker-task-liveness
  3. Set the expected interval: 5 minutes
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a Mesos Task (Python)

# mesos_task_worker.py
import os, time, requests

HEARTBEAT_URL = os.environ.get('VIGILMON_HEARTBEAT_URL')
WORK_INTERVAL_SECONDS = int(os.environ.get('WORK_INTERVAL', '60'))

def do_work():
    # ... actual task work ...
    pass

def ping_vigilmon():
    if HEARTBEAT_URL:
        try:
            requests.get(HEARTBEAT_URL, timeout=5)
        except Exception:
            pass

while True:
    try:
        do_work()
        ping_vigilmon()
    except Exception as e:
        print(f'Task error: {e}')
        # No heartbeat on error — Vigilmon alerts after grace period
    time.sleep(WORK_INTERVAL_SECONDS)

Marathon Application with Vigilmon Heartbeat

Add the heartbeat URL as an environment variable in your Marathon app definition:

{
  "id": "/my-worker/processor",
  "instances": 3,
  "env": {
    "VIGILMON_HEARTBEAT_URL": "https://vigilmon.online/heartbeat/abc123xyz"
  },
  "healthChecks": [
    {
      "protocol": "HTTP",
      "path": "/health",
      "portIndex": 0,
      "intervalSeconds": 30,
      "timeoutSeconds": 5,
      "maxConsecutiveFailures": 3
    }
  ]
}

Combine Marathon's internal health checks (which restart unhealthy tasks) with Vigilmon heartbeats (which alert you when the task is running but not making progress).

Chronos Job with Heartbeat

{
  "name": "orc-batch-processor",
  "schedule": "R/2016-01-01T00:00:00Z/PT30M",
  "command": "python /opt/jobs/process.py && curl -sf $VIGILMON_HEARTBEAT_URL",
  "environmentVariables": [
    {
      "name": "VIGILMON_HEARTBEAT_URL",
      "value": "https://vigilmon.online/heartbeat/abc123xyz"
    }
  ]
}

The && ensures Vigilmon only receives a heartbeat if the job exits successfully.

Step 4: Alert Routing for Mesos Cluster Failures

Mesos failures cascade: a lost master failover causes frameworks to briefly stop scheduling, which causes tasks to pile up in the queue. Agent disconnections reduce cluster capacity and cause scheduling failures for resource-heavy tasks.

| Monitor | Alert Channel | Priority | |---|---|---| | Mesos master /health/mesos | Slack + PagerDuty | P1 | | Agent availability /health/mesos/agents | Slack + PagerDuty | P1 | | Framework health /health/mesos/frameworks | Slack + email | P2 | | Heartbeat: long-running tasks | Slack | P2 | | Heartbeat: Chronos batch jobs | Email | P3 |

Set response time thresholds as early warning signals:

  • Alert at 5000ms for the master health endpoint (slow Mesos state API responses signal master overload)
  • Alert at 8000ms for agent checks (slow slave responses signal network or node issues)

For clusters running production workloads, add a Vigilmon status page that shows master health, agent count, and active framework count — this gives operations teams instant cluster state visibility during incidents without needing ZooKeeper or Mesos UI access.

Summary

Mesos failures are scheduling failures — tasks don't start, running tasks get killed, and frameworks lose their resource offers without surfacing explicit errors to operators. External monitoring catches degradation before workloads fall behind:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/mesos | Master liveness, leader election, agent count thresholds | | HTTP monitor on /health/mesos/agents | Agent availability, inactive agent detection | | HTTP monitor on /health/mesos/frameworks | Required framework registration status | | Heartbeat monitor | Long-running task liveness, Chronos job completion |

Get started free at vigilmon.online — your first Mesos cluster monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles