Monitoring MITRE Caldera with Vigilmon

MITRE Caldera lets your security team run controlled adversary emulation exercises based on MITRE ATT&CK — deploying agents, executing attack techniques, and measuring which detections fire. But Caldera is only useful when it's running: if the C2 server goes down mid-operation, agents lose contact, operations stall, and your red team exercises produce incomplete data. Vigilmon monitors your Caldera deployment so infrastructure failures don't corrupt your security validation data.

What You'll Set Up

• Caldera server health monitor (port 8888 — REST API and web UI)
• Agent beacon health via a cron heartbeat
• C2 communication channel monitoring (HTTP/HTTPS beacon endpoint)
• Plugin health verification at startup
• REST API authentication monitor
• Operation execution heartbeat
• Disk usage alert for agent payloads and operation data (>80%)

Prerequisites

• MITRE Caldera running on a Linux host (Python 3.9+, aiohttp)
• Port 8888 accessible from your monitoring vantage point
• A free Vigilmon account

Step 1: Monitor the Caldera Server (Port 8888)

The Caldera server is the central command-and-control node — it hosts the web UI, REST API, and agent C2. When it goes down, all operations halt and deployed agents become orphaned.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter: http://your-caldera-host:8888/
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Add keyword check: Caldera (appears in the page title on a healthy server).
7. Click Save.

HTTPS: If you've configured Caldera with the ssl plugin, use https:// and port 8443 (or your configured SSL port). Enable Ignore SSL errors in Vigilmon if using a self-signed certificate.

Step 2: Monitor the REST API with Authentication

Caldera's REST API is used by automated red team workflows, CI/CD integrations, and purple team tooling. API downtime silently breaks all programmatic access.

1. Click Add Monitor and set Type to HTTP / HTTPS.
2. Enter: http://your-caldera-host:8888/api/v2/abilities
3. Set Check interval to 2 minutes.
4. Add a Request Header: KEY: YOUR_CALDERA_API_KEY
5. Set Expected HTTP status to 200.
6. Add keyword check: "id" (ability objects include an id field).
7. Click Save.

This monitor validates both API availability and API key authentication in a single check. A 401 response indicates the API key has changed (after Caldera reconfiguration) and your integrations need updating.

Step 3: Monitor Agent C2 Beacon Connectivity

Caldera agents (sandcats and other agent types) check in to the Caldera server over HTTP/HTTPS. The beacon endpoint must be responsive for agents to receive new abilities and upload execution results.

Caldera agents contact the server at the configured app.contact.http address (set in your conf/local.yml). Monitor that endpoint:

1. Click Add Monitor and set Type to HTTP / HTTPS.
2. Enter your Caldera agent contact URL (e.g., http://your-caldera-host:8888/beacon)
3. Set Check interval to 1 minute.
4. Set Expected HTTP status to 200 (or 403 — the beacon endpoint rejects unauthorized agent IDs but still returns a response, confirming the service is up).
5. Click Save.

A non-responsive beacon endpoint means all deployed agents are cut off from C2. In a real adversary emulation exercise, this produces false "defended" results because abilities simply can't be delivered.

Step 4: Set Up an Agent Beacon Heartbeat

Beyond monitoring the server-side beacon endpoint, confirm that agents are actually checking in from the endpoint side. This requires a heartbeat from each monitored system running a Caldera agent.

Create a Cron Heartbeat monitor in Vigilmon:

1. Click Add Monitor and set Type to Cron Heartbeat.
2. Name it Caldera Agent Beacon - [hostname].
3. Set Expected interval to 5 minutes (slightly longer than the agent's configured beacon interval).
4. Copy the heartbeat URL.

On each endpoint running a Caldera agent, add a script that fires the heartbeat after a successful agent check-in. The simplest approach is a local cron that confirms the agent process is running:

# /etc/cron.d/caldera-agent-heartbeat
* * * * * caldera_user pgrep -f "sandcat" > /dev/null && curl -fsS YOUR_HEARTBEAT_URL > /dev/null
* * * * * caldera_user sleep 60; pgrep -f "sandcat" > /dev/null && curl -fsS YOUR_HEARTBEAT_URL > /dev/null

A >20% drop in active agents (measured by missing heartbeats) indicates agent connectivity issues or network filtering affecting your emulation coverage.

Step 5: Monitor Plugin Health at Startup

Caldera is extensible via plugins (stock, compass, training, emu, atomic, ssl, and others). Plugin load failures at startup silently reduce Caldera's capability set — for example, a failed emu plugin means MITRE Engage emulation plans are unavailable.

Add a startup health check that validates plugins loaded successfully:

# Add to a custom Caldera plugin or run as a post-startup check
import aiohttp
import asyncio

async def check_plugins():
    async with aiohttp.ClientSession() as session:
        async with session.get(
            'http://localhost:8888/api/v2/plugins',
            headers={'KEY': 'YOUR_API_KEY'}
        ) as resp:
            plugins = await resp.json()
            loaded = [p['name'] for p in plugins if p.get('enabled')]
            required = ['stockpile', 'compass', 'training']
            missing = [p for p in required if p not in loaded]
            if missing:
                print(f"MISSING PLUGINS: {missing}")
                return False
            return True

Run this check after Caldera starts and send the result to a Vigilmon webhook to record startup health as a data point.

Step 6: Monitor Operation Execution via Heartbeat

The most reliable way to confirm Caldera is actually running operations (not just serving the UI) is to schedule a minimal operation via the API and verify it completes.

Create a Canary Operation

1. In Caldera, create a minimal adversary profile with a single benign ability (e.g., whoami on the local system).
2. Schedule it to run every 30 minutes against your Caldera host itself using a local agent.
3. After each operation completes, a Caldera post-operation hook sends a heartbeat to Vigilmon.

You can implement the post-operation webhook using Caldera's built-in notification hook (via the API) or a polling script:

#!/bin/bash
# Check last operation status and ping heartbeat if successful
RESULT=$(curl -s -H "KEY: YOUR_API_KEY" \
  "http://localhost:8888/api/v2/operations?limit=1" | \
  python3 -c "import sys,json; ops=json.load(sys.stdin); \
  print('ok' if ops and ops[0]['state']=='finished' else 'fail')")

if [ "$RESULT" = "ok" ]; then
  curl -fsS YOUR_OPERATION_HEARTBEAT_URL > /dev/null
fi

Create a matching Cron Heartbeat in Vigilmon with a 35-minute interval. A stalled or failed canary operation stops the heartbeat and triggers an alert.

Step 7: Monitor Disk Usage

Caldera stores agent payloads, operation reports, and fact database files on disk. A full disk prevents new agent deployments and operation report generation.

Install the Vigilmon agent on your Caldera host and configure disk monitoring:

| Path | Alert Threshold | Why | |---|---|---| | /opt/caldera/ (or install dir) | 80% | Payload and report storage | | /tmp/ | 80% | Agent staging area |

# Install Vigilmon agent
curl -fsSL https://vigilmon.online/agent/install.sh | sudo bash

Configure Disk Usage alerts at 80% in the Vigilmon dashboard for each monitored path.

Step 8: Configure Alerting

Alert Channels

In Vigilmon, go to Alert Channels and connect:

• Email — immediate notification to the red team lead and security engineering
• Slack / Teams — red team channel visibility during exercises
• PagerDuty — on-call escalation for production purple team environments
• Webhook — integrate with your security operations ticketing system

Recommended Alert Rules

| Monitor | Condition | Severity | |---|---|---| | Server Health (8888) | Down > 1 min | Critical | | REST API (authenticated) | Auth failure or down | High | | C2 Beacon Endpoint | Down > 1 min | Critical | | Agent Beacon Heartbeat | Missing per agent | High | | Operation Execution Heartbeat | Missing > 35 min | High | | Disk Usage | > 80% | High |

Status Page

Create a Vigilmon status page for your red team and security engineering leads:

1. Go to Status Pages → Create Status Page.
2. Add all Caldera monitors.
3. Share the URL with team leads before each exercise begins.

Starting an adversary emulation exercise against an unhealthy Caldera instance produces false negatives — your defenses appear to catch everything when in reality Caldera simply couldn't run the techniques.

Operational Considerations

Ability execution latency — High latency between Caldera sending an ability and receiving execution results often indicates slow agents (resource-constrained endpoints) or network latency between the endpoint and C2. Track P95 execution latency via operation result timestamps and alert on sustained regressions.

Fact collection completeness — Caldera collects facts (credential hashes, file paths, network information) for use in chained ability execution. Missing facts cause ability chains to skip steps silently. Review fact collection success rates in operation reports after each exercise.

Agent dropout during operations — If >20% of agents drop during an operation, the exercise results are statistically unreliable. Build alerting into your operation post-processing to flag operations with high agent dropout.

Unauthorized access attempts — Caldera's API key authentication should log failed authentication attempts. Monitor your Caldera access logs for repeated 401 responses from unexpected source IPs, especially since Caldera port 8888 should never be publicly accessible.

Plugin version drift — Caldera plugins (especially atomic which mirrors the Atomic Red Team project) receive frequent updates. Stale plugins may miss new technique coverage. Automate weekly plugin update checks and Caldera restarts in a maintenance window.

Conclusion

MITRE Caldera is how your security team validates that detection and response controls actually work. Infrastructure failures during adversary emulation exercises produce misleading results — your blue team thinks they caught everything when Caldera simply couldn't execute the techniques.

With Vigilmon, you get:

• C2 server uptime monitoring ensuring agents can reach the Caldera server throughout exercises
• Agent beacon heartbeats catching agent dropout that invalidates exercise results
• API authentication alerts catching key rotation issues before they break integrations
• Operation execution heartbeat validating the full execution pipeline end-to-end
• Disk alerts preventing payload delivery failures mid-exercise

Get started free at vigilmon.online.