Monitoring Robot Framework Test Suites with Vigilmon: Detect Silent Acceptance Test Failures

Your Robot Framework nightly acceptance suite had been green for weeks. Then a misconfigured pabot split caused two of four parallel runners to silently exit before any tests loaded. Exit code 0, no failures logged, no humans alerted — and three user stories went untested through a release.

This is the silent acceptance test failure problem. Standard uptime monitors watch HTTP endpoints, not "did Robot Framework actually run today?" The fix is heartbeat monitoring: Robot Framework pings a unique URL on successful completion, and Vigilmon alerts you the moment that ping stops arriving on schedule.

This tutorial shows you how to wire Robot Framework into Vigilmon so silent test outages never slip past your team again.

What You'll Cover

Heartbeat monitoring for scheduled Robot Framework suites
A Robot Framework listener that pings Vigilmon after a clean run
Detecting runners that quit before any tests load
Multi-suite monitoring (smoke, regression, acceptance)
Alerting on suites that hang or time out
CI integration for GitHub Actions and GitLab CI

Prerequisites

Robot Framework 5.x or 6.x installed (pip install robotframework)
A free account at vigilmon.online

The Problem: What Robot Framework Reporting Misses

Robot Framework generates detailed HTML reports and XML output — but nothing tells you when:

The CI job invoking robot was accidentally disabled or deleted
A suite file path change causes zero tests to be discovered (exit 0)
A pabot runner crashes before the first test keyword executes
The Python environment is broken and the robot process exits immediately
A nightly scheduled trigger is dropped after a pipeline migration

Vigilmon's heartbeat pattern closes every one of these gaps by treating the absence of a ping as an alert condition.

Step 1: Create a Heartbeat Monitor in Vigilmon

Log in to Vigilmon and click New Monitor → Heartbeat.
Name it something like Robot Framework — Nightly Acceptance.
Set Expected interval to match your schedule — 24 hours for nightly, 4 hours for pre-deploy smoke.
Set Grace period to 30 minutes to absorb slow keyword library imports.
Save and copy the Ping URL — it looks like https://vigilmon.online/api/heartbeat/<unique-id>.

Step 2: Create a Vigilmon Listener

Robot Framework's Listener API lets you hook into the full test lifecycle. Create a listener that fires the heartbeat only when the suite passes:

# VigilmonListener.py
import os
import urllib.request

VIGILMON_HEARTBEAT_URL = os.environ.get("VIGILMON_HEARTBEAT_URL", "")

class VigilmonListener:
    ROBOT_LISTENER_API_VERSION = 3

    def __init__(self, heartbeat_url=None):
        self.heartbeat_url = heartbeat_url or VIGILMON_HEARTBEAT_URL
        self._failed = False

    def end_test(self, data, result):
        if not result.passed:
            self._failed = True

    def close(self):
        if self._failed or not self.heartbeat_url:
            return
        try:
            req = urllib.request.Request(
                self.heartbeat_url,
                method="POST",
                headers={"Content-Length": "0"},
            )
            with urllib.request.urlopen(req, timeout=10):
                print("Vigilmon heartbeat sent")
        except Exception as exc:
            print(f"Vigilmon heartbeat failed: {exc}")

The close() method is invoked after the last test finishes. Any failed test sets self._failed = True, which suppresses the heartbeat — causing Vigilmon to alert after the grace period.

Step 3: Register the Listener and Run Robot Framework

Command line

VIGILMON_HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/<id>" \
  robot --listener VigilmonListener tests/

robot.yaml / pyproject.toml

# robot.yaml (rcc / robocorp convention)
tasks:
  Default:
    command:
      - robot
      - --listener
      - VigilmonListener
      - tests/

GitHub Actions

# .github/workflows/acceptance.yml
name: Robot Framework Acceptance

on:
  schedule:
    - cron: "0 1 * * *"   # 01:00 UTC nightly
  push:
    branches: [main]

jobs:
  robot:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run Robot Framework
        env:
          VIGILMON_HEARTBEAT_URL: ${{ secrets.VIGILMON_ROBOT_HEARTBEAT_URL }}
        run: |
          robot --listener VigilmonListener \
                --outputdir results \
                tests/

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: robot-results
          path: results/

GitLab CI

# .gitlab-ci.yml
robot:acceptance:
  stage: test
  image: python:3.12
  variables:
    VIGILMON_HEARTBEAT_URL: $VIGILMON_ROBOT_HEARTBEAT_URL
  only:
    - schedules
  script:
    - pip install -r requirements.txt
    - robot --listener VigilmonListener --outputdir results tests/
  artifacts:
    when: always
    paths:
      - results/

Step 4: Listener with Heartbeat URL as Argument

If you run multiple Robot suites against separate Vigilmon monitors, pass the URL directly to the listener instead of using an environment variable:

robot \
  --listener "VigilmonListener:https://vigilmon.online/api/heartbeat/<smoke-id>" \
  tests/smoke/

robot \
  --listener "VigilmonListener:https://vigilmon.online/api/heartbeat/<regression-id>" \
  tests/regression/

Robot Framework passes the string argument after the colon to __init__ as heartbeat_url.

Step 5: Parallel Runs with pabot

When using pabot for parallel execution, wrap the run in a shell script so the heartbeat fires only after all parallel shards complete:

#!/usr/bin/env bash
# run_tests.sh
set -e

pabot --processes 4 --outputdir results tests/

# pabot exits non-zero on any failure — reaching here means success
if [ -n "$VIGILMON_HEARTBEAT_URL" ]; then
  curl -fsS -X POST "$VIGILMON_HEARTBEAT_URL" --max-time 10 --retry 3
  echo "Vigilmon heartbeat sent"
fi

# GitHub Actions step
- name: Run parallel acceptance tests
  env:
    VIGILMON_HEARTBEAT_URL: ${{ secrets.VIGILMON_ROBOT_HEARTBEAT_URL }}
  run: bash run_tests.sh

set -e ensures the heartbeat is skipped if pabot returns a non-zero exit code.

Step 6: Multi-Suite Monitoring

Create a separate Vigilmon monitor for each test tier:

| Suite | Interval | Grace | Monitor name | |---|---|---|---| | Nightly full regression | 25 h | 60 min | Robot — Nightly Regression | | Pre-deploy smoke | 4 h | 15 min | Robot — Smoke Tests | | Weekly cross-browser | 8 days | 2 h | Robot — Weekly Cross-Browser | | Data-driven acceptance | 24 h | 30 min | Robot — Data-Driven Suite |

Step 7: Configure Alert Channels

In Vigilmon, go to Notifications → New Channel and configure:

Email — immediate alert when a heartbeat is missed
Slack webhook — ping your #qa-alerts or #robot-tests channel

Sample missed-heartbeat notification:

🔴 MISSED HEARTBEAT: Robot Framework — Nightly Acceptance
Last ping: 25 hours ago
Expected interval: 24 hours

Recovery notification:

✅ HEARTBEAT RECOVERED: Robot Framework — Nightly Acceptance
Gap: 25 hours

What You're Now Catching

| Silent failure mode | How Vigilmon detects it | |---|---| | CI job disabled or deleted | No ping arrives → alert after grace period | | Zero tests discovered (bad path) | Suite ends with 0 tests, no ping → alert | | pabot runner crashes at startup | Process exits before tests run, no ping → alert | | Broken Python environment | robot exits immediately, no ping → alert | | Suite hangs on a keyword | CI timeout kills process, no ping → alert | | Nightly trigger dropped | Scheduled job missing, no ping → alert |

Robot Framework gives you powerful keyword-driven and data-driven acceptance test coverage — right up until the runner goes quiet. Vigilmon's heartbeat monitors give you the external signal your RF reports can't: a definitive alert when your acceptance tests haven't run in longer than expected.

Wire Vigilmon into your Robot Framework suite today — register free at vigilmon.online.