Monitoring Ansible Semaphore with Vigilmon: Health Endpoints, Web UI Uptime, TCP Port Checks & SSL Alerts

Ansible Semaphore is the open-source web UI for running Ansible playbooks — it schedules tasks, stores inventory and credentials, and keeps an audit trail of every playbook run. When Semaphore goes down, operators lose visibility into scheduled automation, running playbooks silently time out, and teams fall back to manual ad-hoc commands with no audit trail. Vigilmon gives you external visibility into Semaphore before your team notices: the health API endpoint, web UI availability, TCP port reachability, and SSL certificate expiry.

What You'll Build

• A monitor on Semaphore's /api/ping health endpoint to catch process failures
• A monitor on the web UI login page for end-to-end availability
• A TCP port check for direct connectivity to the Semaphore service
• SSL certificate monitoring for your Semaphore domain
• An alerting setup that routes failures to the right responder

Prerequisites

• A running Ansible Semaphore instance (v2.x) accessible via HTTPS
• A domain pointing to Semaphore (e.g., https://semaphore.example.com)
• A free account at vigilmon.online

Step 1: Verify Semaphore's Health Endpoint

Ansible Semaphore exposes a lightweight, unauthenticated health endpoint:

# Test the ping endpoint
curl -s https://semaphore.example.com/api/ping
# Expected response: "PONG"

# Check HTTP status
curl -I https://semaphore.example.com/api/ping
# Expected: HTTP/2 200

The /api/ping endpoint returns the plain-text string PONG with HTTP 200 when Semaphore is running and its API is healthy. This is the preferred machine-readable check — it does not require authentication and adds negligible load.

For the web UI check, the login page confirms that the frontend assets, session middleware, and database connection are all working together:

curl -I https://semaphore.example.com/auth/login
# Expected: HTTP/2 200

Reverse proxy users: If Semaphore runs behind nginx or Caddy with a sub-path (e.g., /semaphore), prepend the path: https://semaphore.example.com/semaphore/api/ping. Check your proxy configuration for the location block that forwards to Semaphore's listen port (default: 3000).

Step 2: Create a Vigilmon HTTP Monitor for the Health Endpoint

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://semaphore.example.com/api/ping.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: PONG (exact response body when Semaphore is healthy).
7. Label: Semaphore API ping.
8. Click Save.

This monitor catches:

• Semaphore process crashes or OOM kills
• Database connection failures (Semaphore returns 500 when it can't reach its DB)
• Configuration errors that prevent the API from starting
• Failed upgrades that leave the API in a broken state

Alert sensitivity: Set alerts to trigger after 2 consecutive failures. Brief blips during playbook execution under heavy load are normal; two consecutive failures reliably indicate a real outage.

Step 3: Create a Vigilmon HTTP Monitor for the Web UI

The web UI check validates that the full Semaphore stack — API, frontend, session store — is serving requests:

1. Add Monitor → HTTP.
2. URL: https://semaphore.example.com/auth/login.
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: Semaphore (appears in the page title).
7. Label: Semaphore web UI.
8. Click Save.

When the UI monitor fires but the API ping monitor is green, you have a frontend-specific problem — a failed static asset build, misconfigured reverse proxy, or session middleware issue — without a full API outage.

Step 4: Create a TCP Monitor for the Semaphore Port

Semaphore listens on TCP port 3000 by default (or your configured port in config.json). Monitoring the raw TCP port catches network-level failures that HTTP monitors miss — firewall rule changes, routing failures, and service restarts with a different port:

# Verify the port is open
nc -zv semaphore.example.com 3000
# or via curl
curl -v telnet://semaphore.example.com:3000

1. Add Monitor → TCP.
2. Host: semaphore.example.com.
3. Port: 3000 (or your configured Semaphore port).
4. Check interval: 2 minutes.
5. Response timeout: 10 seconds.
6. Label: Semaphore TCP port.
7. Click Save.

Port 80/443 through a proxy: If Semaphore is behind a reverse proxy on standard ports, monitor port 443 at the proxy level and additionally monitor port 3000 directly if it's accessible for internal checks. Two separate checks reveal whether failures are at the proxy or at Semaphore itself.

Step 5: Monitor SSL Certificates

Semaphore is commonly accessed from automation scripts, CI pipelines, and the Ansible CLI itself — all of which perform strict TLS validation. An expired certificate causes:

• All API calls from automation to fail with TLS errors
• Browser warnings that interrupt operator access during incidents
• Webhook deliveries from source control to fail silently

1. Add Monitor → SSL Certificate.
2. Domain: semaphore.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

Let's Encrypt users: If you use Certbot or acme.sh for automatic renewal, Semaphore's certificate should renew automatically — but renewals fail when the ACME HTTP challenge port is blocked or when the renewal cron job silently stops running. Vigilmon's 30-day alert gives you a month of lead time to diagnose renewal failures before they cause an outage.

Step 6: Monitor the Semaphore API Authentication Endpoint

The /api/auth/login endpoint exercises the full authentication stack — API, database query, and session creation. It's the best smoke test for the parts of Semaphore that operators actually use:

curl -s -o /dev/null -w "%{http_code}" -X POST \
  https://semaphore.example.com/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"auth":"","password":""}'
# Returns 400 (bad request) when the API is healthy — the request is intentionally invalid

1. Add Monitor → HTTP.
2. URL: https://semaphore.example.com/api/auth/login.
3. Method: POST.
4. Request body: {"auth":"","password":""}.
5. Content-Type header: application/json.
6. Expected status: 400 (invalid credentials confirm the API is up and parsing requests).
7. Check interval: 5 minutes.
8. Label: Semaphore auth API.
9. Click Save.

A 400 response to an intentionally malformed request is the healthy signal — it means Semaphore received, parsed, and rejected the request as expected. A 502, 503, or connection error means the API is down.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | API ping | Non-200 or PONG missing | Check systemctl status semaphore; inspect Semaphore logs | | Web UI | Non-200 or keyword missing | Check reverse proxy config; verify frontend assets built correctly | | TCP port | Connection refused or timeout | Check firewall rules; confirm Semaphore's configured port in config.json | | SSL certificate | < 30 days to expiry | Trigger manual certificate renewal; check Certbot/ACME cron job | | Auth API | Non-400/200 | API routing or database connectivity issue |

Alert after: 2 consecutive failures for HTTP monitors. 1 failure for TCP monitors — TCP timeouts are rarely transient.

Common Ansible Semaphore Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Semaphore process crash | API ping unreachable; alert within 60 s | | Database connection failure | API ping returns 500; auth API also fails | | Frontend asset build failure | UI monitor fires; API ping stays green | | Firewall blocks port 3000 | TCP monitor fires; monitors via proxy may stay green | | SSL certificate expires | SSL monitor alerts at 30-day threshold; API calls from automation fail | | Config change breaks auth | Auth API returns unexpected status; ping may stay green | | Reverse proxy misconfiguration | UI and auth monitors fire; TCP direct port may still respond | | Disk full (playbook output storage) | Playbook runs fail; API may continue responding | | DNS misconfiguration | All monitors fire simultaneously |

Semaphore downtime has an asymmetric blast radius: automated playbook schedules silently stop, and the team only discovers the gap when a scheduled maintenance window or patch run was supposed to execute. Vigilmon gives you early warning through the API health endpoint, web UI availability, TCP port reachability, and SSL certificate expiry — catching Semaphore failures in seconds before they cascade into unexecuted automation.

Start monitoring Ansible Semaphore in under 5 minutes — register free at vigilmon.online.