Monitoring Semaphore UI with Vigilmon

Semaphore UI is a clean, self-hosted web interface for running Ansible playbooks and Terraform plans without touching the terminal. It's popular for teams who want a lightweight CI-style task runner without the overhead of Jenkins or AWX. But when Semaphore's web UI or database goes silent, automation jobs stop being scheduled and no one notices until someone checks manually. Vigilmon gives you continuous uptime monitoring across every layer of Semaphore so failures surface immediately.

What You'll Set Up

• HTTP uptime monitor for the Semaphore web UI
• Health endpoint check via /api/ping
• TCP check for SSH connectivity to managed hosts
• SSL certificate alerts for proxied Semaphore instances
• Cron heartbeat to confirm scheduled task runs

Prerequisites

• Semaphore UI installed and running (default port :3000)
• A free Vigilmon account

Step 1: Monitor the Semaphore Web UI

The primary availability signal for Semaphore is whether the web interface responds on port 3000. Add an HTTP monitor:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Semaphore URL: http://your-server:3000 or https://semaphore.yourdomain.com if proxied.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

This catches the common failure modes: the Semaphore process crashing, the container exiting, or the host running out of memory under heavy playbook execution.

Step 2: Check the /api/ping Health Endpoint

Semaphore exposes a lightweight health endpoint at /api/ping that returns a 200 OK when the API server is healthy. Add a dedicated monitor for this endpoint:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-server:3000/api/ping
3. Set Expected HTTP status to 200.
4. Optionally set Expected response body contains to "pong" — Semaphore returns "pong" on success.
5. Set interval to 1 minute.
6. Click Save.

The /api/ping check is lighter than loading the full UI and confirms that the API layer serving task execution is alive, not just the static frontend.

Step 3: Verify Database Connectivity

Semaphore supports MySQL, PostgreSQL, and BoltDB as its backend. If the database connection drops, Semaphore appears to start but fails silently when scheduling or running tasks. The /api/ping endpoint validates the database connection before returning 200, so keeping that monitor active covers this layer.

For additional confidence with MySQL or PostgreSQL, add a TCP port monitor to the database host:

1. Click Add Monitor → TCP Port.
2. Enter the database host and port: your-db-host:3306 (MySQL) or your-db-host:5432 (PostgreSQL).
3. Set interval to 2 minutes.
4. Click Save.

A TCP failure on the database port before a Semaphore API failure helps you pinpoint the root cause faster during an incident.

Step 4: Monitor SSH Connectivity to Managed Hosts

Semaphore runs Ansible and Terraform against remote hosts over SSH. If the SSH connection between Semaphore's host and a managed node is blocked (firewall change, key rotation, host down), playbooks fail at execution time. Use Vigilmon's TCP check to monitor SSH reachability:

1. Click Add Monitor → TCP Port.
2. Enter the managed host address and SSH port: target-host:22.
3. Set interval to 5 minutes.
4. Click Save.

Repeat this for each critical target host in your Semaphore inventory. A TCP failure on port 22 before a failed playbook run gives you the alert you need to fix connectivity before the next scheduled job fires.

Step 5: SSL Certificate Alerts for Proxied Instances

Many teams put Semaphore behind nginx or Caddy with a TLS certificate. Certificate expiry will lock out the entire team from the UI.

1. Open the Semaphore HTTP monitor created in Step 1.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

For a Caddy-proxied setup, also verify that the Caddy process itself is alive:

# Check Caddy is listening on 443
ss -tlnp | grep 443

If you use Let's Encrypt via nginx, add a cron entry to log renewal attempts and verify with the Vigilmon heartbeat below.

Step 6: Heartbeat for Scheduled Task Runs

Semaphore has a built-in scheduler for recurring playbooks. If the scheduler itself freezes, tasks stop running silently. Use Vigilmon's cron heartbeat to confirm jobs are executing on schedule.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected interval to match your most critical scheduled task (e.g., 60 minutes for hourly).
3. Copy the heartbeat URL.
4. In Semaphore, add a final playbook task or shell step that calls the URL on success:

- name: Signal Vigilmon heartbeat
  uri:
    url: "https://vigilmon.online/heartbeat/YOUR_ID"
    method: GET
  ignore_errors: false

If the scheduled playbook never completes, Vigilmon alerts when the heartbeat window expires.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP / UI | :3000 or proxied domain | Process crash, container exit | | Health endpoint | /api/ping | API failure, DB disconnect | | TCP / Database | :3306 or :5432 | Database unreachable | | TCP / SSH | Managed host :22 | SSH blocked, host unreachable | | SSL certificate | Proxied domain | Let's Encrypt expiry | | Cron heartbeat | Heartbeat URL | Scheduler freeze, missed jobs |

Semaphore UI makes Ansible and Terraform accessible to your whole team — but a silent failure at any layer means automation stops while no one notices. With Vigilmon watching the web UI, the health API, database connectivity, SSH reachability, and scheduled heartbeats, you catch problems before they cause missed deployments or stale infrastructure.