tutorial

Monitoring SaltStack (Salt Master) with Vigilmon: Health Endpoints, API Uptime, TCP Port Checks & SSL Alerts

How to monitor SaltStack Salt Master with Vigilmon — health check endpoint monitoring, Salt API uptime, TCP port checks, and SSL certificate alerts.

SaltStack (Salt) is a high-speed infrastructure automation platform built around the Salt Master, which orchestrates minions via encrypted ZeroMQ or SSH channels. When the Salt Master goes down, minions become unreachable, scheduled states stop applying, and emergency remediations can't execute — leaving your infrastructure frozen in whatever state it was in when the master failed. Vigilmon gives you external visibility into Salt Master before your infrastructure team notices: the Salt API endpoint, TCP port reachability for minion connections, and SSL certificate expiry.

What You'll Build

  • A monitor on the Salt API (/ or /login) to catch master process failures
  • HTTP monitors for Salt API liveness and authentication
  • TCP port checks for the Salt Master's publish port (4505) and return port (4506)
  • SSL certificate monitoring for your Salt API domain
  • An alerting setup that distinguishes API failures from network-level issues

Prerequisites

  • A running Salt Master with salt-api configured (cherrypy or tornado driver)
  • A domain or IP pointing to Salt API (e.g., https://salt.example.com)
  • A free account at vigilmon.online

Step 1: Verify Salt API Endpoints

The salt-api service provides an HTTP interface to the Salt Master. Verify it's responding before creating monitors:

# Test the Salt API root (returns metadata about the API)
curl -s https://salt.example.com/
# Expected: JSON with "return" key listing available clients

# Check HTTP status
curl -I https://salt.example.com/
# Expected: HTTP/2 200

# Test the login endpoint (does not log in — just confirms it exists)
curl -I https://salt.example.com/login
# Expected: HTTP/2 200 (GET returns form or instructions)

The root endpoint / returns JSON with information about the Salt API and its available clients when the service is healthy.

For the Salt Master's ZeroMQ ports, minions connect on:

  • Port 4505 (publish port — master broadcasts commands to minions)
  • Port 4506 (return port — minions send job results back)
# Verify ports are open from an external perspective
nc -zv salt.example.com 4505
nc -zv salt.example.com 4506

Firewall note: Ports 4505 and 4506 may be restricted to your internal network. If so, TCP monitors from an external check network won't reach them — use internal monitoring or skip TCP monitors and rely on the Salt API checks instead.

Step 2: Create a Vigilmon HTTP Monitor for Salt API Root

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://salt.example.com/.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: return (present in Salt API root JSON response).
  7. Label: Salt API root.
  8. Click Save.

This monitor catches:

  • salt-api process crashes or failed restarts
  • Configuration errors that prevent the API from binding
  • Salt Master daemon failures that cascade into salt-api errors
  • Database or PKI directory corruption that breaks the master

Alert sensitivity: Set alerts to trigger after 2 consecutive failures. A single failure during a Salt state run that temporarily saturates the API is possible; two consecutive failures indicate a real outage.

Step 3: Create a Vigilmon HTTP Monitor for the Login Endpoint

The /login endpoint exercises Salt API's authentication layer. Monitoring it confirms the auth module (PAM, LDAP, or file-based) is reachable:

  1. Add Monitor → HTTP.
  2. URL: https://salt.example.com/login.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Label: Salt API login.
  7. Click Save.

Method note: A GET to /login returns instructions or a redirect with HTTP 200 in most salt-api configurations. If your deployment returns a different code on GET, adjust the expected status accordingly — what matters is that the endpoint responds without a 5xx error.

Step 4: Create TCP Monitors for Salt Master Ports

Salt Master's ZeroMQ ports are the critical communication channels for all minion control. Monitoring them confirms that minions can reach the master at the network level:

Port 4505 (publish port):

  1. Add Monitor → TCP.
  2. Host: salt.example.com (or the Salt Master IP).
  3. Port: 4505.
  4. Check interval: 2 minutes.
  5. Response timeout: 10 seconds.
  6. Label: Salt Master publish port (4505).
  7. Click Save.

Port 4506 (return port):

  1. Add Monitor → TCP.
  2. Host: salt.example.com.
  3. Port: 4506.
  4. Check interval: 2 minutes.
  5. Response timeout: 10 seconds.
  6. Label: Salt Master return port (4506).
  7. Click Save.

When the TCP monitors fire but the Salt API HTTP monitors stay green:

  • The ZeroMQ layer is blocked by a firewall
  • The salt-master daemon crashed while salt-api kept running
  • A port number was changed in /etc/salt/master

When both TCP and HTTP monitors fire simultaneously, the entire Salt Master stack is down.

Step 5: Monitor SSL Certificates

Salt API is accessed by automation scripts, pipeline tools, and Reactor hooks — all performing strict TLS validation. An expired certificate causes:

  • All salt-api calls from scripts and CI pipelines to fail with TLS errors
  • Event-driven Reactor responses to fail silently
  • Browser access to any salt-api web UI to throw security warnings
  1. Add Monitor → SSL Certificate.
  2. Domain: salt.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Self-signed certificates: Salt uses its own PKI for minion-master communication, but salt-api typically uses a separate TLS certificate for its HTTPS interface. Monitor the certificate that browsers and API clients use — the one on port 443 — not the internal Salt PKI certificates.

Step 6: Monitor Salt API Authentication Flow

Test the authentication endpoint end-to-end with an intentionally invalid request that confirms the auth stack is working:

curl -s -o /dev/null -w "%{http_code}" -X POST \
  https://salt.example.com/login \
  -H "Content-Type: application/json" \
  -d '{"username":"","password":"","eauth":"pam"}'
# Returns 401 (unauthorized) when auth stack is healthy
  1. Add Monitor → HTTP.
  2. URL: https://salt.example.com/login.
  3. Method: POST.
  4. Request body: {"username":"","password":"","eauth":"pam"}.
  5. Content-Type header: application/json.
  6. Expected status: 401.
  7. Check interval: 5 minutes.
  8. Label: Salt API auth stack.
  9. Click Save.

A 401 response confirms the Salt API received the request, invoked the eauth module, and rejected the invalid credentials — the entire auth path is healthy. A 500 or connection error means the auth module or Salt Master connectivity is broken.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Salt API root | Non-200 or return missing | Check systemctl status salt-api; inspect /var/log/salt/api | | Login endpoint | Non-200 | Check salt-api config in /etc/salt/api; verify eauth module | | Port 4505 | Connection refused or timeout | Check systemctl status salt-master; verify firewall allows 4505 | | Port 4506 | Connection refused or timeout | Check systemctl status salt-master; verify firewall allows 4506 | | SSL certificate | < 30 days to expiry | Renew TLS cert for salt-api; check Certbot/ACME renewal cron | | Auth stack | Non-401 | eauth module failure; check PAM/LDAP config and Salt Master connectivity |

Alert after: 2 consecutive failures for HTTP monitors. 1 failure for TCP monitors — loss of the publish or return port immediately disconnects all minions.

Common SaltStack Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | salt-master daemon crash | TCP ports 4505/4506 unreachable; alert within 2 min | | salt-api crash, master running | HTTP monitors fire; TCP port monitors stay green | | Firewall blocks ZeroMQ ports | TCP monitors fire; HTTP monitors stay green | | Salt PKI directory corruption | Master starts but minions reject auth; API monitor may still pass | | SSL certificate expires | SSL monitor alerts at 30-day threshold; API script calls fail | | eauth module misconfiguration | Auth stack monitor fires; root endpoint stays green | | Disk full (job cache) | Salt state runs fail; master may still respond to API | | DNS misconfiguration | All monitors fire simultaneously | | High minion count saturates return port | Intermittent TCP timeouts on 4506; HTTP monitors slow |

Salt Master downtime is operationally silent at first: minions stop receiving commands, scheduled states don't apply, and emergency patches can't be pushed — but no error appears in your CI system or monitoring dashboard until someone tries to run a salt command. Vigilmon closes that gap with proactive checks on the API endpoint, ZeroMQ ports, and SSL certificate, alerting your team in seconds rather than waiting for an operator to notice that automation stopped.

Start monitoring SaltStack in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles