tutorial

Monitoring Puppet Server with Vigilmon: Health Endpoints, API Uptime, TCP Port Checks & SSL Alerts

How to monitor Puppet Server with Vigilmon — health check endpoint monitoring, Puppet API uptime, TCP port checks, and SSL certificate alerts.

Puppet Server is the central policy authority for Puppet-managed infrastructure — it compiles catalogs, serves module code, signs agent certificates, and maintains the authoritative node classification. When Puppet Server goes down, every agent's next scheduled catalog run fails silently, configuration drift accumulates uncorrected, and new nodes can't be bootstrapped. Emergency policy changes can't be applied. Vigilmon gives you external visibility into Puppet Server before your infrastructure team notices: the /status/v1/simple health endpoint, Puppet API availability, TCP port reachability, and SSL certificate expiry.

What You'll Build

  • A monitor on Puppet Server's /status/v1/simple endpoint to catch process failures
  • An HTTP monitor for the Puppet CA API health
  • A TCP port check for the Puppet agent port (8140)
  • SSL certificate monitoring for your Puppet Server domain
  • An alerting setup that separates catalog compilation failures from network issues

Prerequisites

  • A running Puppet Server (v6+) accessible over HTTPS on port 8140
  • A hostname pointing to Puppet Server (e.g., puppet.example.com)
  • A free account at vigilmon.online

Step 1: Verify Puppet Server Health Endpoints

Puppet Server exposes dedicated status endpoints for monitoring:

# Simple health check — returns "running" when healthy (unauthenticated)
curl -sk https://puppet.example.com:8140/status/v1/simple
# Expected: "running"

# Check HTTP status
curl -skI https://puppet.example.com:8140/status/v1/simple
# Expected: HTTP/1.1 200 OK

# Detailed status — shows all service components and their states
curl -sk https://puppet.example.com:8140/status/v1/services | python3 -m json.tool

The /status/v1/simple endpoint returns the plain string running with HTTP 200 when Puppet Server and all its services are healthy. This is the preferred monitoring endpoint — it's lightweight, unauthenticated, and designed for automated checks.

The detailed /status/v1/services endpoint returns JSON with individual component states:

{
  "puppet-server": {
    "state": "running",
    "status": {
      "jruby-puppet": { "state": "running" },
      "file-sync-storage-service": { "state": "running" }
    }
  }
}

PuppetDB integration: PuppetDB runs as a separate service on a different port. Puppet Server's /status/v1/services includes PuppetDB connectivity status. If PuppetDB is down, Puppet Server may still compile catalogs but node reports and exported resources fail. See the PuppetDB section below for additional checks.

Step 2: Create a Vigilmon HTTP Monitor for the Simple Health Endpoint

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://puppet.example.com:8140/status/v1/simple.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds (Puppet Server can be slow to respond during JVM GC or catalog compilation spikes).
  5. Expected status: 200.
  6. Keyword: running (exact response when healthy).
  7. Label: Puppet Server health.
  8. Click Save.

This monitor catches:

  • Puppet Server JVM process crashes or OOM kills
  • JRuby pool exhaustion (Puppet Server returns errors when all JRuby instances are busy)
  • Failed puppet-server service restarts after upgrades
  • File sync service failures that prevent catalog compilation

Alert sensitivity: Set alerts to trigger after 2 consecutive failures. JRuby pool contention during high catalog compilation load can cause brief delays; two consecutive failures indicate a sustained service problem.

Step 3: Create a Vigilmon HTTP Monitor for the Detailed Status Endpoint

The /status/v1/services endpoint reveals component-level failures that the simple endpoint may aggregate:

  1. Add Monitor → HTTP.
  2. URL: https://puppet.example.com:8140/status/v1/services.
  3. Check interval: 2 minutes.
  4. Response timeout: 20 seconds.
  5. Expected status: 200.
  6. Keyword: "state":"running" (present for each healthy component).
  7. Label: Puppet Server component status.
  8. Click Save.

When this monitor fires but the simple health endpoint stays green:

  • A non-critical service component degraded without taking the whole server down
  • JRuby pool has reduced capacity (fewer instances running than configured)
  • File sync is stale but Puppet Server continues compiling from cached data

Step 4: Create a TCP Monitor for the Puppet Agent Port

Puppet agents connect to Puppet Server exclusively over TCP port 8140 (HTTPS). If this port is blocked, every agent run fails with a connection error:

# Verify port 8140 is open
nc -zv puppet.example.com 8140
# or
curl -vk telnet://puppet.example.com:8140
  1. Add Monitor → TCP.
  2. Host: puppet.example.com.
  3. Port: 8140.
  4. Check interval: 2 minutes.
  5. Response timeout: 10 seconds.
  6. Label: Puppet Server agent port (8140).
  7. Click Save.

When the TCP monitor fires but HTTP monitors were passing:

  • A firewall rule change blocked port 8140
  • The JVM process crashed after the last HTTP check
  • A load balancer health check misconfiguration pulled Puppet Server from rotation

When both TCP and HTTP monitors fire simultaneously, the Puppet Server JVM is down or the host is unreachable.

Step 5: Monitor SSL Certificates

Puppet uses TLS for all agent-server communication, and agents validate the server certificate against the Puppet CA. An expired server certificate causes:

  • Every puppet agent --test run to fail with certificate verify failed
  • Node bootstrapping to fail during the certificate signing step
  • PuppetDB report submission to fail if it goes through Puppet Server
  1. Add Monitor → SSL Certificate.
  2. Domain: puppet.example.com.
  3. Port: 8140 (Puppet Server's HTTPS port, not 443).
  4. Alert when expiry is within: 30 days.
  5. Alert again: 14 days, 7 days, 3 days, 1 day.
  6. Click Save.

Puppet CA certificates: Puppet Server manages its own internal CA, issuing agent certificates signed by the Puppet CA. The certificate you monitor with Vigilmon is the Puppet Server's host certificate (not the CA certificate itself). Check /etc/puppetlabs/puppet/ssl/certs/puppet.example.com.pem for its expiry date — this is what agents validate.

Certificate rotation: Puppet Server host certificates default to 5-year validity, so certificate expiry is rare — but when it happens, it takes down every agent simultaneously. The 30-day Vigilmon alert gives you ample time to rotate the host certificate before agents break.

Step 6: Monitor the Puppet CA API

The Puppet CA (Certificate Authority) API handles new agent certificate signing requests. If the CA is unavailable, new nodes can't join:

# CA list endpoint returns 403 (forbidden) for unauthenticated requests when healthy
curl -skI https://puppet.example.com:8140/puppet-ca/v1/certificate_statuses/all
# Expected: HTTP/1.1 403 Forbidden (confirms CA is up and enforcing auth)
  1. Add Monitor → HTTP.
  2. URL: https://puppet.example.com:8140/puppet-ca/v1/certificate_statuses/all.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 403.
  6. Label: Puppet CA API.
  7. Click Save.

A 403 response to an unauthenticated request is the healthy signal — Puppet CA received the request and correctly rejected it without valid client certificate auth. A 502, 503, or connection error means the CA service within Puppet Server is down.

Step 7: Monitor PuppetDB Connectivity (Optional)

If PuppetDB is in your stack, its availability affects exported resources, resource collectors, and compliance reporting. PuppetDB runs on port 8081 by default:

# PuppetDB status endpoint
curl -sk https://puppetdb.example.com:8081/status/v1/simple
# Expected: "running"
  1. Add Monitor → HTTP.
  2. URL: https://puppetdb.example.com:8081/status/v1/simple.
  3. Check interval: 60 seconds.
  4. Expected status: 200.
  5. Keyword: running.
  6. Label: PuppetDB health.
  7. Click Save.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Simple health | Non-200 or running missing | Check systemctl status puppetserver; review /var/log/puppetlabs/puppetserver/puppetserver.log | | Component status | Non-200 or component not running | Identify failing component; check JRuby pool size with puppet infrastructure status | | TCP port 8140 | Connection refused or timeout | Check firewall rules; verify puppet-server process is running | | SSL certificate | < 30 days to expiry | Rotate Puppet Server host certificate via puppetserver ca subcommands | | Puppet CA API | Non-403 | CA service issue; check Puppet Server logs for CA errors | | PuppetDB | Non-200 or running missing | Check systemctl status puppetdb; review PuppetDB logs |

Alert after: 2 consecutive failures for HTTP monitors. 1 failure for TCP monitors — a TCP failure immediately means all Puppet agent runs are failing.

Common Puppet Server Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | JVM OOM / Puppet Server crash | Simple health unreachable; TCP monitor fires; alert within 60–120 s | | JRuby pool exhausted | Component status shows degraded JRuby; catalog compilations queue and time out | | File sync failure | Component status shows file sync not running; agents may get stale catalogs | | Puppet CA failure | CA API monitor fires; new node bootstrapping fails | | PuppetDB connection failure | Component status may show PuppetDB degraded; exported resources fail | | SSL certificate expires | SSL monitor alerts at 30-day threshold; all agent runs fail simultaneously | | Firewall blocks port 8140 | TCP monitor fires; all agent runs fail immediately | | puppetserver.conf misconfiguration after upgrade | Simple health returns 500; check server logs | | DNS misconfiguration | All monitors fire simultaneously | | Disk full (catalog cache / SSL dir) | Catalog compilation fails; server logs fill; JVM may crash |

Puppet Server outages are deferred-pain failures: agents silently fall back to running against their last cached catalog (if configured), drift accumulates, and compliance reports stop updating — until an operator notices that the last catalog compilation timestamp is hours ago. Vigilmon catches Puppet Server failures the moment they happen: the /status/v1/simple endpoint alerts within 60 seconds of a JVM crash, while TCP monitors catch network-level failures that silently break every agent run before the first catalog compilation timeout fires.

Start monitoring Puppet Server in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles