tutorial

Monitoring Chef Infra Server with Vigilmon: Health Endpoints, API Uptime, TCP Port Checks & SSL Alerts

How to monitor Chef Infra Server with Vigilmon — health check endpoint monitoring, Chef API uptime, TCP port checks, and SSL certificate alerts.

Chef Infra Server is the central hub for all Chef-managed infrastructure — it stores cookbooks, node data, roles, and environments, and serves as the policy source that every Chef client pulls from during convergence runs. When Chef Infra Server goes down, every node's next scheduled chef-client run fails, policy updates can't be pushed, and node attributes stop syncing. Emergency cookbook changes can't be uploaded. Vigilmon gives you external visibility into Chef Infra Server before your operations team notices: the /_status health endpoint, Chef API availability, TCP port reachability, and SSL certificate expiry.

What You'll Build

  • A monitor on Chef Infra Server's /_status endpoint to catch service failures
  • An HTTP monitor on the Chef API for authentication layer health
  • A TCP port check for HTTPS connectivity to the Chef API
  • SSL certificate monitoring for your Chef Infra Server domain
  • An alerting setup that distinguishes component failures from network issues

Prerequisites

  • A running Chef Infra Server (v14+) accessible via HTTPS
  • A domain pointing to Chef Infra Server (e.g., https://chef.example.com)
  • A free account at vigilmon.online

Step 1: Verify Chef Infra Server Health Endpoints

Chef Infra Server exposes a dedicated status endpoint for monitoring tools:

# Check the status endpoint (unauthenticated)
curl -s https://chef.example.com/_status
# Expected: JSON with "status": "pong" and component statuses

# Check HTTP status
curl -I https://chef.example.com/_status
# Expected: HTTP/2 200

# Inspect component health (shows PostgreSQL, Solr, RabbitMQ status)
curl -s https://chef.example.com/_status | python3 -m json.tool

A healthy Chef Infra Server returns JSON like:

{
  "status": "pong",
  "upstreams": {
    "chef_solr": "pong",
    "chef_expander": "pong",
    "chef_sql_db": "pong"
  }
}

If any upstream component is unhealthy, the corresponding key returns a failure message rather than "pong" — the HTTP status may still be 200, but the body reveals partial failures.

For the Chef API itself (requires authentication for most endpoints):

# The /organizations endpoint returns 401 for unauthenticated requests when healthy
curl -I https://chef.example.com/organizations
# Expected: HTTP/1.1 401 Unauthorized

Chef Manage (web UI): If you have Chef Manage installed, the web UI serves on the same port. The /_status endpoint is API-level, not web-UI-level — it does not verify Manage's availability. Add a separate monitor on /manage/login if you need to check the web interface.

Step 2: Create a Vigilmon HTTP Monitor for the Status Endpoint

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://chef.example.com/_status.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds (Chef's status endpoint aggregates upstream checks, which can take a few seconds).
  5. Expected status: 200.
  6. Keyword: pong (present in the response body when all components are healthy).
  7. Label: Chef Infra Server status.
  8. Click Save.

This monitor catches:

  • Chef Infra Server process failures (nginx or Erchef crashes)
  • PostgreSQL connection failures (cookbooks and node data become inaccessible)
  • Solr/OpenSearch failures (search API returns errors; knife search stops working)
  • RabbitMQ/queue failures (bifrost authorization failures)
  • Failed upgrades or migrations that leave components unhealthy

Alert sensitivity: Set alerts to trigger after 2 consecutive failures. The status endpoint aggregates multiple upstream checks, and brief upstream blips are possible during high-throughput periods.

Step 3: Create a Vigilmon HTTP Monitor for the Chef API

The /organizations endpoint is a lightweight test of the Chef API's authentication layer without requiring valid credentials:

  1. Add Monitor → HTTP.
  2. URL: https://chef.example.com/organizations.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 401.
  6. Label: Chef API auth.
  7. Click Save.

A 401 response to an unauthenticated request confirms that the Chef API (Erchef) is running and its authentication middleware is enforcing access control as expected. A 502, 503, or connection error means Erchef is down or the nginx frontend can't route to it.

When the API auth monitor fires but the /_status monitor stays green:

  • The nginx proxy routing to Erchef is misconfigured
  • Erchef crashed after the status endpoint was last checked
  • A Chef Infra Server upgrade partially completed, breaking the API endpoint mapping

Step 4: Create a TCP Monitor for the HTTPS Port

Chef clients (chef-client) connect to the server exclusively over HTTPS (port 443). If port 443 is blocked, every node's convergence run fails immediately:

# Verify HTTPS port is reachable
nc -zv chef.example.com 443
# or
curl -v telnet://chef.example.com:443
  1. Add Monitor → TCP.
  2. Host: chef.example.com.
  3. Port: 443.
  4. Check interval: 2 minutes.
  5. Response timeout: 10 seconds.
  6. Label: Chef Infra Server HTTPS port.
  7. Click Save.

When the TCP monitor fires but HTTP monitors were previously passing:

  • A firewall rule change blocked port 443
  • The nginx frontend crashed without taking the Erchef process with it
  • A routing change at your load balancer or cloud security group blocked external access

Step 5: Monitor SSL Certificates

Chef clients validate the Chef Infra Server's TLS certificate on every connection. An expired certificate causes:

  • Every chef-client run to fail with SSL_connect returned=1 errno=0 state=error: certificate verify failed
  • knife commands from workstations to fail
  • Bootstrap of new nodes to fail
  1. Add Monitor → SSL Certificate.
  2. Domain: chef.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Chef-managed certificates: Chef Infra Server can manage its own self-signed certificate, or you can configure a custom certificate in /etc/opscode/chef-server.rb via nginx['ssl_certificate'] and nginx['ssl_certificate_key']. Monitor whichever certificate your clients trust — if clients pin to a custom cert, that's the one to monitor.

Internal CA: If your clients trust an internal CA, the certificate may not be detected by external monitors as expiring correctly. In this case, also set up an internal alerting mechanism for the certificate used in nginx['ssl_certificate'].

Step 6: Monitor the Chef Infra Server's Search API

Chef's search API (powered by Solr/OpenSearch) is critical for policy-driven workflows — search in recipes and knife search both depend on it. A search API failure may not affect basic chef-client runs but breaks policy and compliance workflows:

# Search API returns 401 for unauthenticated requests when healthy
curl -I "https://chef.example.com/organizations/YOUR_ORG/search"
# Expected: HTTP/1.1 401
  1. Add Monitor → HTTP.
  2. URL: https://chef.example.com/organizations/YOUR_ORG/search.
  3. Check interval: 5 minutes.
  4. Expected status: 401.
  5. Label: Chef search API.
  6. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | /_status | Non-200 or pong missing | Run chef-server-ctl status; check component logs in /var/log/opscode/ | | API auth (/organizations) | Non-401 | Check chef-server-ctl status erchef; review nginx config | | TCP port 443 | Connection refused or timeout | Check firewall rules; verify chef-server-ctl status nginx | | SSL certificate | < 30 days to expiry | Renew TLS cert; update nginx['ssl_certificate'] in chef-server.rb if using custom cert | | Search API | Non-401 | Check chef-server-ctl status opscode-solr4; review Solr logs |

Alert after: 2 consecutive failures for HTTP monitors. 1 failure for TCP monitors — a TCP failure immediately means all chef-client runs are failing.

Common Chef Infra Server Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Erchef crash | /_status returns 500; API auth monitor fires | | PostgreSQL failure | /_status shows chef_sql_db unhealthy; cookbooks and nodes inaccessible | | Solr/OpenSearch failure | /_status shows chef_solr unhealthy; search API monitor fires | | nginx crash | TCP monitor fires; Erchef may still be running | | SSL certificate expires | SSL monitor alerts at 30-day threshold; all chef-client runs fail | | Disk full (cookbook store) | cookbook uploads fail; Erchef may return errors on cookbook endpoints | | Failed chef-upgrade | /_status may show partial component failures | | Firewall blocks port 443 | TCP monitor fires; HTTP monitors show connection errors | | DNS misconfiguration | All monitors fire simultaneously | | RabbitMQ/queue saturation | Authorization calls slow; /_status may show degraded bifrost |

Chef Infra Server failures are invisible until the next chef-client convergence cycle runs — by which time nodes may be drifted, compliance checks failed, and remediation policies undelivered. Vigilmon's external monitoring closes that gap: the /_status health endpoint reveals component-level failures within seconds, while TCP and SSL monitors catch network-layer and certificate issues that break every chef-client run before operators notice the drift.

Start monitoring Chef Infra Server in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles