Uptime monitoring for Icinga monitoring platform

Icinga is a mature, enterprise-grade monitoring platform trusted by organizations worldwide to watch their entire infrastructure. With its distributed architecture, the Icinga 2 daemon handles check execution while Icinga Web 2 provides the dashboard — and both layers need to be healthy for your monitoring system to function. When Icinga goes down, you go blind across your entire monitored estate.

This tutorial covers production-grade uptime monitoring for Icinga using Vigilmon. We will walk through:

• Monitoring the Icinga Web 2 UI availability
• Monitoring the Icinga 2 API health endpoint
• TCP port monitoring for the Icinga 2 daemon
• SSL certificate monitoring for HTTPS
• Heartbeat monitoring for Icinga scheduled check intervals and notification pipelines
• Webhook alerts for DOWN/UP events

Prerequisites

• Icinga 2 daemon running (version 2.13 or later) with the API feature enabled
• Icinga Web 2 running on Apache or Nginx
• A free account at vigilmon.online

Part 1: Verify the Icinga health endpoints

Icinga exposes several URLs and ports for external health monitoring:

Icinga Web 2 UI — the web frontend at /icingaweb2/ (or your configured prefix). A 200 response with the login page confirms Apache/Nginx and the PHP layer are working.

Icinga 2 API /v1/status — the Icinga 2 daemon API returns comprehensive status information including check statistics, notification counts, and process health. Requires Basic Auth or client certificate authentication.

TCP port 5665 — the Icinga 2 cluster port. Open and accepting connections confirms the daemon process is running, even when the API layer has issues.

# Test Icinga Web 2 login page
curl -o /dev/null -w "%{http_code}" https://icinga.example.com/icingaweb2/

# Test Icinga 2 API status (expects 401 without credentials — means API is up)
curl -k -o /dev/null -w "%{http_code}" https://icinga.example.com:5665/v1/status

# Test Icinga 2 API with credentials
curl -k -u apiuser:apipassword \
  -H "Accept: application/json" \
  https://icinga.example.com:5665/v1/status | jq '.results[0].name'

# Test TCP port 5665
nc -zv icinga.example.com 5665

Part 2: Monitor Icinga Web 2 UI

1. Log in to vigilmon.online and click Add Monitor.
2. Choose HTTP(S) monitor.
3. Enter: https://icinga.example.com/icingaweb2/
4. Set interval to 1 minute.
5. Add a keyword check: must contain Icinga (present in the page title and login form).
6. Add your alert channel.
7. Click Save.

The keyword check guards against reverse proxy errors that return 200 with a generic error page rather than the actual Icinga Web 2 login.

Part 3: Monitor the Icinga 2 API health endpoint

The Icinga 2 REST API (/v1/status) exercises the daemon health, check scheduling state, and notification pipeline. Create a dedicated API user for external monitoring:

# /etc/icinga2/conf.d/api-users.conf
object ApiUser "vigilmon" {
  password = "your-strong-password-here"
  permissions = [ "status/query" ]
}

Reload Icinga 2 to apply:

systemctl reload icinga2

Then add the monitor in Vigilmon:

1. In Vigilmon, click Add Monitor.
2. Choose HTTP(S) monitor.
3. Enter: https://icinga.example.com:5665/v1/status
4. Set interval to 1 minute.
5. Add a custom header: Authorization: Basic dmln... (Base64 of vigilmon:yourpassword).
6. Add a custom header: Accept: application/json.
7. Add a keyword check: must contain "results" (always present in a valid API response).
8. Add your alert channel.
9. Click Save.

This gives you a deep health signal — the API confirms the Icinga 2 daemon is running, processing checks, and serving the API layer correctly.

Part 4: TCP port monitoring for the Icinga 2 daemon

The Icinga 2 cluster daemon listens on TCP port 5665. This port is used for cluster node communication and for the API. Monitoring the raw TCP port catches cases where the daemon crashes but the API health check times out rather than returning a clean error.

1. In Vigilmon, click Add Monitor.
2. Choose TCP monitor (or Port monitor).
3. Enter host: icinga.example.com, port: 5665.
4. Set interval to 1 minute.
5. Add your alert channel.
6. Click Save.

Name this monitor Icinga 2 daemon TCP 5665 to distinguish it from the API layer check.

Part 5: SSL certificate monitoring

Icinga 2 uses its own PKI for cluster communication, and Icinga Web 2 typically runs behind a web server with a Let's Encrypt or enterprise CA certificate. Both need monitoring.

Monitor the web UI certificate

1. In Vigilmon, click Add Monitor.
2. Choose SSL monitor.
3. Enter: icinga.example.com (the web UI hostname, port 443).
4. Set alert threshold to 14 days before expiry.
5. Add your alert channel.

Monitor the Icinga 2 API certificate

The Icinga 2 daemon uses its own certificate on port 5665. Add a second SSL monitor:

1. In Vigilmon, click Add Monitor.
2. Choose SSL monitor.
3. Enter: icinga.example.com:5665.
4. Set alert threshold to 14 days before expiry.
5. Add your alert channel.

The Icinga 2 CA-issued certificate defaults to a 15-year validity, but if you rotate it or use shorter-lived certificates, the SSL monitor catches expiry before it disrupts cluster communication.

Part 6: Heartbeat monitoring for check intervals and notification pipelines

Icinga 2's check scheduler runs service checks at configured intervals. If the daemon enters a degraded state (exhausted worker threads, excessive check latency, or database connection failure with IDO), checks may queue but not execute. The web UI remains reachable while the check pipeline is effectively stalled.

Heartbeat monitoring provides an external, independent signal that checks are actually executing.

How heartbeat monitoring works

A heartbeat monitor expects a ping from your system at a defined interval. If the ping does not arrive within the grace period, Vigilmon fires an alert. Your Icinga 2 check script or notification command sends the ping at the end of a successful cycle.

Create a heartbeat monitor for check scheduling

1. In Vigilmon, click Add Monitor.
2. Choose Heartbeat monitor.
3. Name it Icinga 2 check scheduler.
4. Set interval to 5 minutes.
5. Set grace period to 3 minutes.
6. Save and copy the heartbeat URL (format: https://vigilmon.online/ping/hb_xxxxxxxxxx).

Instrument Icinga 2 with a command check

Create an Icinga 2 check command that queries the API check statistics and pings Vigilmon:

#!/bin/bash
# /etc/icinga2/scripts/vigilmon-heartbeat.sh

ICINGA_API_URL="https://localhost:5665/v1/status"
ICINGA_USER="vigilmon"
ICINGA_PASS="your-strong-password-here"
HEARTBEAT_URL="https://vigilmon.online/ping/hb_xxxxxxxxxx"

# Query check statistics from Icinga 2 API
RESPONSE=$(curl -sk -u "$ICINGA_USER:$ICINGA_PASS" \
  -H "Accept: application/json" \
  "$ICINGA_API_URL")

if echo "$RESPONSE" | grep -q '"results"'; then
  # Extract active checks executed in the last minute
  curl -s "$HEARTBEAT_URL" > /dev/null
  echo "OK: Icinga 2 API healthy, heartbeat sent"
  exit 0
else
  echo "CRITICAL: Icinga 2 API did not return expected results"
  exit 2
fi

Add a cron job to run this check independently of Icinga 2 itself:

*/5 * * * *  icinga  /etc/icinga2/scripts/vigilmon-heartbeat.sh >> /var/log/icinga2/vigilmon-heartbeat.log 2>&1

Create a heartbeat monitor for the notification pipeline

Add a second heartbeat monitor to verify Icinga notifications are being processed. Instrument a low-severity test notification at a regular interval:

#!/bin/bash
# /usr/local/bin/icinga-notification-heartbeat.sh
# Sends a test notification and verifies it was processed

HEARTBEAT_URL="https://vigilmon.online/ping/hb_yyyyyyyyyy"
ICINGA_API_URL="https://localhost:5665/v1/actions/send-custom-notification"
ICINGA_USER="vigilmon"
ICINGA_PASS="your-strong-password-here"

RESPONSE=$(curl -sk -u "$ICINGA_USER:$ICINGA_PASS" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{"type": "Host", "filter": "host.name==\"localhost\"", "author": "vigilmon", "comment": "Heartbeat test notification"}' \
  "$ICINGA_API_URL")

if echo "$RESPONSE" | grep -q '"results"'; then
  curl -s "$HEARTBEAT_URL" > /dev/null
  echo "$(date): Notification pipeline healthy"
fi

Part 7: Webhook alerts

Integrate Vigilmon DOWN/UP events with your team's alerting channels:

# Example: shell-based webhook receiver (for teams using simple HTTP endpoints)
#!/bin/bash
# /usr/local/bin/vigilmon-receiver.sh
# Called by a simple HTTP server (e.g. nc or a minimal web framework)

PAYLOAD=$(cat)
MONITOR=$(echo "$PAYLOAD" | jq -r '.monitor_name')
STATUS=$(echo "$PAYLOAD" | jq -r '.status')
URL=$(echo "$PAYLOAD" | jq -r '.url')

if [ "$STATUS" = "down" ]; then
  # Send to your alerting system
  curl -s -X POST https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
    -H "Content-Type: application/json" \
    -d "{\"text\": \":red_circle: *$MONITOR* is DOWN — $URL\"}"
elif [ "$STATUS" = "up" ]; then
  curl -s -X POST https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
    -H "Content-Type: application/json" \
    -d "{\"text\": \":large_green_circle: *$MONITOR* recovered — $URL\"}"
fi

Vigilmon sends this payload structure:

{
  "monitor_id": "mon_abc123",
  "monitor_name": "Icinga 2 API /v1/status",
  "status": "down",
  "url": "https://icinga.example.com:5665/v1/status",
  "checked_at": "2026-06-30T10:00:00Z",
  "response_code": 0,
  "response_time_ms": 30000
}

Part 8: Monitor summary table

| Monitor | URL / Target | Type | Interval | |---------|-------------|------|----------| | Icinga Web 2 UI | https://icinga.example.com/icingaweb2/ | HTTP | 1 min | | Icinga 2 API /v1/status | https://icinga.example.com:5665/v1/status | HTTP | 1 min | | Icinga 2 daemon TCP | icinga.example.com:5665 | TCP | 1 min | | SSL certificate (web) | icinga.example.com:443 | SSL | Daily | | SSL certificate (API) | icinga.example.com:5665 | SSL | Daily | | Check scheduler heartbeat | heartbeat URL | Heartbeat | 5 min | | Notification pipeline heartbeat | heartbeat URL | Heartbeat | 15 min |

Summary

Your Icinga deployment now has four layers of monitoring:

1. Web UI monitor — confirms Icinga Web 2 is reachable and serving the actual login page, polled every 60 seconds.
2. API and TCP monitors — exercise the Icinga 2 daemon directly, catching daemon failures and API layer issues independently.
3. SSL monitors — alert you 14 days before certificates expire on both the web UI and the Icinga 2 cluster API port.
4. Heartbeat monitors — verify that the check scheduler and notification pipeline are executing on schedule, catching silent failures in the check engine that the web UI would never surface.

Vigilmon handles check scheduling, multi-region polling, alert routing, and uptime history. Your Icinga installation gets independent monitoring coverage — so the platform that watches everything else is also being watched.

Monitor your Icinga deployment free at vigilmon.online

#icinga #nagios #monitoring #devops #infrastructure #itops