Uptime monitoring for AWX

AWX is the open-source upstream project for Red Hat Ansible Automation Platform — a web UI for managing Ansible playbooks, inventories, credentials, and scheduled job templates across your infrastructure. DevOps and platform teams depend on AWX to orchestrate configuration management, provisioning, and deployment automation. When AWX goes down, scheduled Ansible jobs stop running and manual remediation workflows block. Worse, job failures may go unnoticed if they happen while the task manager is unhealthy.

This tutorial covers production-grade uptime monitoring for AWX using Vigilmon. We will walk through:

• Monitoring the AWX web UI availability
• Monitoring the /api/v2/ping/ health endpoint
• AWX task manager heartbeat monitoring
• SSL certificate alerts for the AWX web interface
• Heartbeat monitoring for scheduled AWX job templates and workflows

Prerequisites

• AWX 21+ (or Ansible Automation Platform 2.x for the same concepts)
• AWX deployed via the AWX Operator on Kubernetes, or standalone with Docker Compose
• Administrative access to the AWX web UI
• A free account at vigilmon.online

Part 1: Verify the AWX web UI and API are reachable

Before adding monitors, confirm AWX is serving the web interface and the API health endpoint is responding.

# Check the web UI responds
curl -sk -o /dev/null -w "%{http_code}" https://awx.example.com/
# Expected: 200

# Check the health ping endpoint (no authentication required)
curl -s https://awx.example.com/api/v2/ping/ | jq .

Expected response from /api/v2/ping/:

{
  "ha": false,
  "version": "23.5.0",
  "active_node": "awx-web-7c4f8d9b4d-xkrq2",
  "install_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "instances": [
    {
      "node": "awx-web-7c4f8d9b4d-xkrq2",
      "node_type": "hybrid",
      "uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "heartbeat": "2026-07-12T10:00:00.000000Z",
      "capacity": 51,
      "version": "23.5.0"
    }
  ]
}

The instances array and heartbeat field confirm the AWX task manager has recently reported in. If heartbeat is stale by more than a few minutes, the task manager is unhealthy even if the web UI loads.

Part 2: Monitor the AWX web UI availability

The web UI is your users' primary interface for launching jobs, reviewing playbook output, and managing inventories. Add an HTTP monitor that confirms users can reach the login page:

1. Log in to vigilmon.online and click Add Monitor.
2. Choose HTTP(S) monitor.
3. Enter: https://awx.example.com/
4. Set interval to 1 minute.
5. Set expected status code to 200.
6. Add a keyword check: must contain Ansible Automation (present in the AWX page title and meta).
7. Add your alert channel (email, Slack, webhook).
8. Click Save.

Vigilmon probes from multiple geographic regions. If the AWX web server stops responding — due to a crashed pod, failed ingress, or memory exhaustion — you are alerted within 60 seconds.

Part 3: Monitor the /api/v2/ping/ health endpoint

The /api/v2/ping/ endpoint is AWX's official health check. It requires no authentication, returns structured JSON, and reflects the health of both the web tier and the task manager backend. It is the most reliable indicator of AWX operational status.

Add the monitor:

1. Click Add Monitor → HTTP(S) monitor.
2. Enter: https://awx.example.com/api/v2/ping/
3. Set interval to 1 minute.
4. Set expected status code to 200.
5. Add a keyword check: must contain "version" (present only in a valid healthy response).
6. Add your alert channel.
7. Click Save.

Unlike the web UI check, this monitor catches backend failures that the web server survives — for example, the Django application tier crashing while the static frontend continues to load from cache.

Part 4: Task manager heartbeat monitoring

AWX's task manager (awx-task pod or awxd process) runs scheduled jobs, handles job dispatching, and manages Ansible execution environments. The /api/v2/ping/ response includes a heartbeat timestamp for each instance node. Use a probe script to check this timestamp and send a Vigilmon heartbeat only if the task manager is current.

Create the task manager health probe

#!/usr/bin/env bash
# /usr/local/bin/awx-task-heartbeat.sh

set -euo pipefail

AWX_URL="https://awx.example.com"
HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_TOKEN"
MAX_HEARTBEAT_AGE_SECONDS=300  # alert if task manager heartbeat is > 5 minutes old

# Fetch ping response (no auth required)
PING_RESPONSE=$(curl -fsS --max-time 10 "$AWX_URL/api/v2/ping/")

# Extract the most recent heartbeat timestamp
LAST_HEARTBEAT=$(echo "$PING_RESPONSE" | \
  python3 -c "
import sys, json, datetime
data = json.load(sys.stdin)
instances = data.get('instances', [])
if not instances:
    sys.exit(1)
times = [i.get('heartbeat', '') for i in instances if i.get('heartbeat')]
latest = max(times)
print(latest)
")

if [ -z "$LAST_HEARTBEAT" ]; then
  echo "[$(date)] ERROR: No heartbeat timestamp in AWX ping response" >&2
  exit 1
fi

# Check age of last heartbeat
HEARTBEAT_EPOCH=$(date -d "$LAST_HEARTBEAT" +%s 2>/dev/null || \
  python3 -c "import datetime; print(int(datetime.datetime.fromisoformat('${LAST_HEARTBEAT%Z}').replace(tzinfo=datetime.timezone.utc).timestamp()))")
NOW_EPOCH=$(date +%s)
AGE=$((NOW_EPOCH - HEARTBEAT_EPOCH))

if [ "$AGE" -gt "$MAX_HEARTBEAT_AGE_SECONDS" ]; then
  echo "[$(date)] ERROR: AWX task manager heartbeat is ${AGE}s old (max: ${MAX_HEARTBEAT_AGE_SECONDS}s)" >&2
  exit 1
fi

echo "[$(date)] AWX task manager healthy — last heartbeat ${AGE}s ago"
curl -fsS --retry 3 "$HEARTBEAT_URL" > /dev/null
echo "[$(date)] Heartbeat sent to Vigilmon"

Make it executable and schedule it:

chmod +x /usr/local/bin/awx-task-heartbeat.sh

# Add to crontab — run every 5 minutes
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/awx-task-heartbeat.sh >> /var/log/awx-heartbeat.log 2>&1") | crontab -

Create the heartbeat monitor in Vigilmon

1. In Vigilmon, click Add Monitor → Heartbeat monitor.
2. Name it: AWX task manager heartbeat.
3. Set expected interval to 5 minutes (matching your cron schedule).
4. Set grace period to 2 minutes.
5. Copy the heartbeat URL and paste it into the script's HEARTBEAT_URL variable.
6. Click Save.

If the awx-task pod crashes or the task manager stops checking in, the heartbeat stops. Vigilmon alerts you — and your scheduled Ansible jobs fail silently until you investigate.

Part 5: Heartbeat monitoring for scheduled AWX job templates

AWX job templates can be scheduled to run playbooks on a recurring basis — nightly patching, weekly compliance scans, hourly configuration drift checks. If a scheduled job is cancelled, times out, or silently fails with exit code 0 before completing its work, you want to know.

Add a Vigilmon heartbeat callback to AWX job templates

AWX supports Webhooks as post-job callbacks. Configure a Vigilmon heartbeat ping as a shell task at the end of your playbook:

# playbook/site.yml (excerpt)
- name: Notify Vigilmon on completion
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Send heartbeat to Vigilmon
      uri:
        url: "{{ lookup('env', 'VIGILMON_HEARTBEAT_URL') }}"
        method: GET
        status_code: 200
      when: lookup('env', 'VIGILMON_HEARTBEAT_URL') != ''

Set the VIGILMON_HEARTBEAT_URL environment variable in AWX:

1. In AWX, navigate to Resources → Credentials → Add.
2. Choose Custom Credential Type (or use an existing one).
3. Create a credential named Vigilmon Heartbeat with a field VIGILMON_HEARTBEAT_URL.
4. Assign this credential to your job template under Credentials.

Alternatively, add it directly to the AWX job template as an Extra Variable:

vigilmon_heartbeat_url: "https://vigilmon.online/api/heartbeat/YOUR_TOKEN"

And reference it in the playbook task:

- name: Send heartbeat to Vigilmon
  uri:
    url: "{{ vigilmon_heartbeat_url }}"
    method: GET
    status_code: 200

Create the heartbeat monitor in Vigilmon

1. In Vigilmon, click Add Monitor → Heartbeat monitor.
2. Name it based on the job template: AWX nightly patch run.
3. Set expected interval to match your AWX schedule (e.g., 24 hours for a nightly job).
4. Set grace period to 1 hour.
5. Copy the heartbeat URL and configure it in AWX.
6. Click Save.

Repeat for each critical scheduled job template. If the AWX scheduler skips a run, the job is manually cancelled, or the playbook fails before reaching the heartbeat task, Vigilmon alerts you.

Part 6: SSL certificate monitoring

AWX's web interface is served over HTTPS. An expired certificate prevents users from accessing the UI, blocks AWX API calls from your CI/CD pipelines, and prevents webhook callbacks from reaching AWX. For AWX Operator deployments, certificate rotation is managed by cert-manager, which can silently fail.

1. In Vigilmon, click Add Monitor → SSL monitor.
2. Enter your AWX hostname: awx.example.com.
3. Set alert threshold to 14 days before expiry.
4. Add your alert channel.
5. Click Save.

Check the current certificate expiry manually:

openssl s_client -connect awx.example.com:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

For AWX Operator deployments, check the certificate secret:

# List TLS secrets in the AWX namespace
kubectl get secrets -n awx --field-selector type=kubernetes.io/tls

# Check certificate expiry
kubectl get secret awx-ingress-tls -n awx \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -dates

For standalone AWX with Let's Encrypt:

# Check certbot renewal status
certbot certificates

# Test renewal dry-run
certbot renew --dry-run

Vigilmon's 14-day alert gives you time to manually renew if automation fails — before users lose access to the automation platform they depend on.

Part 7: Webhook alerts for AWX events

Route Vigilmon downtime events to your infrastructure team:

// Webhook receiver for AWX monitoring events
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhook/vigilmon', (req, res) => {
  const { monitor_name, status, url, checked_at, response_time_ms } = req.body;

  if (status === 'down') {
    console.error('[VIGILMON] CRITICAL: AWX monitor DOWN', {
      monitor: monitor_name,
      url,
      at: checked_at,
    });
    // AWX down = scheduled Ansible jobs stopped = infrastructure automation halted
    notifyInfraTeam({
      service: 'AWX',
      monitor: monitor_name,
      message: 'Ansible Automation Platform unavailable — scheduled jobs may be failing silently',
    });
  } else if (status === 'up') {
    console.info('[VIGILMON] AWX monitor recovered', {
      monitor: monitor_name,
      responseMs: response_time_ms,
    });
  }

  res.sendStatus(204);
});

app.listen(3000);

Or configure Vigilmon's native Slack integration to post AWX downtime alerts to your #infra-ops or #automation channel.

Summary

Your AWX deployment now has five layers of external monitoring:

1. Web UI check — confirms the AWX login page is serving correctly with a keyword check for the page title.
2. API ping check — polls /api/v2/ping/ to confirm the Django application tier is healthy and returning structured data.
3. Task manager heartbeat — a probe script checks the task manager's reported heartbeat age and pings Vigilmon only when healthy.
4. Job template heartbeat — scheduled AWX playbooks ping Vigilmon on success; missed pings mean a job was skipped or failed before completion.
5. SSL certificate monitor — alerts you 14 days before the AWX web certificate expires and blocks user access.

Vigilmon handles check scheduling, multi-region polling, alert routing, and uptime history. You get notified within 60 seconds of any failure in your AWX automation platform — before a silent job failure becomes an infrastructure incident.

Monitor your AWX infrastructure free at vigilmon.online

#awx #ansible #automation #monitoring #devops #kubernetes