How to Monitor Sweep AI with Vigilmon

Sweep AI is an AI-powered junior developer that turns GitHub issues into pull requests automatically. Label an issue sweep, and Sweep reads the codebase, plans the implementation, writes the code, runs tests, and opens a PR — no human intervention required. It supports Python, TypeScript, and most major languages. When the Sweep integration fails, labeled issues accumulate without action and the silence is indistinguishable from "Sweep is thinking." Vigilmon monitors the Sweep pipeline so you know immediately when the automation is broken rather than busy.

What You'll Set Up

• HTTP uptime monitor for the Sweep API or self-hosted deployment
• Cron heartbeat to verify end-to-end issue-to-PR automation
• SSL certificate monitoring
• Webhook delivery monitoring for the GitHub integration

Prerequisites

• Sweep AI configured on your GitHub organization or repository
• GitHub CLI (gh) installed for the heartbeat verification script
• A free Vigilmon account

Why Monitor Sweep AI?

Sweep is async and autonomous — it takes time to produce a PR, and failures look like delays:

• GitHub App webhook failures mean labeled issues never reach Sweep — they look like Sweep is ignoring them.
• LLM API quota exhaustion causes Sweep to stall mid-implementation without closing the issue or posting a failure comment.
• Self-hosted deployment crashes leave the GitHub App integration connected but unresponsive.
• Repository permission changes (removed collaborator access, branch protection changes) cause Sweep to fail when it tries to push a branch.

Without monitoring, these failure modes can go undetected for days — especially for lower-priority issues where a delayed PR raises no alarms.

Key Metrics to Watch

| Signal | What It Means | |---|---| | Sweep API /health HTTP 200 | Service is reachable | | Webhook endpoint HTTP 200/405 | GitHub can deliver issue events | | Issue-to-PR heartbeat | Full automation pipeline working | | PR creation latency | How long Sweep takes to respond | | SSL certificate | TLS valid for webhook delivery |

Step 1: Monitor the Sweep Health Endpoint

For self-hosted Sweep deployments:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Sweep instance URL: https://sweep.yourdomain.com/health
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

For cloud-hosted Sweep (sweep.dev), monitor the service's status URL instead. This gives you an alert if Sweep becomes unreachable from your network or if the cloud service is degraded.

Step 2: Monitor the GitHub Webhook Endpoint

Sweep receives GitHub issue events via webhook. If the webhook endpoint is unreachable, no issues will trigger automation regardless of how many you label:

1. Add a new HTTP / HTTPS monitor.
2. Enter the Sweep webhook URL — for self-hosted deployments this is typically https://sweep.yourdomain.com/webhook/github.
3. Set Method to GET.
4. Set Expected HTTP status to 200 or 405 (a POST-only webhook endpoint legitimately returns 405 for GET).
5. Set Check interval to 1 minute.

Confirm the webhook is also registered and active in your GitHub organization settings under Settings → Webhooks. A reachable endpoint with a deregistered webhook still won't receive events.

Step 3: Cron Heartbeat for End-to-End PR Automation

The most valuable monitor for Sweep is an end-to-end test that verifies the full issue-to-PR pipeline. Set up a scheduled job that creates a test issue, labels it for Sweep, and confirms a PR appears:

#!/bin/bash
# verify-sweep.sh — run every 6 hours via cron

REPO="yourorg/monitor-test-repo"
ISSUE_TITLE="[Sweep health check] Add a comment to health_check.py $(date +%Y%m%d%H%M)"

# Create a simple, low-cost test issue
ISSUE_URL=$(gh issue create \
  --repo "$REPO" \
  --title "$ISSUE_TITLE" \
  --body "Add a single-line comment '# health check' to health_check.py" \
  --label "sweep")

if [ -z "$ISSUE_URL" ]; then
  echo "Failed to create test issue"
  exit 1
fi

ISSUE_NUMBER=$(echo "$ISSUE_URL" | grep -o '[0-9]*$')

echo "Created issue #$ISSUE_NUMBER, waiting for Sweep PR..."

# Wait up to 10 minutes for Sweep to open a PR
for i in $(seq 1 60); do
  sleep 10

  PR_COUNT=$(gh pr list \
    --repo "$REPO" \
    --search "sweep $ISSUE_TITLE" \
    --json number \
    --jq 'length')

  if [ "$PR_COUNT" -gt "0" ]; then
    # Close the test issue
    gh issue close "$ISSUE_NUMBER" --repo "$REPO"
    # Ping Vigilmon
    curl -s https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID
    echo "Sweep responded successfully"
    exit 0
  fi
done

# Sweep didn't open a PR within 10 minutes
gh issue close "$ISSUE_NUMBER" --repo "$REPO" --comment "Health check timed out — Sweep did not respond."
echo "Sweep did not open a PR within 10 minutes"
exit 1

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 360 minutes (6 hours).
3. Copy the heartbeat URL into the script.

The 10-minute timeout in the script is intentionally generous — Sweep can take several minutes to plan and implement even a small change. Adjust it based on your observed typical response time.

Step 4: Monitor PR Creation Latency

Beyond a binary pass/fail, you may want to track how long Sweep takes to respond to issues. Add a second heartbeat job that records latency:

#!/bin/bash
# measure-sweep-latency.sh

REPO="yourorg/monitor-test-repo"
START=$(date +%s)

# Create issue and wait for PR (same pattern as above)
# ...

END=$(date +%s)
ELAPSED=$((END - START))

echo "Sweep responded in ${ELAPSED}s"

# Ping heartbeat only if response was within SLA (e.g. 8 minutes)
if [ "$ELAPSED" -lt "480" ]; then
  curl -s https://vigilmon.online/heartbeat/YOUR_LATENCY_HEARTBEAT_ID
fi

This heartbeat goes silent if Sweep is responding but too slowly — catching degraded performance before it becomes a full outage.

Step 5: SSL Certificate Monitoring

GitHub requires valid HTTPS certificates for webhook delivery. A certificate expiry on your self-hosted Sweep instance immediately breaks all issue automation:

1. Open the HTTP monitor for your Sweep instance (from Step 1).
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

If Sweep runs behind a reverse proxy (nginx, Caddy, Traefik), the certificate monitor should target the Sweep subdomain specifically, not the proxy's top-level domain.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and connect your engineering Slack workspace.
2. For the API health monitor, set Consecutive failures before alert to 2.
3. For the end-to-end heartbeat, set Consecutive failures before alert to 1 — a missed automation cycle is an immediate issue since it means multiple labeled issues may already be stuck.
4. Route alerts to the same channel where GitHub PR notifications appear, so the team sees both "PR from Sweep" and "Sweep is down" in one place.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health | Sweep API /health | Service down | | HTTP webhook | Sweep /webhook/github | Webhook receiver unreachable | | Cron heartbeat (pass/fail) | Full issue-to-PR pipeline | Automation broken end-to-end | | Cron heartbeat (latency) | PR response time SLA | Sweep degraded but not down | | SSL certificate | Sweep domain | TLS expiry breaking webhook delivery |

Sweep AI multiplies your team's throughput by handling routine issues automatically. Its value depends entirely on the pipeline running reliably — and that pipeline is invisible when it works and equally invisible when it breaks. Vigilmon's external monitoring closes that gap, giving you the same confidence in your AI developer that you'd have in any other piece of critical infrastructure.