How to Monitor CodeRabbit with Vigilmon

CodeRabbit is an AI-powered code review platform that automatically reviews pull requests with context-aware, line-by-line feedback. It detects security vulnerabilities, suggests documentation improvements, and learns from your team's reactions over time — integrated directly into GitHub, GitLab, and Azure DevOps. When CodeRabbit's webhook pipeline breaks, PRs merge without review summaries and your team loses its AI reviewer without any indication that something went wrong. Vigilmon lets you monitor the CodeRabbit integration so silent failures don't slip past your review process.

What You'll Set Up

• HTTP uptime monitor for the CodeRabbit API
• Cron heartbeat to verify webhook delivery end-to-end
• SSL certificate monitoring for self-hosted CodeRabbit deployments
• Alerting for integration health

Prerequisites

• CodeRabbit configured on your GitHub, GitLab, or Azure DevOps organization
• Access to CodeRabbit's configuration or a self-hosted deployment
• A free Vigilmon account

Why Monitor CodeRabbit?

CodeRabbit sits in the critical path of your pull request workflow. Failures are easy to miss:

• Webhook delivery failures mean new PRs don't trigger reviews — the PR sits open with no bot comment, which developers may interpret as "no issues found."
• API quota exhaustion causes CodeRabbit to silently skip reviews rather than queue them.
• Self-hosted deployment crashes leave the GitHub App integration hanging with no response.
• Configuration drift (e.g. a rotated API key, a changed repository setting) can disable reviews on specific repos without affecting others.

Monitoring ensures your team knows immediately when CodeRabbit stops participating in your review process.

Key Metrics to Watch

| Signal | What It Means | |---|---| | CodeRabbit API /health HTTP 200 | Service is up (self-hosted) | | Webhook endpoint reachable | GitHub can deliver PR events | | Review heartbeat | A test PR actually receives a review | | SSL certificate | TLS valid for webhook delivery |

Step 1: Monitor the CodeRabbit API (Self-Hosted)

For self-hosted CodeRabbit deployments, add a health check monitor:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your CodeRabbit instance URL: https://coderabbit.yourdomain.com/health
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

For cloud-hosted CodeRabbit (coderabbit.ai), you can monitor the service status page URL instead. The important thing is that you have an independent alert when the service is unreachable from your network.

Step 2: Monitor the Webhook Receiver Endpoint

CodeRabbit receives webhook events from GitHub/GitLab/Azure DevOps when PRs are opened or updated. If the webhook endpoint is down, all review triggers are lost. Monitor it directly:

1. Add a new HTTP / HTTPS monitor.
2. Enter the webhook receiver URL — for self-hosted this is typically https://coderabbit.yourdomain.com/webhook.
3. Set Method to GET with Expected HTTP status 200 or 405 (Method Not Allowed is a valid response for a POST-only endpoint, confirming the server is running).
4. Set Check interval to 1 minute.

This confirms the endpoint is reachable by GitHub's webhook delivery infrastructure, not just from inside your network.

Step 3: Cron Heartbeat for End-to-End Review Verification

A process-level monitor confirms CodeRabbit is running, but it doesn't verify that reviews are actually being posted to PRs. Set up a scheduled verification job that opens a test PR and confirms CodeRabbit comments:

#!/bin/bash
# verify-coderabbit.sh — run every 4 hours via cron

# Create a minimal test PR using the GitHub CLI
PR_URL=$(gh pr create \
  --repo yourorg/monitor-test-repo \
  --title "CodeRabbit health check $(date +%Y%m%d%H%M)" \
  --body "Automated health check PR" \
  --base main \
  --head health-check-branch 2>/dev/null)

if [ -z "$PR_URL" ]; then
  echo "Failed to create test PR"
  exit 1
fi

PR_NUMBER=$(echo "$PR_URL" | grep -o '[0-9]*$')

# Wait for CodeRabbit to post its review (up to 3 minutes)
for i in $(seq 1 18); do
  sleep 10
  COMMENT_COUNT=$(gh pr view "$PR_NUMBER" \
    --repo yourorg/monitor-test-repo \
    --json comments \
    --jq '[.comments[] | select(.author.login == "coderabbitai")] | length')

  if [ "$COMMENT_COUNT" -gt "0" ]; then
    # Close the test PR
    gh pr close "$PR_NUMBER" --repo yourorg/monitor-test-repo
    # Ping Vigilmon heartbeat
    curl -s https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID
    exit 0
  fi
done

# CodeRabbit didn't comment within 3 minutes
gh pr close "$PR_NUMBER" --repo yourorg/monitor-test-repo
echo "CodeRabbit did not respond"
exit 1

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 240 minutes (matching the 4-hour cron schedule).
3. Copy the heartbeat URL into the script.

This end-to-end check confirms that webhook delivery, authentication, and PR commenting are all working together.

Step 4: Monitor Per-Repository Webhook Health

If CodeRabbit is configured on multiple repositories, webhook delivery failures are often per-repo (a misconfigured secret, a disabled integration). Use the GitHub API to check webhook delivery status in a scheduled job:

#!/bin/bash
# check-webhook-deliveries.sh

REPOS=("yourorg/repo-one" "yourorg/repo-two" "yourorg/repo-three")

for REPO in "${REPOS[@]}"; do
  # Get recent webhook deliveries for this repo
  FAILURES=$(gh api \
    "repos/$REPO/hooks" \
    --jq '.[].id' | \
    xargs -I{} gh api "repos/$REPO/hooks/{}/deliveries" \
    --jq '[.[] | select(.status_code != 200)] | length')

  if [ "$FAILURES" -gt "0" ]; then
    echo "Webhook failures on $REPO: $FAILURES"
    # Don't ping heartbeat — alert will fire from missing ping
    exit 1
  fi
done

curl -s https://vigilmon.online/heartbeat/YOUR_WEBHOOK_HEALTH_HEARTBEAT_ID

Add a second Cron Heartbeat in Vigilmon with a 60-minute interval for this job.

Step 5: SSL Certificate Monitoring (Self-Hosted)

GitHub and GitLab require HTTPS webhook endpoints with valid certificates. A certificate expiry immediately breaks all webhook delivery:

1. Open the HTTP monitor for your CodeRabbit instance (from Step 1).
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

Since webhook failures cause silent review gaps rather than visible errors, certificate expiry monitoring provides an early-warning system for a category of outage that might otherwise go unnoticed for days.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your engineering Slack workspace.
2. For the API health monitor, set Consecutive failures before alert to 2.
3. For the cron heartbeat monitors, set Consecutive failures before alert to 1 — a missed review verification cycle should page immediately.
4. Route alerts to the channel where PRs are announced, so developers see the "CodeRabbit is down" alert alongside their PR notifications.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health | CodeRabbit API /health | Service down (self-hosted) | | HTTP webhook | /webhook endpoint | Webhook receiver unreachable | | Cron heartbeat | End-to-end PR review test | Full pipeline broken | | Cron heartbeat | Per-repo webhook delivery | Per-repository integration failure | | SSL certificate | CodeRabbit domain | TLS expiry breaking webhook delivery |

CodeRabbit improves code quality silently and continuously — which means its failures are equally silent. A Vigilmon setup that tests the full review pipeline end-to-end gives you confidence that every pull request in your organization is getting the AI review your team depends on.