PR-Agent (open-source edition of Qodo Merge) is an AI-powered pull request review tool that automatically reviews PRs, suggests improvements, generates PR descriptions, and answers questions about code changes. Self-hosted as a FastAPI webhook server, it integrates with GitHub, GitLab, and Bitbucket — but when the webhook server goes down, PR reviews simply stop happening. Silently.
In this tutorial you'll set up monitoring for a self-hosted PR-Agent deployment using Vigilmon — covering the webhook server, LLM provider health, webhook delivery, review latency, and TLS certificate expiry.
Why monitoring PR-Agent matters
PR-Agent is a webhook-driven service: GitHub sends a webhook event when a PR is opened, and PR-Agent responds by posting a review comment. This flow has multiple points of failure:
- Webhook server down — GitHub sends the webhook; it gets a connection error; the PR review never happens; developers submit PRs and wait for reviews that will never come
- LLM API failure — PR-Agent receives the webhook but can't reach GPT-4/Claude/Ollama; the review silently fails after retries; no error appears in the PR timeline
- Webhook delivery failures — GitHub marks the webhook as failed after 3 timeouts; automatic retries stop; new PRs get no review until the delivery is manually retried
- Review latency spike — under high load or LLM API slowness, review comments take 5+ minutes to post; developers don't know whether to wait or proceed
- TLS certificate expiry — GitHub can't deliver webhooks to an expired HTTPS endpoint; webhook deliveries fail with SSL errors; the failure is logged in GitHub but invisible to developers
External monitoring from Vigilmon catches all of these before your team notices PR-Agent has gone quiet.
What you'll need
- PR-Agent running as a webhook server (port 3000 by default)
- A GitHub/GitLab/Bitbucket repository with webhooks configured to point at your server
- A free Vigilmon account
Step 1: Verify the webhook server health endpoint
PR-Agent's FastAPI server exposes a health endpoint by default. Confirm it's accessible:
# Health check
curl -s http://your-pr-agent-host:3000/health
# Expected: {"status":"ok"} or HTTP 200
# Alternatively probe the root path
curl -s -o /dev/null -w "%{http_code}" http://your-pr-agent-host:3000/
# Expected: 200 or 404 (server alive either way)
If PR-Agent doesn't expose /health in your version, you can add one by extending the FastAPI app in pr_agent/servers/github_app.py:
@app.get("/health")
async def health():
return {"status": "ok", "version": "pr-agent"}
Step 2: Monitor the webhook server (port 3000)
The webhook server is the most critical component. If it's down, no PR receives a review.
- Log in to vigilmon.online and go to Monitors → New Monitor
- Choose HTTP / HTTPS
- URL:
https://your-pr-agent-host:3000/health - Check interval: 1 minute
- Expected status:
200 - Timeout: 10 seconds
- Save the monitor
This is your primary alert. A 1-minute check interval means you detect server crashes within 60 seconds.
Step 3: Monitor TLS certificate expiry
PR-Agent's webhook endpoint must have a valid TLS certificate — GitHub, GitLab, and Bitbucket reject webhook deliveries to endpoints with expired or self-signed certificates.
- Go to Monitors → New Monitor
- Choose SSL Certificate
- Enter your PR-Agent hostname (e.g.
pr-agent.example.com) - Alert threshold: 30 days before expiry
- Save the monitor
Vigilmon will alert you at 30 days, 14 days, and 7 days before expiry — well before GitHub starts rejecting your webhooks.
Step 4: Monitor LLM provider connectivity
PR-Agent depends on GPT-4, Claude, or a self-hosted Ollama instance for review reasoning. Add a connectivity probe alongside the PR-Agent process:
# llm_health_server.py
import os, requests, time
from fastapi import FastAPI
app = FastAPI()
@app.get("/llm-health")
def llm_health():
provider = os.environ.get("CONFIG__AI__PROVIDER", "openai")
start = time.time()
try:
if provider == "openai":
key = os.environ["OPENAI_API_KEY"]
r = requests.get(
"https://api.openai.com/v1/models",
headers={"Authorization": f"Bearer {key}"},
timeout=10,
)
elif provider == "anthropic":
key = os.environ["ANTHROPIC_API_KEY"]
r = requests.get(
"https://api.anthropic.com/v1/models",
headers={"x-api-key": key, "anthropic-version": "2023-06-01"},
timeout=10,
)
elif provider == "ollama":
base = os.environ.get("OLLAMA_API_BASE", "http://localhost:11434")
r = requests.get(f"{base}/api/tags", timeout=10)
else:
return {"status": "unknown_provider", "provider": provider}
latency_ms = int((time.time() - start) * 1000)
return {
"status": "ok" if r.status_code == 200 else "error",
"provider": provider,
"latency_ms": latency_ms,
"http_status": r.status_code,
}
except Exception as e:
return {"status": "error", "provider": provider, "reason": str(e)}
Run on port 3001, then add an HTTP monitor in Vigilmon:
- URL:
http://your-host:3001/llm-health - Expected body contains:
"status":"ok" - Check interval: 5 minutes
Step 5: Monitor review latency
PR-Agent's value depends on posting review comments quickly. If reviews take more than 2 minutes, developers either submit before reading feedback or start complaining. Add a synthetic latency probe:
# Add to llm_health_server.py
import json, pathlib, time
# PR-Agent can write a latency log when ENABLE_REVIEW_LATENCY_LOG=true
LATENCY_LOG = pathlib.Path(os.environ.get("PR_AGENT_LATENCY_LOG", "/tmp/pr_agent_latency.json"))
@app.get("/review-latency")
def review_latency():
if not LATENCY_LOG.exists():
return {"status": "ok", "message": "no_latency_log"}
entries = json.loads(LATENCY_LOG.read_text())
if not entries:
return {"status": "ok", "p50_ms": None, "p95_ms": None}
recent = sorted([e["latency_ms"] for e in entries[-100:]])
p50 = recent[len(recent) // 2]
p95 = recent[int(len(recent) * 0.95)]
# Alert if P95 exceeds 2 minutes (120,000ms)
return {
"status": "degraded" if p95 > 120000 else "ok",
"p50_ms": p50,
"p95_ms": p95,
}
Add an HTTP monitor for /review-latency — alert when "status":"degraded".
Step 6: Monitor GitHub/GitLab API rate limits
PR-Agent makes multiple GitHub API calls per review: reading PR diffs, posting comments, and updating labels. Rate limit exhaustion causes reviews to fail silently.
# Add to llm_health_server.py
@app.get("/github-rate-limit")
def github_rate_limit():
token = os.environ.get("GITHUB_TOKEN", "")
try:
r = requests.get(
"https://api.github.com/rate_limit",
headers={"Authorization": f"token {token}"} if token else {},
timeout=10,
)
data = r.json()
remaining = data["rate"]["remaining"]
return {
"status": "degraded" if remaining < 100 else "ok",
"remaining": remaining,
"reset_at": data["rate"]["reset"],
}
except Exception as e:
return {"status": "error", "reason": str(e)}
Add an HTTP monitor for /github-rate-limit with a 5-minute check interval.
Step 7: Monitor configuration file health
PR-Agent uses .pr_agent.toml for per-repository configuration. A malformed config causes silent failures for that repository's reviews. Add a config validation check to your startup:
# Add to llm_health_server.py
import tomllib, glob
@app.get("/config-health")
def config_health():
config_files = glob.glob("/app/**/.pr_agent.toml", recursive=True)
errors = []
for f in config_files:
try:
with open(f, "rb") as fh:
tomllib.load(fh)
except Exception as e:
errors.append({"file": f, "error": str(e)})
return {
"status": "ok" if not errors else "degraded",
"configs_checked": len(config_files),
"errors": errors,
}
Add an HTTP monitor for /config-health with a 10-minute check interval — alert on degraded status.
Step 8: Configure alerting
- Go to Alert Channels → Add Channel
- Add Slack for your engineering team and email for the ops team
- Assign channels to monitors
Recommended routing:
| Monitor | Alert | Why | |---------|-------|-----| | Webhook server (port 3000) | Slack #oncall + Email | No PR reviews delivered | | TLS certificate | Email | GitHub rejects webhooks | | LLM provider | Slack #dev | Reviews fail silently | | Review latency | Slack #dev | Developer experience degraded | | GitHub rate limits | Slack #dev | API calls fail | | Config health | Slack #dev | Per-repo reviews silently broken |
Summary: PR-Agent monitor checklist
| Monitor | Type | Endpoint | Interval |
|---------|------|----------|----------|
| Webhook server | HTTP | :3000/health | 1 min |
| TLS certificate | SSL | hostname | Daily |
| LLM provider | HTTP | :3001/llm-health | 5 min |
| Review latency | HTTP | :3001/review-latency | 5 min |
| GitHub rate limits | HTTP | :3001/github-rate-limit | 5 min |
| Config health | HTTP | :3001/config-health | 10 min |
With these monitors in place you'll catch webhook server crashes, LLM outages, TLS expiry, and config errors before your team notices that PR reviews have stopped appearing.
What's next
- Webhook delivery monitoring — check your GitHub webhook delivery logs via the GitHub API and alert when the failure rate exceeds a threshold
- Status page — publish an internal status page for PR-Agent so your team can check operational status without pinging ops
Get started free at vigilmon.online — no credit card, monitors start running in under a minute.