When an alert fires at 2 AM, your on-call engineer shouldn't need to think hard. They should follow a document that tells them exactly what to check, what to try, who to escalate to, and what questions to answer after the incident. That document is the runbook — and most teams' runbooks are either missing, outdated, or too vague to help.
This guide covers what makes a runbook effective, how to structure it so it actually gets used during incidents, how to link runbooks to specific alerts in your monitoring configuration, and how to build a feedback loop that keeps runbooks accurate over time.
Why Most Runbooks Fail
The majority of engineering teams that have runbooks don't use them. Here's why:
They're too long. A 20-page document is a reference manual. Nobody reads reference manuals while their site is down. A useful runbook for an on-call scenario is 1–2 pages maximum — dense and decision-tree shaped.
They're not linked to alerts. If your on-call engineer has to search for the runbook after an alert fires, they've already lost a minute. Runbooks are only effective when they're one click away from the alert itself.
They go stale. A runbook written when a service was first deployed may reference hostnames that no longer exist, commands that no longer work, or people who no longer work there. Stale runbooks are worse than no runbooks — they waste time following dead ends.
They describe what you already know. "Check if the service is running" is not a runbook entry. The runbook should capture what a junior engineer would miss — the non-obvious, the gotchas, the tribal knowledge that lives in senior engineers' heads.
Anatomy of an Effective Runbook
An effective on-call runbook has five sections. Every alert that could wake someone up should have a runbook with all five.
1. Alert Summary (3–5 lines)
What does this alert mean? What service is affected? What's the user-visible impact?
The person reading this may have been woken up. Give them orienting context in under 30 seconds.
Example:
Alert: api-endpoint-down
Service: Public API gateway (api.myapp.com)
Impact: All API clients are receiving 5xx responses or connection timeouts.
Mobile app features that require API calls will fail. Dashboard will show errors.
SLA impact: Yes — this is in scope for our 99.9% uptime SLA.
2. Immediate Verification Steps
What should the engineer check first to confirm the alert is real and understand scope? These should be specific commands or links, not generic advice.
Example:
1. Check the Vigilmon status dashboard: [link]
- Is this multi-region (all probe locations failed)? → Genuine outage
- Is this single-region? → Possible probe-side issue, check once more in 2 minutes
2. Try the endpoint manually:
curl -I https://api.myapp.com/health
Expected: HTTP 200
If you get: connection refused → load balancer issue
If you get: HTTP 503 → backend services down
If you get: SSL error → certificate issue (see SSL runbook)
3. Check the load balancer health in AWS console:
[direct link to target group health check]
If backends show "unhealthy" → proceed to backend diagnosis
If backends show "healthy" but endpoint unreachable → DNS or CDN issue
The key: every step should produce a result that points to the next step or to a specific escalation path. Binary outcomes ("healthy / not healthy") are easier to act on than "check if things look okay."
3. Common Causes and Mitigations
List the most frequent root causes for this alert, in order of likelihood, with the mitigation for each. This is where institutional knowledge lives.
Example:
Cause A (most common — ~60% of occurrences): Deployment artifact issue
Signs: New deployment was running within the last 30 minutes
Verify: `git log --since="1 hour ago" -- deployments/api/`
Mitigation: Roll back to previous deployment
kubectl rollout undo deployment/api-gateway
Verify: Endpoint should recover within 60–90 seconds after rollback
Escalate to: Backend team lead to investigate the broken deployment
Cause B (~25%): Database connection pool exhausted
Signs: Pod logs show "too many connections" or connection timeout errors
Verify: kubectl logs -l app=api-gateway --tail=50 | grep -i "connection"
Check: RDS console → Monitoring → DatabaseConnections (alert if > 90% of max_connections)
Mitigation: Restart API gateway pods to release connections:
kubectl rollout restart deployment/api-gateway
Escalate after: If restart doesn't help within 5 min, escalate to DB team
Cause C (~15%): Memory pressure / OOMKilled
Signs: Pod status shows OOMKilled in recent events
Verify: kubectl describe pod -l app=api-gateway | grep -A5 "OOMKilled"
Mitigation: Restart affected pods:
kubectl delete pod -l app=api-gateway
This buys time. Escalate to Backend team lead — memory issue needs root cause fix.
Notice what these entries contain: specific commands, specific thresholds, specific escalation targets, and realistic time estimates. Vague entries like "check database health" or "look at logs" are not runbook content.
4. Escalation Chain
Who do you call if the mitigation steps don't work, or if you're in a situation not covered by the common causes?
Primary on-call: [linked to on-call rotation schedule]
Owns: Initial triage, Cause A and B mitigations
Escalate after: 15 minutes without resolution, or any Cause C situation
Backend team lead: [name, contact method, acceptable wake-up conditions]
Owns: Deployment issues, memory pressure, application-level root cause
Escalate after: 30 minutes without resolution, or if customer impact is confirmed
CTO / VP Engineering: [name, contact method]
Owns: Customer communication, SLA breach declaration, cross-team coordination
Wake: If estimated resolution > 30 minutes AND customer impact confirmed
Include acceptable wake-up thresholds for each level. Nothing burns out senior engineers faster than being woken up for issues that should have been resolved at L1.
5. Post-Incident Checklist
This section runs after the incident resolves. It exists to capture what happened while memory is fresh and to create accountability for follow-up work.
After resolution:
□ Document the timeline in the incident ticket (first alert, first response, resolution)
□ Identify the root cause category (A / B / C / other)
□ If "other": add a new Cause entry to this runbook before closing the incident
□ Was the alert caught by monitoring before customers reported it? Note Y/N
□ Create follow-up tickets for:
- Any mitigation that took longer than the runbook estimated
- Any runbook step that was wrong or missing
- Any tooling change that would have made this faster
□ Update this runbook if any step was inaccurate
The post-incident checklist is the mechanism that keeps runbooks alive. Without it, runbooks are created once and drift.
Linking Alerts to Runbooks
A runbook is only useful if the on-call engineer finds it immediately when an alert fires. The link should be embedded in the alert itself.
In alert annotations (Prometheus/Alertmanager)
- alert: ApiEndpointDown
expr: probe_success{job="api"} == 0
for: 2m
annotations:
summary: "API endpoint {{ $labels.instance }} is unreachable"
runbook_url: "https://runbooks.internal.myapp.com/api-endpoint-down"
description: |
The API endpoint has been unreachable for 2+ minutes.
Common causes: deployment issue, DB connection pool, OOMKill.
First step: check https://api.myapp.com/health manually.
In Vigilmon alert channels
Vigilmon supports customizable alert message content. Include the runbook URL in your notification message template so every Slack or email alert carries a direct link to the relevant runbook.
When an alert fires in Vigilmon, the notification can include:
- Which monitor failed and from which regions
- The runbook URL for that specific endpoint
- The current Vigilmon status page link for user-facing communication
The goal: from alert notification to runbook in one click, and from runbook to status page in one more.
In PagerDuty and incident management tools
Most incident management tools (PagerDuty, OpsGenie, Incident.io) let you attach runbook URLs to service configurations or alert rules. Configure this so that every page that creates a PagerDuty incident automatically surfaces the runbook link in the incident detail.
The Escalation Chain in Practice
Escalation chains fail in two directions: too slow (incidents stall at L1 because escalation feels like admitting failure) and too fast (senior engineers get woken up for issues that L1 could have resolved).
Make escalation feel safe. Your runbook should specify escalation thresholds explicitly — not as judgment calls, but as policy. "If the mitigation in Cause A doesn't resolve the issue within 15 minutes, escalate to the backend team lead" removes the social cost of escalating. The on-call engineer isn't admitting failure; they're following the process.
Set wake-up thresholds in writing. Team leads and managers should document their personal escalation expectations in the runbook or the on-call guide. "I want to be woken up for any P1 that isn't resolved within 30 minutes, regardless of time" is a much clearer signal than implied expectations.
Track escalation patterns. If escalations from L1 to L2 are rising week over week, either L1 is missing training/context (fix: add more runbook entries, run pairing sessions), or L2 is setting the escalation threshold too low (fix: update thresholds and redistribute ownership).
Status Pages as Escalation Communication
While your team works an incident, someone else is usually fielding a parallel stream of customer questions: "is your service down?" "when will it be fixed?" "can I get a credit?"
Status pages decouple incident response from customer communication. Instead of engineers stopping to answer support tickets while triaging, your status page gives customers a real-time channel that updates as the incident progresses.
Vigilmon's built-in status pages let you update incident status with a single action and embed a status badge on your product's support page or documentation site. During an incident, your runbook's post-communication step becomes: "Update the Vigilmon status page to 'investigating'" — one action that notifies all customers simultaneously.
Include status page updates as explicit steps in your runbook escalation chain:
When incident confirmed (before root cause identified):
→ Update status page to "Investigating: [service name] degraded"
→ Notify customer success team if any enterprise customers are affected
When mitigation is in progress:
→ Update status page to "Identified: [brief description], mitigation in progress"
When resolved:
→ Update status page to "Resolved"
→ Write a brief post-incident summary (5–10 sentences) for the status page
This makes customer communication part of the incident process rather than a distraction from it.
Runbook Maintenance: The Post-Incident Loop
A runbook that doesn't evolve becomes a liability. The only sustainable maintenance model is one that happens automatically as a byproduct of incident response.
The rule: If an incident reveals that a runbook entry was wrong, incomplete, or missing, updating the runbook is part of closing the incident. Not a follow-up ticket. Not a "should do someday." A blocker on incident resolution.
Enforce this through your post-incident checklist. Add a required field to your incident ticket: "Was the runbook used? Was it accurate? What was updated?"
Review runbooks on a schedule as well — quarterly is sufficient. Assign ownership to the team or individual who operates each service. Unowned runbooks go stale fastest.
Signals that a runbook needs updating:
- An incident was resolved using steps not in the runbook
- The incident required escalation that the runbook didn't anticipate
- The incident took longer than the runbook's estimated mitigation time
- A new infrastructure component was introduced (new database, new service, new deployment process)
Starting From Zero: Your First Runbooks
Don't try to create runbooks for everything at once. Start with the highest-impact alerts:
- Your most-fired alert — the one that wakes people up most often. Whatever shows up in your incident history with the most occurrences, start there.
- Your highest-severity alert — the P1 that represents your worst-case scenario (service completely down, payment processing failed, authentication broken).
- Your most-misunderstood alert — the one where post-mortems consistently show the on-call engineer wasn't sure what to do.
For each, talk to the engineer who most recently resolved that type of incident. Have them walk you through what they did. That conversation is your first draft. Write it down in the five-section format above.
Three solid runbooks are worth more than twenty placeholder documents that say "to be completed."
Summary
Effective on-call runbooks share five characteristics:
- Short and dense. One to two pages per alert, not a reference manual.
- Linked from the alert. One click from notification to runbook.
- Decision-tree structured. Each step produces an outcome that points to the next step.
- Specific, not generic. Real commands, real thresholds, real escalation targets.
- Maintained through post-incident loops. Updated as a condition of closing each incident.
The runbook is not documentation for its own sake. It's the mechanism that turns a 2 AM alert into a 15-minute incident instead of a 2-hour one. Combined with high-signal monitoring that pages on real events (not false positives), a good runbook is one of the highest-leverage investments in your team's operational reliability.
Set up monitoring that pages on real outages at vigilmon.online — multi-region consensus eliminates false positives, status pages handle customer communication, and webhook integrations connect to PagerDuty and your incident management toolchain.
Tags: #devops #oncall #runbook #sre #monitoring #incidents #observability