Operational playbooks for PAGE alerts (24/7 escalation, master spec §II.6).
Runbook goal: on-call knows what to do within 15 minutes (ack window) without knowing the architecture.
Contents
| File | Alert symptom | Severity |
|---|---|---|
| health_down.md | /health returns non-200 OR no response | PAGE |
| database_unreachable.md | Neon Postgres connection errors | PAGE |
| liveblocks_outage.md | Liveblocks API errors > 5% OR connection refused | PAGE |
| webhook_signature_spike.md | 401 on /webhooks/github > 10/min (potential attack) | PAGE |
| url_import_ssrf_spike.md | SSRF guard rejections > 10/min | INVESTIGATE |
| url_import_deployment.md | (operational guide, not an alert) | OPERATIONAL |
Quarterly review
Per master spec §II.9 — every quarter:
- Walk through each runbook
- Verify links + URLs are current
- Remove resolved gotchas
- Add new failure modes encountered
Template for a new runbook
# Runbook: <alert name>
## Symptoms
- What on-call sees (precise wording)
- Where (Grafana panel / Sentry / etc.)
## Severity & escalation
- PAGE 24/7 / NOTIFY / TICKET
- Ack window: 15 min
- Escalate if not resolved within: 30 min → engineering lead
## Immediate actions (< 5 min)
1. ...
2. ...
## Diagnosis (5-20 min)
- Check X — if Y, go to Z runbook
- Check W — ...
## Recovery
- Steps to restore
- Verification: what success looks like
## Aftermath
- Post-mortem trigger: if downtime > X min
- Backfill needed: ...
## Known false positives
- ...