Symptoms

• Grafana alert: up{job="arno-api"} == 0 OR http_request_duration_ms{path="/health"} timeout
• Healthchecks.io ping missed (configured to https://arno-api.vadimpianof.workers.dev/health)
• Direct check: curl https://arno-api.vadimpianof.workers.dev/health returns non-200 OR hangs

Severity & escalation

• PAGE 24/7 — core API down = no writes, no Liveblocks auth, no GitHub sync
• Ack window: 15 min
• Escalate if not resolved within 30 min → engineering lead
• If investigation points to Cloudflare-side outage → check status.cloudflare.com first; in that case ETA depends on CF

Immediate actions (< 5 min)

1. Reproduce: curl -i https://arno-api.vadimpianof.workers.dev/health
2. Check Cloudflare status: https://www.cloudflarestatus.com/ (opens in a new tab) — if Workers / Dashboard incident → wait for resolution, log incident in incident channel
3. Check recent deploy: gh run list --repo vadimpianov/arno --limit 3 — if last CI push ≤30 min ago — may be our bug
4. Tail Worker logs: cd apps/api && npx wrangler tail --config wrangler.toml --format=pretty
   • If you see runtime errors (uncaught throw, OOM) — go to Recovery step "rollback"
   • If no logs (worker not running) — CF-side issue
5. Sentry check: arno-backend project → most recent errors. Sort by first_seen desc

Diagnosis (5-20 min)

Branch A: CF status page shows incident

• ETA listed → wait
• Communicate ETA in incident channel + status page (if available)
• Skip remaining diagnostic — this is not our problem

Branch B: Recent deploy (≤30 min)

• Rollback: gh workflow run deploy.yml -f rollback=true --ref main OR direct wrangler rollback
• Verify: curl /health returns 200 ≤2 min after rollback
• Open incident retro task

Branch C: No recent deploy, CF up

• Probably config drift OR external dep cascade
• Check wrangler secret list --config wrangler.toml — secrets present (DATABASE_URL, JWT_SECRET etc)
• Check Neon dashboard → if Postgres suspended/down → go to database_unreachable.md
• Check Liveblocks dashboard → if down → go to liveblocks_outage.md. Note: /health does NOT depend on Liveblocks, but if they are down and our health hits them — fix /health to not depend on them

Branch D: Worker exists but returns 500

• Tail wrangler — find stack trace
• Common causes:
  • JWT_SECRET malformed (rotation gone wrong)
  • DATABASE_URL connection string wrong
  • Required ENV not set

Recovery

Verification

• curl https://arno-api.vadimpianof.workers.dev/health returns {"ok":true,"ts":<recent>}
• Healthchecks.io shows green ping ≤2 min later
• Sentry error rate returns to baseline ≤5 min

Aftermath

• Post-mortem trigger: downtime ≥ 5 min
• Document: timeline, root cause, action items
• Update runbook if new failure mode

Known false positives

• Network blip from Healthchecks.io probes — a single missed ping is not PAGE-worthy. Healthchecks config: alert after 2 consecutive misses
• Cold start latency on free Workers tier — we are on Paid, not an actual case