Symptoms

• Sentry: spike error_type: NeonConnectionError OR PostgresError
• API endpoints return 500 with code: "internal" and message mentioning DB
• /ready endpoint reports degraded: true with hard dep "postgres"
• Grafana: pg_up == 0 OR Postgres connection latency > 5s

Severity & escalation

• PAGE 24/7 — writes fully disabled. Read fallback (snapshots in KV) lives 60s after last refresh, then goes stale
• Ack window: 15 min
• Escalate after 30 min → engineering lead
• On Neon-side outage without ETA → consider PITR restore to secondary region (Phase 16+ post-MVP)

Immediate actions (< 5 min)

1. Reproduce: wrangler tail should show a DB errors stream

   cd apps/api && npx wrangler tail --config wrangler.toml --format=pretty
2. Check Neon status: https://neonstatus.com (opens in a new tab)
3. Open Neon dashboard: https://console.neon.tech (opens in a new tab) → project arno-prod (Frankfurt)
   • If it shows Suspended (free tier auto-suspend) → click to open, wake up. Resume ~10s
   • If it shows Error / Down → wait OR contact Neon support
4. Verify connection from CLI:

   psql 'postgresql://neondb_owner:***@ep-dry-block-al36bkvg.c-3.eu-central-1.aws.neon.tech/neondb?sslmode=require' -c 'SELECT 1'

Diagnosis (5-20 min)

Branch A: Neon suspended (free tier)

• This is expected behaviour on idle period > 5 min for free
• Mitigation: upgrade to Neon Launch ($19/mo) when DAU > 50 OR business critical (per cost ladder)
• Open project in dashboard → resume → verify /ready is green

Branch B: Neon up but Workers can't connect

• Check DATABASE_URL wrangler secret format — must be postgresql://user:pass@host/db?sslmode=require
• Check Neon allowlist (IP) — Workers exit IPs are unstable, so Neon must accept any IP (default for serverless tier)
• Check rate limit — Neon free 100 concurrent connections, we use the Drizzle HTTP driver (stateless, no pool)
• Run direct query via tools/migrate/ (Pool driver) → if it works → HTTP driver issue (unlikely)

Branch C: Neon-side outage

• Status page shows incident → wait
• Inform users via status page if > 5 min downtime
• Consider PITR restore: Neon → branch → PITR to point before outage

Recovery

Verification

• /ready endpoint returns degraded: false
• API endpoints with write (POST /api/v1/projects) succeed
• Sentry error rate returns to baseline ≤5 min after recovery
• Grafana pg_up == 1 persistent for 5 min

Aftermath

• Post-mortem trigger: downtime ≥ 10 min
• Document: timeline, root cause, was PITR needed?
• If free tier suspend happens often (>1/week) → upgrade to Launch

Known false positives

• PITR backups window (nightly ~03:00 UTC) — short read latency spike, not unreachable. Do not PAGE
• Test queries from ARNO CI — may briefly increase connection count; do not PAGE