Symptoms

• Sentry: spike LiveblocksError OR WebSocket connection failed from frontend
• Multi-user workflow co-edit not working (cursors not syncing, broadcasts dropped)
• POST /api/v1/liveblocks/auth returns 500 OR 502
• Liveblocks dashboard (https://liveblocks.io (opens in a new tab)) shows status issue OR fail rate

Severity & escalation

• PAGE 24/7 — real-time collab disabled. However, degraded mode is possible: REST + versions work (Phase 12 MD edit), just without presence/cursors
• Ack window: 15 min
• Escalate after 30 min → engineering lead
• Long outage (>2h): communicate degraded mode to users via UI banner

Immediate actions (< 5 min)

1. Check Liveblocks status: https://status.liveblocks.io/ (opens in a new tab) (or dashboard top bar)
2. Check our auth endpoint: curl -X POST arno-api.vadimpianof.workers.dev/api/v1/liveblocks/auth ... with valid JWT — error?
3. Check Liveblocks dashboard:
   • https://liveblocks.io (opens in a new tab) → project arno
   • Connection status / error rate / rate limit usage
4. Verify secret: wrangler secret list --config wrangler.toml | grep LIVEBLOCKS — LIVEBLOCKS_SECRET_KEY present

Diagnosis (5-20 min)

Branch A: Liveblocks-side outage

• Status page shows incident → wait
• Switch frontend to degraded mode banner:
  • "Real-time collab is temporarily unavailable. Changes are saved and visible after reload."
  • MD editor (Phase 12) continues to work via REST
• Workflow canvas: read-only mode (user cannot edit nodes/edges while Liveblocks is down — workflow primary storage lives there)

Branch B: Our auth endpoint broken

• Tail wrangler — find exception in liveblocks.ts::POST /api/v1/liveblocks/auth
• Common causes:
  • LIVEBLOCKS_SECRET_KEY env var missing or rotated incorrectly
  • Project ID hardcoded mismatch (see master spec §I.2.2 — project:${id} room naming)
  • User ownership check failed silently

Branch C: Rate limit hit

• Liveblocks Free: 100 connections/mo concurrent. If we crossed → 429 on auth
• Check dashboard → Usage
• Mitigation: upgrade to Liveblocks Pro $99/mo per cost ladder; OR apply startup credits (parking master spec §III.6)

Recovery

Verification

• POST /api/v1/liveblocks/auth with valid JWT returns 200 with session token
• Browser test: open project → cursor from 2nd window visible
• Sentry LiveblocksError rate < 0.1%
• Liveblocks dashboard "Healthy" status

Aftermath

• Post-mortem trigger: downtime > 30 min (collab harder to recover from than DB read-only)
• Document: degraded mode duration, which users were impacted
• If rate limit hit — review usage trends, plan upgrade

Known false positives

• Liveblocks Yjs Storage REST API throttling on bulk fetch — periodic 429 for individual room queries, not an actual outage. Do not PAGE. Add backoff in our code.
• WebSocket reconnect storms on a client-side network blip — appears as many errors but self-heals