Runbooks
Environments runbook

Per ADR 0022. Multi-env topology for ARNO.

URL reference

LayerProdTest (feat/**)Local dev
Frontendhttps://arnomake.com (opens in a new tab)test-<topic>.arnomake.com (per-branch, ADR 0022 v2.0)http://localhost:3000 (opens in a new tab)
Backendhttps://api.arnomake.com/health (opens in a new tab)arno-api-test.<cf-subdomain>.workers.dev/healthhttp://localhost:8787 (opens in a new tab)
DBNeon main branchNeon feat-layout-grid branchNeon dev-local branch (optional)
Liveblocks projectarnoarno-testUse arno-test
GitHub OAuth Apparnoarno-testUse arno-test

Setup (one-time per dev)

If you are setting up dev on ARNO for the first time:

  1. Clone repo: git clone https://github.com/vadimpianov/arno.git
  2. Install pnpm + Node 20: corepack enable && nvm use 20
  3. pnpm install in root
  4. Cloudflare CLI: pnpm dlx wrangler login (browser OAuth)
  5. Local dev secrets:
    • cp apps/api/.dev.vars.test.example apps/api/.dev.vars
    • Fill in values (ask repo owner for access to test resources)
  6. Run local stack:
    • Terminal 1: pnpm --filter @arno/api exec wrangler dev (backend on :8787)
    • Terminal 2: pnpm --filter @arno/web dev (frontend on :3000)
  7. Verify: open http://localhost:3000 (opens in a new tab), login via GitHub OAuth

Deploy

Test backend (after a change on a feat/** branch)

cd apps/api
wrangler deploy -c wrangler.test.toml

Output: deployment URL + version ID. Note version ID for rollback.

Test frontend (auto on push)

git push origin feat/layout-grid
# CF Pages auto-builds + deploys preview URL
# Check dash.cloudflare.com/pages → arno → Deployments → preview

Production (only via merge PR)

# Locally
git checkout main
git merge feat/layout-grid  # OR squash-merge via GitHub PR
git push origin main
# GitHub Action `.github/workflows/deploy-main.yml` runs:
#  - typecheck + test + E2E gating
#  - if pass: wrangler deploy (prod backend) + CF Pages prod deploy
#  - if fail: rollback automatic (no merge happens)

Rollback

Backend (CF Workers)

# List recent versions
wrangler versions list -c apps/api/wrangler.toml  # prod
wrangler versions list -c apps/api/wrangler.test.toml  # test
 
# Rollback to specific version
wrangler rollback <version-id> -c apps/api/wrangler.toml

MTTR: ~1 minute.

Frontend (CF Pages)

Dashboard → Pages → arno → Deployments → previous successful deploy → "Rollback to this deployment"

Or via API:

curl -X POST "https://api.cloudflare.com/client/v4/accounts/<acct>/pages/projects/arno/deployments/<deploy-id>/rollback" \
  -H "Authorization: Bearer $CF_API_TOKEN"

MTTR: ~30 seconds.

Database (Neon)

Point-in-time recovery (Pro tier, 7 days retention):

neon branches create --name restore-$(date +%s) --parent main --timestamp "2026-05-25T12:00:00Z"

After verifying restore data on new branch, swap connection strings.

MTTR: ~5 minutes.

GitHub Actions secret scope

The repository has no GitHub Environments protecting its secrets beyond the production environment added for release.yml, so every value under Settings → Secrets → Actions is readable by any workflow on any branch — including deploy-test.yml, which runs on every push to feat/** and fix/** with no review. A one-line change to a workflow on a feature branch can print PROD_DATABASE_URL.

release.yml now runs in the production environment (restricted to main), so the scoping is one step away:

  1. Settings → Environments → productionAdd environment secret for PROD_DATABASE_URL and CLOUDFLARE_API_TOKEN (same values as today).
  2. Settings → Secrets → Actions → delete the repository-level copies.
  3. Run release (prod) once and confirm the migration + deploy steps still resolve them.

Order matters: while both exist the environment copy wins for jobs that declare the environment and the repository copy still serves everything else, so step 1 alone changes nothing and cannot break a release.

CLOUDFLARE_API_TOKEN is also account-wide — it deploys workers and edits DNS for the whole zone, and deploy-proxy.yml hands it to the preview proxy worker, which only needs Pages:Read. Minting a scoped token for the proxy is a separate small change (see infra/preview-proxy/wrangler.toml).

Secrets rotation

Every 90 days (master spec §Quarterly):

  1. Generate new value (e.g. openssl rand -hex 32 for JWT_SECRET)
  2. wrangler secret put <KEY> for prod: -c apps/api/wrangler.toml
  3. wrangler secret put <KEY> for test: -c apps/api/wrangler.test.toml
  4. Update .dev.vars locally
  5. Verify deployments healthy after rotation

Incident response

Frontend broken (build fails on main, prod down)

  1. Sentry alert fires + status page auto-detects
  2. Identify last good main SHA: git log --oneline -10 main
  3. git revert <bad-sha> && git push origin main
  4. CF Pages auto-redeploys < 2 minutes
  5. Verify https://arnomake.com (opens in a new tab) loads

Backend broken (deploy succeeded but API 500s)

  1. Sentry alert + health check failing
  2. wrangler versions list -c apps/api/wrangler.toml → previous version
  3. wrangler rollback <version> — instant
  4. Investigate cause before next deploy

Database corrupted / data loss

  1. Identify scope: which tables, what data, what time window
  2. If scope is local: neon branches create --name recovery --parent main --timestamp <before-corruption>
  3. Validate recovery branch data
  4. Swap DATABASE_URL on prod backend: wrangler secret put DATABASE_URL → new branch URL
  5. After validation, promote recovery branch to main: Neon dashboard → branches → recovery → Set as default

All Cloudflare down (region-wide outage)

  1. Twitter @cloudflarestatus + status.cloudflare.com confirms
  2. Wait it out — CF SLA 99.99% means this is rare
  3. Static fallback page: pre-configured Vercel/GitHub Pages mirror on a separate DNS
  4. Communicate via status page (if status page hosted on different infra)

Quarterly checklist

  • Restore drill: PITR restore on test branch, verify functionality
  • Rollback drill: wrangler rollback on test, measure MTTR
  • Secret rotation: all wrangler secret put updates
  • Dependencies audit: pnpm audit, update vulnerable packages
  • Config drift check: diff wrangler.toml vs wrangler.test.toml, justify divergence
  • Free tier usage: review CF / Neon / Liveblocks consumption, plan upgrades if > 80%
  • Alert false positives: review Sentry alerts, tune thresholds

Contact

  • Repo owner: vadimpianof@gmail.com
  • On-call rotation: solo (escalate to repo owner)
  • Status page: TBD (create as part of Phase 2 setup)