• Date: 2026-06-03 (Phase 2 deviation noted 2026-06-04)
• Status: Accepted
• Phase / Feature: Capture-v3 → editor unlock
• Builds on: ADR 0030 (Page reconstruction), ADR 0032 (Reassembly iframe+bridge)
• Unlocks: in-artboard navigation per ADR 0032

Context

Extension today captures one page per toolbar click — runLayoutCapture in content-script.js:413 walks the current tab once, posts UPLOAD_PAGE_TREE to the background, background POST /api/v1/captured-pages. No crawl, no link discovery, no queue. Verified via grep + flow trace + live DB query on prj-7f8n428i (one row, https://mantine.dev/getting-started/).

ADR 0032 switched reassembly to custom in-artboard navigation: the bridge intercepts every <a href> click and postMessages the absolute href to the parent. Parent looks the URL up in the project's captured-page list. Hit → swap active page in the artboard (bridge stays alive, zoom keeps working, editor primitives stay attached). Miss → silent no-op.

For single-page captures every sidebar link is a miss. The captured tree has every Mantine sidebar entry as a rendered <a>, so visually the artboard looks navigable, but clicking does nothing because the linked pages aren't in the DB.

Crawl-mode closes this gap. One toolbar click captures the seed page and walks every linked same-origin page (up to a configurable depth/limit), uploading each as a separate captured_page row under the same project. After crawl, ADR 0032's setActivePageId lookup hits for every sidebar link by construction — navigation in the artboard works across the whole captured site with no frontend changes.

Decision

Add a "Capture site" mode to the extension that:

1. Captures the active tab (same as today).
2. Collects in-document <a href> URLs that match the same-origin + same-prefix filter.
3. Queues each unique URL, opens it in a background tab, runs runLayoutCapture on it, uploads, closes the tab.
4. Repeats per discovered link up to depth and maxPages caps, with a throttle between requests.
5. Reports progress to the popup (X / Y captured, current URL, cancel button).
6. Stops cleanly on user cancel, tab-close, or extension reload.

Existing per-page capture stays the default toolbar action. "Capture site" is a separate popup button so users opt in explicitly.

Architecture

                     ┌────────────────────────────────────────┐
                     │ popup.js                               │
                     │   buttons: Capture page | Capture site │
                     │   progress bar (during crawl)          │
                     └─────────────────┬──────────────────────┘
                                       │ CRAWL_START { depth, maxPages, throttle }
                                       ▼
┌──────────────────────────────────────────────────────────────────┐
│ background.js                                                    │
│                                                                  │
│   crawlQueue: { seed, queue, visited, depthMap, opts }           │
│                                                                  │
│   async function runCrawl():                                     │
│     1. Capture seed tab (existing path)                          │
│     2. Read returned tree → collectLinks() → filter → enqueue    │
│     3. Loop while queue.size && captured < maxPages:             │
│        - dequeue url                                             │
│        - tab = chrome.tabs.create({url, active:false})           │
│        - await tab onUpdated 'complete'                          │
│        - chrome.scripting.executeScript(runLayoutCapture)        │
│        - on UPLOAD_PAGE_TREE: enqueue discovered links           │
│        - chrome.tabs.remove(tab)                                 │
│        - sleep(throttle)                                         │
│        - sendMessage(CRAWL_PROGRESS) → popup                     │
│     4. CRAWL_DONE / CRAWL_CANCELLED                              │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘
                                       │
                                       ▼ N × POST /api/v1/captured-pages
                                  (backend already supports this — no changes)

Backend contract is satisfied today: POST /api/v1/captured-pages is idempotent on id (UUIDv7), and GET /api/v1/projects/:projectId/captured-pages returns the full list which ReconstructedPage already reads into allPages.

Implementation plan (in order)

Phase 1 — link collection during capture

Surface: packages/url-import-extension/lib/find-layouts.js or new lib/collect-links.js.

• After buildLayoutTree returns, walk document.querySelectorAll('a[href]') once.
• For each anchor, resolve a.href (absolute, base-resolved). Skip:
  • mailto:, tel:, javascript:
  • #fragment-only on current page
  • same URL as location.href (self-link)
• Return links: string[] (deduped, normalized — lowercase origin + pathname, hash dropped, query kept).
• Surface this in the UPLOAD_PAGE_TREE payload as discovered_links: string[] so background can read it without re-querying the tab.

Reference normalization (steal from frontend reconstructed-page.tsx):

function normalizeUrl(u) {
  try {
    const url = new URL(u);
    url.hash = "";
    const path = url.pathname.replace(/\/+$/, "") || "/";
    return (url.origin + path + url.search).toLowerCase();
  } catch { return u.toLowerCase(); }
}

Phase 2 — popup UI (SUPERSEDED 2026-06-04)

Deviation from this ADR (decided during implementation): the toolbar icon stays the single entry point per the popup-less flow lock-in (Rules.md §"Things to NEVER do" → "Регрессировать popup-less capture flow"). Click toolbar runs runCrawl with hard-coded defaults — depth=1, maxPages=25, throttle=1000ms. No popup buttons, no config row, no progress panel beyond the existing spinner icon. Cancel = repeat click → sets crawlState.cancelled for graceful wind-down (the in-flight per-tab capture finishes, then driveCrawl exits the loop and clears the capturing flag itself; pending tabs receive CAPTURE_CANCEL so late UPLOAD_PAGE_TREE becomes a no-op).

Rationale:

• Popup-less flow is a Rules-level invariant — restoring default_popup silently disables chrome.action.onClicked, which is the canonical invoke gesture today.
• Defaults are good enough for the "I want my site captured" outcome. Sites with no in-document links collapse to the legacy single-page outcome (1 capture, ends immediately).
• If per-crawl tuning is needed later, it ships as a settings page or a right-click context menu — not a re-introduced popup.

The original popup spec is kept below for historical context. None of the popup files (popup.html, popup.css, popup.js) change.

Surface: popup.html, popup.css, popup.js.

Add to popup (only when a project is bound and entry_mode is capture):

• Second action button: Capture site (next to existing Capture page).
• Inline config row (collapsed by default):
  • Depth: radio 1 / 2 / 3
  • Max pages: 25 / 50 / 100
  • Throttle: 500ms / 1s / 2s
• Progress section (visible during crawl):
  • Captured 12 / 47
  • Current URL (truncated, monospace)
  • Indeterminate bar
  • Cancel button
• Result section (after crawl):
  • 47 pages captured
  • Open in ARNO button (uses existing openOrFocusWorkflow)

State is driven by chrome.runtime.onMessage listeners on CRAWL_PROGRESS / CRAWL_DONE / CRAWL_ERROR — popup is dumb, background owns truth.

Phase 3 — background queue + sequential crawl

Surface: background.js — add ~150 LOC.

let crawlState = null;
 
async function runCrawl({ tabId, depth, maxPages, throttle }) {
  crawlState = {
    seed: tab.url,
    seedOrigin: new URL(tab.url).origin,
    seedPathPrefix: pathPrefix(tab.url),
    queue: [],          // [{ url, depth }]
    visited: new Set(), // normalizedUrl
    captured: 0,
    maxPages,
    throttle,
    maxDepth: depth,
    cancelled: false,
  };
 
  // 1. Seed: existing per-page capture path on the foreground tab
  const seedResult = await captureTab(tabId);
  crawlState.captured++;
  enqueueLinks(seedResult.discovered_links, /* fromDepth */ 0);
  sendProgress();
 
  // 2. Drain the queue
  while (
    crawlState.queue.length > 0 &&
    crawlState.captured < maxPages &&
    !crawlState.cancelled
  ) {
    const { url, depth: d } = crawlState.queue.shift();
    if (crawlState.visited.has(normalizeUrl(url))) continue;
    crawlState.visited.add(normalizeUrl(url));
 
    let bgTab;
    try {
      bgTab = await chrome.tabs.create({ url, active: false });
      await waitForTabComplete(bgTab.id);
      await sleep(crawlState.throttle);
      const result = await captureTab(bgTab.id);
      crawlState.captured++;
      sendProgress(url);
      if (d < crawlState.maxDepth) {
        enqueueLinks(result.discovered_links, d + 1);
      }
    } catch (e) {
      console.warn("[ARNO crawl] failed for", url, e);
      sendProgress(url, "error");
    } finally {
      if (bgTab?.id) await chrome.tabs.remove(bgTab.id).catch(() => {});
    }
  }
 
  sendDone();
  crawlState = null;
}
 
function enqueueLinks(links, fromDepth) {
  for (const url of links || []) {
    const n = normalizeUrl(url);
    if (crawlState.visited.has(n)) continue;
    // same-origin + same-path-prefix only (configurable later)
    const u = new URL(url);
    if (u.origin !== crawlState.seedOrigin) continue;
    if (!u.pathname.startsWith(crawlState.seedPathPrefix)) continue;
    crawlState.queue.push({ url, depth: fromDepth });
  }
}
 
function waitForTabComplete(tabId) {
  return new Promise((resolve, reject) => {
    const timeout = setTimeout(
      () => { chrome.tabs.onUpdated.removeListener(listener); reject(new Error("load timeout")); },
      30000
    );
    function listener(updatedId, info) {
      if (updatedId === tabId && info.status === "complete") {
        clearTimeout(timeout);
        chrome.tabs.onUpdated.removeListener(listener);
        resolve();
      }
    }
    chrome.tabs.onUpdated.addListener(listener);
  });
}
 
// captureTab: thin wrapper that injects content-script + lib (same as today)
// and resolves once UPLOAD_PAGE_TREE arrives for this tabId. Returns the
// payload so the crawl can read discovered_links.

New message handlers in the existing switch (around line 864):

• CRAWL_START { depth, maxPages, throttle } → runCrawl(...), respond { ok: true }.
• CRAWL_CANCEL → crawlState.cancelled = true, respond { ok: true }.

New outbound messages to popup:

• CRAWL_PROGRESS { captured, queued, current, status: 'ok'|'error' }
• CRAWL_DONE { captured, errors }
• CRAWL_ERROR { message }

Phase 4 — manifest permission audit

Surface: manifest.json (opens in a new tab).

Current permissions cover everything needed:

• tabs (have it) — to read tab.url after create, listen to onUpdated
• scripting (have it) — to inject content-script in background tabs
• webNavigation (have it, unused by crawl, fine to keep)
• activeTab (have it) — sufficient for the seed; background tabs use tabs + explicit scripting.executeScript

No manifest changes required. Host permissions don't restrict captured site origins because we don't fetch() them — we just open tabs to them.

Phase 5 — verification

Manual test plan (no automated harness — capture flow is integration-heavy):

1. Seed-only project — capture site on a single-page (no internal links). Expect 1 captured, crawl ends immediately. Workflow opens, click does silent no-op (already covered).
2. Small docs site — capture site on mantine.dev/getting-started/ depth 1. Expect ~15 captures (sidebar entries). Open workflow → click sidebar items → setActivePageId hits, artboard swaps.
3. Medium docs site — depth 2 on Mantine. Expect ~50 captures (sidebar + nested component pages). Hit cap at 50, crawl ends, partial result usable.
4. Cancel mid-crawl — start, hit Cancel at ~15/50. Expect popup shows "Cancelled · 15 captured", workflow has 15 rows.
5. Background tab error — kill network mid-crawl. Expect failed tab logged + skipped, crawl continues with next URL.
6. Auth-gated — capture site on a Notion private doc. Expect cookies inherit, captures succeed.
7. Cross-origin link — site links to discord.com etc. Expect those skipped (same-origin filter).

Phase 6 — wire into HANDOFF

After Phase 5 verifies, update HANDOFF.md "Сделано" with crawl-mode shipped and remove ADR 0032's "miss → silent no-op" caveat (it becomes unreachable for crawled projects).

Edge cases and defaults

Backend dedupe check

Action item for Phase 1: apps/api/src/captured-pages.ts:42 (POST handler) — confirm what happens when two POSTs arrive with same (project_id, source_url) but different id. If both insert (two rows), the frontend allPages will have duplicates and setActivePageId(match.id) will pick whichever the SQL ordering returns. Either:

• Backend writes ON CONFLICT (project_id, source_url) DO UPDATE (preferred — re-crawl refreshes).
• Or extension checks the existing allPages list before queueing (race-y, requires extra fetch).

Verify by re-capturing the same page twice and watching SELECT count(*) FROM captured_page WHERE project_id = ... AND source_url = ....

Estimated work

• Phase 1 (link collection): 1-2 hours. Self-contained, walks DOM, returns strings.
• Phase 2 (popup UI): 2-3 hours. Buttons + progress section + onMessage wiring. CSS is simple.
• Phase 3 (background queue): 4-6 hours. Most of the work. Tab orchestration has the race conditions — sleep + retry patterns.
• Phase 4 (manifest): 0 (no changes).
• Phase 5 (verification): 2-3 hours of manual capture on real sites.
• Phase 6 (handoff doc): 30 min.

Total: ~2-3 focused days to ship MVP. Resume support, parallel crawl, whitelist UI — v2 sprint, not now.

Non-goals (explicit out of scope for v1)

• Resume after crash — crawl state is in-memory; closing the browser loses progress. Acceptable for ~50-page crawls (<5min).
• Parallel tabs — sequential only. 2-3 tabs at once would halve crawl time but multiplies race conditions in service worker.
• Cross-origin whitelist UI — strict same-origin + same-prefix for v1. Captures one site, not the web.
• Sitemap.xml ingestion — not used. Discovery is link-based only. v2 could add sitemap fallback for sites that hide links behind JS.
• Diff-based re-crawl — re-running crawl re-captures everything. Smart "only changed pages" is v2.
• JavaScript-rendered links — content-script runs after document_idle and scrollToBottom, so most async-rendered links are caught. Truly dynamic SPA routing (e.g. routes that only exist after user interaction) is out.

Open questions (for the next-session implementer to decide)

1. Where does link collection live? In lib/find-layouts.js (alongside tree walk) or new lib/collect-links.js? Prefer alongside if it can reuse the DOM walk; standalone if walking again is cheap enough (it is — querySelectorAll('a[href]') is fast).
2. Throttle granularity — fixed 1s default or adaptive (e.g. observe HTTP 429 and back off)? v1 = fixed; adaptive is overkill.
3. Capture v3 fiber tree per crawled page — yes, every crawled page goes through the same runLayoutCapture so CAPTURE_FIBER + image inlining + font-face harvest all run for free. Confirms the ~500KB-per-page DB estimate.
4. What happens to overlay UI in the seed tab during crawl? Seed capture finishes → overlay disappears. Background tabs don't show overlay (Overlay.create runs but in invisible tab). User sees only popup progress. Confirm popup actually stays open across the crawl duration (popup auto-closes on focus loss — may need a workaround like a tiny persistent window).
5. Domain matching is case-sensitive in URL.origin (Mantine.dev ≠ mantine.dev). Normalize via .toLowerCase() everywhere — normalizeUrl already does this.

References

• ADR 0030 §B1/B2 — captured_page schema + upload contract
• ADR 0032 — why custom nav (and why crawl is its enabler)
• packages/url-import-extension/content-script.js:413 — runLayoutCapture (the per-page entry point to reuse)
• packages/url-import-extension/background.js:864 — message switch (where CRAWL_START slots in)
• apps/api/src/captured-pages.ts:42 (opens in a new tab) — backend POST handler (verify dedupe)
• apps/web/src/components/reconstructed-page.tsx (opens in a new tab) — frontend nav resolver (consumes allPages)