test(bench): add /map recall benchmark

[us]

What

A reproducible benchmark for /map URL-discovery recall — turns "map keeps missing URLs" into a number.

Ground truth = purpose-built scraper sandboxes with fixed, publicly-known structure (deterministic, unprotected, free):

Site	Known ground truth
books.toscrape.com	1000 products, 50 listing pages, 50 categories
quotes.toscrape.com	10 pages

Recall is the metric that matters: a raw count lies. trendyol.com returns ~5082 URLs but ~0 of the Turkish catalog (20/5082 products) — almost all /ar/ + /bg/ foreign-locale category sitemaps.

Run

CRW_API_URL=http://localhost:3000 uv run python bench/map_recall.py   # live
uv run python bench/map_recall.py --selfcheck                          # offline, recorded fixture

Baseline findings (hosted fastCRW, default maxDepth=2)

• books.toscrape.com: products 709/1000 (recall 0.71); main-catalogue pagination only 4/50 pages reached. A clean, unprotected, sitemap-less static site still loses ~29% of products.
• maxDepth 5 and 10 → HTTP 502 on hosted (default depth 2 works) — can't raise depth to compensate.

Root cause (follow-up, not fixed here)

crw-crawl/src/crawl.rs::discover_urls runs a BFS gated by depth < max_depth (default 2). Pagination is a chain (page-2 → page-3 → …), so depth-2 reaches ~page-3 and everything deeper is lost. The fix is pagination-aware traversal (follow rel=next / numbered pages on a separate budget) rather than just bumping default depth — raising depth lengthens the crawl and worsens the gateway 502. Tracking separately.

Scope

Bench tooling only — no product code touched. Offline --selfcheck pins the ~0.71 baseline from a recorded fixture so scorer regressions fail loudly; the live run needs a local crw server.