feat: Add opt-in per-domain request throttling for HTTP 429 backoff#1762

Merged

vdusek merged 29 commits intoapify:masterfrom

MrAliHasan:fix/request-throttler-429-backoff

May 6, 2026

Commits on Feb 20, 2026

fix: add per-domain RequestThrottler for 429 backoff (#1437)
MrAliHasan
committed

Commits on Feb 23, 2026

refactor: replace RequestThrottler with ThrottlingRequestManager
MrAliHasan
committed

Commits on Feb 25, 2026

refactor: reimplement `ThrottlingRequestManager` with per-domain sub-queues and update its integration across crawlers.
MrAliHasan
committed

Commits on Feb 26, 2026

test: fix typing and linting checks in ThrottlingRequestManager tests
MrAliHasan
committed

Commits on Mar 2, 2026

feat: Add explicit domain routing and management to the request throttling manager.
MrAliHasan
committed

Commits on Mar 4, 2026

feat: Implement recreate_purged for ThrottlingRequestManager and refactor domain state management and sub-queue handling.
MrAliHasan
committed

Commits on Mar 5, 2026

deps: Pin ty to version 0.0.18 and update uv.lock to include Python 3.14 wheels for brotlicffi.
MrAliHasan
committed

Commits on Mar 6, 2026

fix: Ensure ThrottlingRequestManager.add_request explicitly handles None returns and updates the ty dependency constraint.
MrAliHasan
committed

Commits on Mar 11, 2026

refactor: Address reviewer feedback on ThrottlingRequestManager and revert uv.lock
MrAliHasan
committed

Commits on Mar 13, 2026

Commits on Mar 26, 2026

refactor: Address review feedback — proper typing, recreate_purged guard, crawl-delay caching, revert uv.lock
MrAliHasan
committed

Commits on Apr 3, 2026

refactor: Add request_manager_opener callback, move record_success into mark_request_as_handled, fix docs
MrAliHasan
committed

Commits on May 5, 2026

Merge branch 'master' into fix/request-throttler-429-backoff
vdusek
committed

Commits on May 6, 2026