[401 escalation] Run the week-4 measurement and resolve both toggles — query by 2026-08-29

[FagnerMartinsBrack]

Follow-up for the change that added 401 to the crawler's fallback ladder
(FALLBACK_STATUS_CODES in h2-fetch.ts, BLOCK_STATUS_CODES in persona-fallback.ts).
Seeded by the failed-articles RCA in #1020 (1 of 29 links).

Evidence URLs are deliberately reduced to hosts throughout this issue — saved URLs are user
reading history, and this repository is public.

Deadline: run the queries by 2026-08-29 (week 4, assuming the change ships 2026-08-01).
crawl-outcomes lines live in each save-link Lambda's own log group at 30-day retention, so the
oldest post-change lines expire 2026-08-31. If the query slips past day 30 the window restarts.

Why this issue exists

401 was in neither escalation set, so a 401 from the first undici request ended the crawl after a
single request — never trying the h2 TLS fingerprint, curl-impersonate, or the honest-bot persona,
the same ladder that routinely rescues 403s from the same CDNs. Because dlqMaxReceiveCount: 1
removes SQS retries by design, that one request was the user's entire save attempt.

The prod evidence (a reuters.com article fronted by CloudFront, 2026-07-26) is consistent
with both readings and cannot distinguish them:

• a bot-deny spelled as 401 — the same host served an image asset a 200 in the same second, and the
  crawl returned in 69.71ms versus 343–1352ms for the 403 siblings that traversed the full ladder;
• a genuine auth gate — a subscriber-only article.

The escalation shipped so this measurement window can resolve it.

NON_RECOVERABLE_FETCH_STATUSES in log-fetch-failure.ts was deliberately left unchanged — a
terminal 401 still logs at ERROR for the duration of this window. Demoting it now would presume the
conclusion the window exists to test, and would hide the very signal that says whether escalation
works.

Queries

Run over the save-link Lambdas' log groups (the Lambdas wired through
projects/save-link/src/runtime/dep-bundles/observability.ts) in CloudWatch Logs Insights.

# terminal 401s post-change
filter @message like "[CrawlArticle] HTTP 401 for"
| parse @message "HTTP 401 for *" as target
| stats count() by target

# crawl outcomes for the same hosts
filter stream = "crawl-outcomes" and event = "tier-outcome"
| stats count() by url, thisTierStatus

target captures the URL plus the optional (server=…) suffix describeEdgeHeaders appends —
group by host when joining against the outcome query, and keep the analysis at host level. Because
dlqMaxReceiveCount: 1 means one crawl per save, outcome lines map 1:1 to user attempts.

Decision rule — resolves both toggles together

Confirmed (bot-deny): any host that produced a terminal HTTP 401 line before the change
(reuters.com is the tracked seed) shows thisTierStatus="success" outcomes after it, or its
terminal-401 rate drops to zero while saves for it continue.

→ Keep the escalation, and add 401 to NON_RECOVERABLE_FETCH_STATUSES in log-fetch-failure.ts.
The residue that still 401s after a rescue-capable ladder is the genuine-auth-gate population,
mirroring exactly why post-ladder 403 logs at info (paywalls, subscriber pages). Update that doc
comment and flip the corresponding crawl-article.test.ts assertion from logError to logInfo.

Refuted (genuine auth gates): after 4 weeks no 401-history host ever succeeds and every 401 is
still terminal post-ladder.

→ Revert the two set-membership changes (one line each), and add 401 to
NON_RECOVERABLE_FETCH_STATUSES — terminal and unactionable belongs at info.

The end-state log level is info on both branches. What the measurement actually decides is
whether the ladder runs. That is why level and escalation could not be flipped independently.

Early exit

RCA volume was 1 of 29 links. If terminal-401 volume spikes on the error dashboard before week 4,
take the decision early on whatever data exists.

Known limits of the shipped change

• No rung attribution. A rescue does not record whether h2, curl, or the persona switch won —
  intermediate 401s are drained silently. The keep/revert decision only needs
  rescue-vs-never-rescued, which the outcome stream answers.
• Local reproduction was inconclusive by construction. Against the seed host, undici and h2 both
  returned 401 (server=CloudFront), but curl-impersonate is not installed outside the Lambda image,
  so the curl rung threw ENOENT — a non-block-class error, which makes withPersonaFallback
  rethrow before reaching the second persona. Only 2 of the 6 rungs actually ran locally, and local
  egress differs from Lambda egress regardless.
• Latency cost on genuine auth gates. Up to 6 requests instead of 1, bounded by the caller's 30s
  headers budget and the h2 leg's own 10s deadline; withRateLimitRetry retries only 429, so 401
  adds no sleep-retry cycles.

Only if Confirmed

Consider adding the seed host to src/packages/crawl-article/scripts/health-sources.ts. Not before —
a canary entry resting on an unconfirmed hypothesis blocks CI and invites the delete-the-entry
pressure CLAUDE.md forbids.