[yeahjack]

Stack and review scope

This is the final stacked integration PR for #3946. Its develop comparison intentionally includes the integrated dependency chain below; none of these PRs is merged by this change.

Review the batching work separately in the batch-only commit diff. Its exact parent is e20801b6ae93a7b47d9d51b93b5f899699353363. The subsequent Windows test-stack configuration retains every parser test and assertion.

Dependency	Integrated commits included here
#4536 bounded recall statistics	f9bdd36, 9514ad5
#4537 payload byte budget and maintenance	321b646
#4538 sensitive-output notices	e78d67b
#4539 recall/tee forget cleanup	e2831be
#4541 invocation and actual-output accounting	58d10e2
#4542 immutable capture references and safe migration	c03f33e, e20801b

Summary

Fixes #3946.

• Batch grouped grep/rg recovery writes into one immediate SQLite transaction, with prepared payload/capture inserts and one final payload-retention pass. Hashing and gzip run before taking the write lock.
• Preserve independent immutable capture references, original per-file offsets, notices, source-invocation attribution, the skipped-files recovery block, bounded statistics, and payload/capture limits. Prune stale captures before insertion and after eviction, and advertise only captures that survive the transaction.
• Skip individually ineligible entries while saving eligible peers, including when an existing immutable payload's stored encoding is larger than a newly prepared encoding. Roll back payloads, captures, stats, and byte accounting together on insertion or retention failure.

The original cap regression is reproduced on develop: four files with max_entries = 2 produce four hints, but the first two recall commands exit 1. The batch now advertises only surviving captures. Duplicate payloads keep distinct immutable offsets rather than borrowing the most recent capture's offset.

Performance

The final comparison uses the exact matched predecessor e20801b6ae93a7b47d9d51b93b5f899699353363, so both binaries contain all companion features. Rust 1.99.0 release profile, shared Linux x86_64 host; 100 files × 80 matching lines; rtk grep --max 2500 -rn needle corpus; 30 interleaved measured runs after three warm-ups:

Recovery	Before median	After median	Before mean ± SD	After mean ± SD
Disabled	20.48 ms	19.91 ms	21.61 ± 3.34 ms	20.47 ± 1.94 ms
Tee	22.83 ms	22.06 ms	23.76 ± 2.78 ms	23.22 ± 2.85 ms
SQLite raw	107.92 ms	25.84 ms	109.86 ± 13.00 ms	27.16 ± 3.66 ms
SQLite gzip	116.28 ms	31.57 ms	114.69 ± 15.21 ms	32.90 ± 4.63 ms

Both SQLite cases preserve 100 per-file capture hints. Gzip median is ~73% lower against this integrated predecessor, largely from removing repeated per-file capture-pruning work. This is not a 73% improvement against unmodified develop. The initial standalone comparison against current develop 311ebcb showed ~15% lower median; both experiments are documented separately.

At the default 200-result display cap (nine captures), gzip median is 34.31 → 32.40 ms; raw 24.91 → 22.52 ms. The one-file check shows no observed regression (gzip 5.29 → 4.80 ms; disabled control 4.74 → 4.25 ms). Small-case differences are within host variability.

A separate phase benchmark uses a fresh store for each codec so immutable deduplication cannot reuse the other encoding:

• Raw: preparation 811 µs; transaction 2,681 µs median
• Gzip: preparation 5,532 µs; transaction 2,553 µs median

The preparation difference isolates roughly 4.7 ms of incremental gzip work from database writes. These are workload-specific observations on a noisy shared host, not timing guarantees or a reuse of the old issue's ddc21826 numbers.

Reproduction: scripts/benchmark-recall-search.py and docs/contributing/RECALL_BENCHMARK.md.

Test plan

• cargo fmt --all -- --check, cargo clippy --all-targets, cargo test --all: 4,459 passed, 9 ignored
• cargo doc --no-deps and git diff --check
• Opt-in release phase benchmark: cargo test --release --bin rtk test_benchmark_recall_batch_phases -- --ignored --nocapture
• Baseline cap regression fails before the fix; final end-to-end grep recovery passes
• Concurrent writers; insert/capture/eviction rollback; count/byte/stat/capture caps; empty and oversized entries; immutable duplicate offsets; arbitrary raw/gzip bytes; Unicode hint positions
• Mixed oversized/valid entries; retained-encoding budget changes; all-ineligible no-mutation behavior; capture-cap survivor hints; maintenance without old-reference resurrection
• Sensitive notice output accounting, attributed filtered recall, recall/capture forget cleanup, and disabled/tee behavior
• Matched-predecessor many-file, default-cap, and single-file release comparisons
• Upstream CI on exact head dddd94247ff595f6b0919668c4d0452b9be80718: Linux/macOS/Windows tests, fmt, clippy, rustdoc, benchmark, test presence, and security checks pass; CLA passes; master-target guard is correctly skipped for develop

Windows test configuration

The combined debug Clap graph exceeded the Windows libtest worker stack in hook_event_option_does_not_consume_nested_command_operands and the existing test_bun_build_parses_to_arg_vector (process exit STATUS_STACK_OVERFLOW, not an assertion failure or a timeout). Windows CLI startup already reserves an 8 MiB main-thread stack; test workers independently default to 2 MiB. The build now supplies the same 8 MiB minimum for Cargo-launched Windows test workers, including local cargo test. Cargo also supplies the setting to cargo run and inherited children, preserving a valid larger caller stack setting; a directly launched installed CLI is unaffected. Direct-harness instructions are documented. No tests, parser assertions, or production parsing semantics are removed or relaxed.

The hook event option from #4541 is root-only, not a Clap global option replicated across subcommands. This stack config is for the full combined command graph; it does not attribute the threshold to one companion PR. Cargo's runtime environment propagation is documented; Rust documents the separate worker-stack setting.

Known separate limitation

Custom tracking-history erasure through RTK_DB_PATH or [tracking].database_path is outside this stack and remains covered by the existing #3044. The forget integration here covers recall/capture storage and legacy tee files; it does not claim that a custom tracking database was erased.

[rtk-wshm-sync-bot]

wshm · Automated triage by AI

📊 Automated PR Analysis

🐛 Type	bug-fix
🟡 Risk	medium

Summary

Batches grouped grep/rg recall writes for SQLite recovery storage into a single transaction, moving hashing/gzip compression before the write lock and consolidating eviction into one final pass. This fixes a cap/retention regression where hints were advertised for rows that didn't survive eviction, causing rtk recall to exit 1.

Review Checklist

• Tests present
• Breaking change
• Docs updated

Linked issues: #3946

---

Analyzed automatically by wshm · This is an automated analysis, not a human review.