perf(recall): batch grouped search writes in one transaction

CONTRIBUTING.md

@@ -263,6 +263,16 @@ For how to write tests (fixtures, snapshots, token savings verification), see [d
 
 ### Pre-Commit Gate (mandatory)
 
+On Windows, `build.rs` gives Cargo-launched test workers at least an 8 MiB stack via
+`RUST_MIN_STACK`, matching the CLI executable's existing main-thread reserve.
+The full Clap command graph can exceed Rust's default 2 MiB worker stack in debug
+builds. This applies to local `cargo test` as well as CI; all parser tests and
+assertions still run. Cargo also supplies this environment to `cargo run` and
+its child processes, preserving a valid larger `RUST_MIN_STACK` setting; a
+directly launched installed CLI is unaffected. If you launch a compiled test
+executable directly instead of using Cargo, set
+`$env:RUST_MIN_STACK = "8388608"` in PowerShell first.
+
 All three must pass before any PR:
 
 ```bash

build.rs

@@ -32,6 +32,20 @@ fn set_windows_stack_size() {
 fn main() {
     set_windows_stack_size();
 
+    if std::env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("windows") {
+        // libtest parses the same graph on worker threads, whose 2 MiB default
+        // is independent of the executable's main-thread reserve. Cargo also
+        // supplies rustc-env values during `cargo test` and `cargo run`, but
+        // this does not change the environment of the installed CLI.
+        println!("cargo:rerun-if-env-changed=RUST_MIN_STACK");
+        let stack_size = std::env::var("RUST_MIN_STACK")
+            .ok()
+            .and_then(|value| value.parse::<usize>().ok())
+            .unwrap_or(0)
+            .max(8 * 1024 * 1024);
+        println!("cargo:rustc-env=RUST_MIN_STACK={stack_size}");
+    }
+
     check_cmds_modules_declared();
 
     let filters_dir = Path::new("src/filters");

docs/TELEMETRY.md

@@ -153,10 +153,14 @@ Under the EU General Data Protection Regulation, you have the right to:
 
 ## Erasure Procedure
 
-1. Run `rtk telemetry forget` — this disables telemetry, deletes your device salt, ping marker, and local tracking database (`history.db`), then sends an erasure request to the server.
+1. Run `rtk telemetry forget` — this disables telemetry, deletes your device salt, ping marker, local tracking database (`history.db`), recall database (including SQLite sidecars), and legacy/posthook tee logs, then sends an erasure request to the server.
 2. If the server is unreachable, the CLI prints your full device hash and fallback instructions to email contact@rtk-ai.app for manual erasure.
 3. You can also email contact@rtk-ai.app directly to request manual erasure.
 
+Recovery cleanup uses `RTK_RECALL_DB` / `[retriever].database_path` and `RTK_TEE_DIR` / `[retriever].tee_directory` (including migrated `[tee].directory`), falling back to the standard recall database and tee directory. Both backends are cleared regardless of the current recovery mode or disable switches. Missing files are harmless. Cleanup removes top-level tee files named `<epoch>_<sanitized-slug>.log` and files matching `<epoch-ms>_<pid>-<sequence>_<sanitized-slug>.log` directly inside the `posthook` subdirectory. Unrelated files and other subdirectories are preserved. File symlinks and symlinked tee or `posthook` directories are refused and cleanup failures return a nonzero exit status.
+
+Stop other RTK commands before erasing recovery data. An exclusive SQLite lock check refuses a busy recall database; removal is not an atomic barrier against another process starting afterward. This removes currently stored output; subsequent commands can store new output unless recovery is also disabled with `rtk config recall disabled`. Configuration and unrelated local files are retained.
+
 ## Data Handling
 
 - Telemetry endpoint URL and auth token are injected at **compile time** via `option_env!()` — they are not in the source code

docs/contributing/RECALL_BENCHMARK.md

@@ -0,0 +1,120 @@
+# Search recall write benchmark
+
+`python3 scripts/benchmark-recall-search.py /path/to/before /path/to/after`
+compares release binaries on Linux. Build both with the same Rust toolchain and
+release profile. The script creates 100 files containing 80 deterministic matching
+lines each, uses `rtk grep --max 2500 -rn needle corpus`, and discards the first
+three runs. It interleaves 20 measured runs per binary and recovery configuration
+in a fixed random order. Config, recall, tracking, and tee data live in a temporary
+directory; telemetry is disabled. No existing RTK data is used.
+
+The four configurations are disabled, tee, SQLite without compression, and SQLite
+with gzip. The difference between disabled and raw SQLite includes hashing,
+SQLite writes, retention, and hint construction. The gzip/raw difference includes
+compression **and the effect of writing a smaller blob**; it is not a pure gzip
+microbenchmark. The JSON output includes all timings, mean, standard deviation,
+median, output size/hash, and the number of SQLite hints. Capture references are
+unique per run, so normalize those identifiers before comparing output hashes.
+Use `--max 200` to measure the default display cap, which stores only eight per-file tails and one skipped-files
+block, rather than all 100 tails. `--files`, `--lines`, and `--runs` are adjustable.
+
+To time the storage phases separately, run:
+
+```sh
+cargo test --release --lib test_benchmark_recall_batch_phases -- --ignored --nocapture
+```
+
+This opt-in test measures preparation (hashing, capture IDs, byte caps, line
+counting, copying, and optional gzip) separately from the transaction (cached
+connection lookup, prepared payload/capture inserts, stats, retention and capture
+pruning, survivor lookups, commit). Each codec uses a fresh store so immutable
+payload deduplication cannot reuse the other codec. It runs on 100 per-file
+payloads, with three warm-ups and 20 measured runs. The preparation
+difference between compression on/off isolates gzip from the database work.
+There is no timing assertion in the regular test suite.
+
+## Storage contract
+
+Grouped search keeps independently recoverable per-file capture references, payloads,
+offsets, and hint positions. Payload preparation happens before the writer
+transaction so compression does not hold SQLite's write lock. Eligible payloads,
+immutable captures, statistics, and byte-accounting updates commit atomically.
+Payload eviction runs once. Orphan capture references are pruned before insertion
+(to prevent resurrection) and after eviction. No hint is emitted until commit
+succeeds. An insert or eviction failure rolls the eligible batch back and leaves
+search output available without recovery links.
+
+If a single search exceeds the payload count/byte limits or the capture-record
+limit, those caps still apply. Hints are returned only for captures and payloads
+surviving the final retention pass, using each capture's immutable offset and line
+count. Oversized entries are skipped without suppressing eligible peers. Existing
+payloads retain their original encoding, whose actual stored size must fit the
+budget too. An entirely ineligible batch does not commit prune/eviction changes.
+
+Tee mode keeps its file-per-tail behavior. Batching itself adds no schema, default
+limit, gzip codec, or connection-cache change beyond its predecessor's contracts.
+
+## Final stacked comparison (2026-10-12)
+
+This comparison isolates batching after the companion changes for bounded stats,
+storage budgets, sensitive-output notices, forget cleanup, invocation accounting,
+and immutable capture references. Both binaries use those same features. Before
+is the exact predecessor `e20801b6ae93a7b47d9d51b93b5f899699353363`; after adds
+only the reconciled batch change. Same Rust 1.99.0 release profile, Linux x86_64
+shared 9-vCPU host, 30 interleaved measured runs and three warm-ups per case:
+
+| 100 files, 2,500 displayed lines | Before median | After median | Before mean ± SD | After mean ± SD |
+| --- | ---: | ---: | ---: | ---: |
+| Disabled | 20.48 ms | 19.91 ms | 21.61 ± 3.34 ms | 20.47 ± 1.94 ms |
+| Tee | 22.83 ms | 22.06 ms | 23.76 ± 2.78 ms | 23.22 ± 2.85 ms |
+| SQLite raw | 107.92 ms | 25.84 ms | 109.86 ± 13.00 ms | 27.16 ± 3.66 ms |
+| SQLite gzip | 116.28 ms | 31.57 ms | 114.69 ± 15.21 ms | 32.90 ± 4.63 ms |
+
+Both SQLite cases retain 100 independent per-file capture hints. The gzip median
+is about 73% lower against this matched predecessor (raw about 76% lower), with
+disabled/tee controls within 1 ms. This larger gain is specific to the integrated
+stack: it also removes repeated per-file capture-pruning work. It must not be
+presented as a 73% improvement against unmodified develop.
+
+At the default 200-result display cap, nine captures are stored: gzip median
+34.31 → 32.40 ms and raw 24.91 → 22.52 ms. A one-file check has no observed
+regression: gzip 5.29 → 4.80 ms; disabled control 4.74 → 4.25 ms. The small-case
+differences are within the shared host's variability.
+
+Capture metadata accumulates across repeated runs and is bounded by retention.
+Each before/after case uses its own isolated store and identical run count. These
+results describe this workload and host, not a timing guarantee.
+
+## Original develop comparison (2026-10-12)
+
+Release builds using Rust 1.99.0, Linux x86_64, on the same shared 9-vCPU cloud
+host. Before is develop `311ebcbfd8efdb09b03219e7615a153b5b5e3c97`; after is the
+initial standalone batch commit `49919c9`, before the companion features were
+integrated. These numbers do not describe the final stacked version. Thirty interleaved measured runs per case, after three warm-ups:
+
+| 100 files, 2,500 displayed lines | Before median | After median | Before mean ± SD | After mean ± SD |
+| --- | ---: | ---: | ---: | ---: |
+| Disabled | 19.85 ms | 20.23 ms | 21.51 ± 4.10 ms | 22.78 ± 10.08 ms |
+| Tee | 22.56 ms | 22.43 ms | 23.62 ± 3.35 ms | 23.87 ± 3.78 ms |
+| SQLite raw | 27.58 ms | 23.40 ms | 29.59 ± 4.96 ms | 24.12 ± 2.99 ms |
+| SQLite gzip | 31.30 ms | 26.54 ms | 33.23 ± 4.23 ms | 28.32 ± 4.43 ms |
+
+Both SQLite cases emitted 100 per-file hints before and after. Their median
+end-to-end times fell about 15%. The disabled/tee controls were essentially
+unchanged at the median; the shared host introduced considerable noise/outliers.
+These are observations for this workload, not portable performance guarantees.
+
+With the default `--max 200`, only nine outputs are stored (eight per-file tails
+plus one skipped-files block). SQLite gzip median was 32.71 → 30.48 ms and raw was
+22.85 → 21.65 ms. The remaining gzip cost is important on the larger combined
+skipped-files block; batching does not remove compression work.
+
+Reproduce both comparisons with:
+
+```sh
+python3 scripts/benchmark-recall-search.py /path/to/before /path/to/after --runs 30
+python3 scripts/benchmark-recall-search.py /path/to/before /path/to/after --runs 30 --max 200
+```
+
+These are fresh measurements of current develop, not a restatement of the older
+`ddc21826` results in issue #3946.

docs/contributing/TECHNICAL.md

@@ -290,11 +290,21 @@ Analytics commands (`rtk gain`, `rtk cc-economics`, `rtk session`) query this da
 On command failure (non-zero exit code) — or when a filter truncates a long list:
 
 1. Raw unfiltered output is stored in a content-addressed sqlite database (`~/.local/share/rtk/recall.db`, gzip, byte-faithful)
-2. A hint line is printed: `[full output: rtk recall 3f9c2a81d4e7]` (failures) or `[+N hidden: rtk recall <hash>]` (truncated lists)
+2. A hint line is printed: `[full output: rtk recall 3f9c2a81d4e7] (unfiltered; may contain sensitive data)` (failures) or `[+N hidden: rtk recall <hash>] (unfiltered; may contain sensitive data)` (truncated lists)
 3. LLM agents run `rtk recall <hash>` to get back exactly what was elided instead of re-running the command
 
+The notice also follows legacy tee hints, outside the brackets so the recovery command or shell-quoted path stays intact. Recovery storage does not redact sensitive values, including fields omitted from a summary; review recovered output before sharing it.
+
 The mode is selectable via `rtk config recall <sqlite|tee|disabled>` — `tee` keeps the legacy per-file behavior (`~/.local/share/rtk/tee/{epoch}_{slug}.log`). Recovery never affects command output or exit code. `rtk gain --recalls` reports how often elided output is actually consulted, per filter.
 
+The optional `retriever.max_total_bytes` budget counts encoded payload bytes,
+not physical SQLite/WAL size. Inserts, trigger-maintained totals and eviction
+share one immediate transaction; a returned hash cannot be evicted by its own
+write. The compatibility default (`0`) keeps the existing count/age-only limits.
+`rtk recall --gc` applies retention and reclaims free pages; `--purge` removes
+stored output while keeping aggregate recall statistics. Maintenance opens only
+the existing configured recall store and validates its schema before mutation.
+
 > **Details**: [`src/core/README.md`](../../src/core/README.md) covers the recall store, tee mode configuration, and the rotation strategy.
 
 ---

docs/guide/analytics/gain.md

@@ -307,7 +307,9 @@ The two sections are never merged because the data quality differs:
 - **SQLITE (exact)**: reads go through `rtk recall <hash>`, the only access path, so the count is exact.
 - **TEE (approximate)**: in legacy [tee mode](../getting-started/configuration.md#recall-system), reads are shell commands (`tail`, `cat`, `grep`, …) observed by the rewrite hook before execution. Editor or assistant file-tool reads are invisible, denied commands are not counted, and the per-file dedup window is finite — so the `≥` rate is approximate, not exact. A high rate is still a reliable signal that the cap is too aggressive.
 
-Stats survive entry eviction and retention cleanup: calibration data is kept even after the underlying outputs are purged.
+Stats survive output-entry eviction and retention cleanup. Separately, statistics are capped at 500 `(filter, mode)` rows across both modes so dynamic command names cannot grow the database indefinitely. When a new row exceeds this limit, RTK drops the row with the fewest elisions plus recalls (oldest insertion breaks ties), keeping the newly added row. Existing oversized stores are trimmed when opened.
+
+Counters for retained rows continue accumulating, but an evicted row's history is discarded, not added to another filter. If that filter appears again its counters restart, so these are bounded calibration statistics, not lifetime totals. A recall of an older output can therefore have no retained matching elision.
 
 ## Troubleshooting
 

docs/guide/getting-started/configuration.md

@@ -42,6 +42,7 @@ ignore_files = ["*.lock", "*.min.js", "*.min.css"]
 mode = "sqlite"             # sqlite (default) | tee (legacy files) | disabled
 max_entry_bytes = 10485760  # sqlite: 10 MiB per entry
 max_entries = 200           # sqlite: FIFO cap (0 = no cap)
+max_total_bytes = 0         # sqlite: aggregate encoded blob cap (0 = no cap)
 retention_days = 30         # sqlite: age eviction (0 = off)
 compression = true          # sqlite: gzip blobs (lossless)
 # database_path = "/custom/recall.db"
@@ -105,18 +106,26 @@ When a command fails — or a filter trims a long list — RTK persists the full
 
 ```
 FAILED: 2/15 tests
-[full output: rtk recall 36365b69eda6]
+[full output: rtk recall c_7d16363b493245f9857c323349f42ba0] (unfiltered; may contain sensitive data)
 ```
 
-Your AI assistant runs `rtk recall <hash>` exactly as printed in the hint — that is the whole agent interface. For humans inspecting the store: `rtk recall <hash> --full | --from N | --lines N | --grep PAT` and `rtk recall --list`. Storage is byte-faithful (`BLOB` + lossless gzip); the stored input is the captured command text, as with the previous tee files.
+Your AI assistant runs `rtk recall <reference>` exactly as printed in the hint — that is the whole agent interface. For humans inspecting the store: `rtk recall <reference> --full | --from N | --lines N | --grep PAT` and `rtk recall --list`. Storage is byte-faithful (`BLOB` + lossless gzip); the stored input is the captured command text, as with the previous tee files.
+
+New hints identify one immutable capture: repeating the same command output cannot change an earlier hint’s saved bytes, starting line, working directory, exit status or truncation metadata. Identical retained payloads are deduplicated. Use the complete `c_...` reference; capture prefixes are not accepted. Existing legacy hashes and their unique prefixes remain readable.
+
+References expire when their payload is evicted or their capture record ages out of the bounded 1,000-record history. Expired references fail rather than pointing at a later run, even if that run saves identical bytes. Caps apply when capturing: `--full` returns all bytes saved for that capture, which may be less than the original command output.
+
+Both SQLite recall and legacy tee files keep the payload supplied by a filter without secret redaction. This may be raw captured output or reconstructed recovery text. A summary that omits a sensitive field does not remove it from stored recovery output. Review recovered content before copying it into a transcript or bug report. The same notice follows full-output and hidden-tail hints; the command or shell-quoted path inside the brackets remains unchanged.
+
+To stop future recovery storage, set `RTK_RECALL=0` for an invocation or run `rtk config recall disabled`. Neither deletes existing entries or tee files.
 
 ### Choosing the recovery mode
 
 The simplest way is the CLI — no file editing needed:
 
 ```bash
 rtk config recall           # show the active mode and its source
-rtk config recall sqlite    # hash-addressed sqlite store (default)
+rtk config recall sqlite    # capture-addressed sqlite store (default)
 rtk config recall tee       # legacy .log files in ~/.local/share/rtk/tee/
 rtk config recall disabled  # no recovery storage
 ```
@@ -130,12 +139,22 @@ To see how often your assistant actually goes back for elided output — and whi
 | `retriever.mode` | `"sqlite"` | `sqlite` (default), `tee` (legacy files), `disabled` |
 | `retriever.max_entry_bytes` | `10485760` | Per-entry storage cap (10 MiB) |
 | `retriever.max_entries` | `200` | FIFO cap on retained entries (0 = no cap) |
+| `retriever.max_total_bytes` | `0` | Combined encoded blob bytes (0 = no aggregate limit) |
 | `retriever.retention_days` | `30` | Age eviction in days (0 = off) |
 | `retriever.compression` | `true` | gzip stored blobs (lossless) |
 | `retriever.tee_max_files` | `20` | tee mode: how many files are kept before rotation |
 | `retriever.tee_max_file_size` | `1048576` | tee mode: per-file cap (1 MiB) |
 | `retriever.tee_on_success` | `false` | tee mode: also archive successful runs — the legacy `[tee] mode = "always"` |
-| Max file size | 1 MB | Truncated above this |
+
+For example, `max_total_bytes = 104857600` limits gzip blobs (or raw blobs when
+compression is off) to 100 MiB, excluding SQLite pages, indexes and WAL files.
+An individual blob larger than the budget is rejected without a recovery hint.
+`--gc` applies retention without waiting for a write; `--purge` removes all stored
+output. Both compact the configured recall database (`RTK_RECALL_DB` takes
+precedence), even when capture is disabled, while preserving recall statistics,
+tracking history and tee files. Active readers can delay WAL compaction; retry
+`--gc` after they finish. Purging does not promise secure erasure from backups or
+filesystem snapshots.
 
 ## Excluding commands from auto-rewrite