Handle UTF-8 JSONL tail windows losslessly

[theamazinghenk]

Summary

• read the fixed-size JSONL tail window as bytes before decoding it
• decode the window lossily so an arbitrary seek offset can start inside a multi-byte UTF-8 character
• add a deterministic regression test that forces the 4 KiB window to begin on a UTF-8 continuation byte

Problem and root cause

JsonlAppendStore::next_offset seeks to len - 4096 for larger streams and then calls read_to_string. That seek position is an arbitrary byte offset, not necessarily a UTF-8 character boundary. If it lands inside a multi-byte character, the read fails with stream did not contain valid UTF-8, so the append is rejected and the stream remains stuck at the same failing window.

The code now reads bytes and uses String::from_utf8_lossy. This is safe for the tail scan because, whenever the window starts after byte zero, the first potentially partial line is already discarded by complete_from. A window beginning at byte zero starts on a valid character boundary.

Impact

JSONL streams containing non-ASCII text no longer lose an append or become permanently stuck merely because the 4 KiB tail window starts midway through a character. Existing stuck streams recover on their next append without migration.

Validation

Validated against current upstream main at c6a5f24bd015fc2acfa423b161cedfdbc47bae12.

• regression test on the unmodified implementation: fails with Validation("append store read error: stream did not contain valid UTF-8")
• regression test with this change: passes
• cargo fmt --check
• cargo clippy --all-targets -- -D warnings
• cargo build --all-targets
• cargo test (including doctests)
• cargo run --example basic_graph

No public API changes and no new dependencies.

[coderabbitai]

Warning

Review limit reached

Your included review limit has been reached.

You’re in a promotional period — use the checkbox below to run this review for free:

• Run review for free

_{On-demand reviews are free for the next 30 days. After that, they cost $0.25 per reviewed file.}

How can I continue?

Run this review now using the option above, or comment @coderabbitai review --use-credits.

You can also wait for the limit to reset (next review available in 40 minutes), then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 02c96db8-62a5-4dc6-be02-7e784c38a988

📥 Commits

Reviewing files that changed from the base of the PR and between c6a5f24 and 8c53d06.

📒 Files selected for processing (2)

• src/harness/store/mod.rs
• src/harness/store/test.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[tinysweeper]

tinysweeper found nothing blocking. Approving.

$0.0000 · 0 in / 0 out · 216 embedded · openrouter/openai/text-embedding-3-small

[tinysweeper]

How this change flows

0 changed behaviours across 6 relationships. 6 surrounding behaviours are shown (60 graph nodes walked). 37 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["Result"]:::impacted
  n1["ThreadGoal"]:::impacted
  n2["load_cards"]:::impacted
  n3["validate_thread_id"]:::impacted
  n4["load"]:::impacted
  n5["edit"]:::impacted
  n2 -->|uses| n0
  n4 -->|uses| n0
  n4 -->|uses| n1
  n5 -->|uses| n0
  n5 -->|calls| n2
  n5 -->|calls| n3
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff

Loading

_{Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.}