harness/store: decode the JSONL tail window losslessly

[theamazinghenk]

JsonlAppendStore::append fails permanently on any stream whose file happens to
have a UTF-8 continuation byte at offset len - 4096:

append store read error: stream did not contain valid UTF-8

The append fails, so the record is lost. Nothing else reports it — the file that
would carry the trace is the one that was not written. And it does not recover:
the failed append left the file unchanged, so len is the same next time, so
len - 4096 points at the same broken byte. One unlucky em-dash freezes the
stream forever.

JsonlAppendStore::next_offset (src/harness/store/mod.rs:328) avoids reading
the whole file by seeking to a fixed-size tail window:

let start = len.saturating_sub(window);   // window = 4096
file.seek(SeekFrom::Start(start))?;
let mut buf = String::new();
file.read_to_string(&mut buf)?;           // ← here

start is an arbitrary byte offset — nobody placed it on a character boundary.
read_to_string requires the entire read to be valid UTF-8, so if start lands
inside a multi-byte character the read fails, and with it the whole append.

With ASCII this never happens: every character is one byte, so every offset is a
character boundary. It shows up as soon as the content is not English. An em-dash
(—) and a euro sign (€) are three bytes each; the more non-ASCII in the file,
the higher the chance that byte len - 4096 falls inside a character. Note that
only a continuation byte (10xxxxxx, 128–191) is fatal — a leading byte is a
valid character boundary and decodes fine, which is part of why this looks
intermittent.

It is effectively a die rolled on every append: len changes with every written
line, so the window slides and points at a different byte each time. One bad roll
is enough, because after that the file stops growing.

Observed in production on a session journal of 8003 bytes. 8003 - 4096 = 3907,
and bytes 3905–3907 are e2 80 94 — U+2014, an em-dash. The read therefore
starts on the third byte of that character. The file has been stuck at 8003
bytes since the first failure. Of the 17 journal files in that directory exactly
one was frozen; the other sixteen happen to have an ordinary byte at
len - 4096.

Read bytes, then decode: read_to_end into a Vec<u8>, then
String::from_utf8_lossy.

Lossy decoding is not a concession here, for a reason already present in the
function: when the window does not start at the beginning of the file
(start > 0), next_offset discards the first line anyway, because it may be
cut in half (complete_from). The damaged bytes are by definition at the start
of the window, i.e. in exactly the line that is discarded. No replacement
character can ever reach a line that is used. And when start == 0 the window
begins at the start of the file — always a character boundary — so
from_utf8_lossy replaces nothing.

Thirteen lines, nine of them comment. No new dependency. No behaviour change for
files that work today.

A file that is already frozen starts growing again. No migration or repair
step is needed: once next_offset stops tripping, the next append on such a
stream succeeds and the file grows past its stuck length.

jsonl_append_survives_a_tail_window_that_starts_mid_character in
src/harness/store/test.rs. It does not guess: it pads the first record one byte
at a time until byte len - 4096 is provably a UTF-8 continuation byte
(b & 0b1100_0000 == 0b1000_0000), and only then writes the file. So it fails
always rather than occasionally.

Checked in both directions:

• on unpatched 5e026cd it fails with exactly the production message —
  Validation("append store read error: stream did not contain valid UTF-8");
• with the patch it passes, and all 34 tests in harness::store stay green.

Summary by CodeRabbit

• Bug Fixes

  • Fixed JSONL journal appends that could fail when the read window began in the middle of a multi-byte UTF-8 character.
  • Append operations now continue normally while safely discarding incomplete or damaged leading bytes.

• Tests

  • Added coverage to verify records remain intact and offsets advance correctly in this scenario.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 81a7a74b-780a-47ff-8561-8879824e4f8e

📥 Commits

Reviewing files that changed from the base of the PR and between bdb18f3 and 3e5b9d5.

📒 Files selected for processing (2)

• src/harness/store/mod.rs
• src/harness/store/test.rs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

JsonlAppendStore::next_offset now reads tail bytes and uses lossy UTF-8 decoding. A regression test verifies that appending remains successful when the tail window starts inside a multi-byte character.

Changes

JSONL append handling

Layer / File(s)	Summary
Tail decoding and regression validation src/harness/store/mod.rs, src/harness/store/test.rs	next_offset decodes tail bytes with String::from_utf8_lossy. The regression test verifies that the append returns offset 2 and that the store length becomes 3.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 3e5b9

This localized change allows append-store tail windows to handle UTF-8 continuation bytes without losing records or freezing affected files. No actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: senamakel, codeghost21

Poem

A rabbit found a broken byte,
Mid-character, out of sight.
The journal hops and grows once more,
With three bright records in its store.
No lost append beneath the moon.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the JSONL tail-window decoding change. However, “losslessly” is technically imprecise because the implementation uses lossy UTF-8 conversion at an incomplete character bou…
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title clearly identifies the JSONL tail-window decoding change. However, “losslessly” is technically imprecise because the implementation uses lossy UTF-8 conversion at an incomplete character boundary.

✨ Finishing Touches 💡 1

⚔️ Resolve merge conflicts 💡

• Resolve merge conflict in branch upstream-02

Warning

Your free Security trial is over. An organization admin can activate Security or dismiss this notice.

---

_{Comment @coderabbitai help to get the list of available commands.}

[theamazinghenk]

Duplicate — this fix is already merged upstream as #109. Closing to avoid noise. Apologies for the duplicate.

[tinysweeper]

tinysweeper found nothing blocking. Approving.

             $0.0065 · 63,577 in / 1,584 out · 8,131 cached (13%) · deepseek/deepseek-v4-flash, openrouter/openai/text-embedding-3-small, z-ai/glm-5.2 · 218 embedded
critique:    $0.0019 · 23,038 in / 247 out   · 0 cached (0%)      · deepseek/deepseek-v4-flash
security:    $0.0031 · 22,534 in / 840 out   · 8,131 cached (36%) · deepseek/deepseek-v4-flash, z-ai/glm-5.2
tests:       $0.0010 · 11,932 in / 88 out    · 0 cached (0%)      · deepseek/deepseek-v4-flash
description: $0.0004 · 4,410 in  / 111 out   · 0 cached (0%)      · deepseek/deepseek-v4-flash

[tinysweeper]

How this change flows

1 changed behaviour across 10 relationships. 6 surrounding behaviours are shown (60 graph nodes walked). 44 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["jsonl_rejects_unsafe_stream_names<br/>changed"]:::changed
  n1["new"]:::impacted
  n2["expect"]:::impacted
  n3["tick"]:::impacted
  n4["...ves_its_card_blocked_rather_than_stranded"]:::impacted
  n5["Result"]:::impacted
  n6["complete_run"]:::impacted
  n0 -->|calls| n1
  n0 -->|tests| n1
  n3 -->|calls| n2
  n3 -->|tests| n2
  n4 -->|calls| n2
  n4 -->|tests| n2
  n4 -->|calls| n3
  n4 -->|calls| n6
  n4 -->|tests| n6
  n6 -->|uses| n5
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff

Loading

_{Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.}