feat: Snapchat arroyo.db rows the WAL removes, with row provenance columns

[abrignoni]

Follow-up to #1051. Contributes to #957, where the reasoning behind the approach is written up.

What prompted it

While validating #1051 I noticed the row count moves the wrong way. arroyo.db read on its own has 11 conversation_message rows. Read with its write-ahead log applied it has 8. The log carries removals, not just additions, so the smaller number is the correct current state and the larger one is stale.

Those rows are still in the database file as of its last checkpoint, and nothing surfaced them. Three of the eight decode to plaintext at the same protobuf path the live rows use. One whole conversation is present in conversation, feed_entry and conversation_identifier before the log is applied and absent from all three after, with a participant that does not appear in the Friend table in main.db at all.

One table per subject, not two

An earlier revision of this PR added separate "Not In Committed State" artifacts. This version instead adds three provenance columns to the existing two artifacts, so recovered rows sit in chronological context with the live ones:

column	values
Record Origin	Live or Recovered. Populated on every row.
Recovery Method	the technique, currently WAL diff. Empty on Live rows.
Recovery Location	where in the evidence the row came from. Empty on Live rows.

Record Origin is a closed two-value set so a viewer can branch on it, and it is populated on every row rather than blank for live ones, because a blank cell is ambiguous between "live" and "not populated". Recovery Method is naturally empty on live rows. So whichever column an examiner picks to display under a LAVA conversation bubble, they get a sensible result: the first marks every message, the second marks only the recovered ones.

Combining also removes the timeline double-count that the split design carried.

Naming was checked, not invented

Against all five cores before choosing:

• Record Source was the obvious pick and is already taken, meaning "which table did this row come from", in allTrails.py and slack.py.
• Source is overloaded 51 times across the cores (31 iLEAPP, 17 RLEAPP, 3 VLEAPP) for source file or table.
• Recovery Status is taken in galleryVault.py for how many bytes of a decrypted file came back.
• L360noshowalerts.py already does live-vs-recovered with Source / WAL Location / WAL Offset, which is the precedent this generalises.

Record Origin, Recovery Method and Recovery Location are unused anywhere in the five cores.

Deliberately not called Deleted. Seventeen ALEAPP artifacts already use Deleted, Is Deleted, Deleted At for the app's own deletion flag. Reusing that name for tool-recovery provenance would conflate "the app recorded this as deleted" with "our tool recovered this", which are different claims that can appear on the same row. Recovered says how the row was obtained, which is known, rather than what happened to it, which is not.

Method

The same file is opened twice through SQLite, once with immutable=1 which ignores the log, and once with mode=ro which applies it, and the results are compared on the primary key.

SQLite does the decoding on both sides, so named columns, type affinity and overflow pages come for free and there is no hand-written B-tree parser to maintain. That is the main argument for it over the frame parser in #956, and the two find different things, so neither replaces the other. Written up properly in #957.

Comparing row counts would have missed most of this. A count check flags 2 of 30 tables in arroyo.db; comparing primary keys flags 6. The four it misses hold the same number of rows under different keys, and the conversation finding is one of those four.

What it is not

Not deleted-record carving. It does not read freelist pages, unallocated space or freeblocks, and it does not parse WAL frames, so records that only ever lived inside the log are not recovered. It compares keys rather than full row content, so a row whose key survives while its content changed is not reported either.

Why a Recovered row is not in the live read is not established, and the notes say so. Removal by the app, a server re-sync rewriting those pages, and deletion are all consistent with the same result. On this image most of the recovered message rows carry Team Snapchat broadcast content, which is consistent with a re-sync.

It yields no Recovered rows when no -wal accompanies the database, verified by removing the sidecar and confirming both reads agree at 11.

Evidence handling

The immutable=1 read is strictly read-only and, unlike mode=ro, does not even create a -shm sidecar. Confirmed against a lone copy of the database: no new files afterwards and the same MD5. It routes through the same get_sqlite_db_path() that open_sqlite_db_readonly() uses, so Windows long paths and URI-special characters behave identically.

Scope

The diff helper is kept local to snapchat.py rather than put into ilapfuncs.py, so #957 gets to decide the shared shape rather than inheriting mine. It is written so lifting it into core is a move rather than a rewrite.

Validation

Profile run against corpus key hc_pixel8pro_a17 (Android 17, com.snapchat.android vc 302522).

artifact	rows
Snapchat - Messages (arroyo.db)	16 (8 Live, 8 Recovered)
Snapchat - Conversations (arroyo.db)	5 (4 Live, 1 Recovered)

• No parser errors.
• Zero primary keys appear under both origins, so no row is double reported.
• Live rows have both recovery columns empty; every Recovered row has both populated.
• Rows sort chronologically across the merged set.
• TSV parses to 16 and 5 records at a uniform 22 and 21 fields. One recovered message contains an embedded newline and csv.writer quotes it correctly; it needs a real CSV reader rather than a line-based check to verify.
• LAVA manifest counts correct, provenance columns present in the artifact table, data_views on the messages artifact only.

CI run locally: check_claim_language.py clean, check_html_safety.py clean, pylint 10.00/10 at CI flags, lint_changed.py no new warnings.

Note on the LAVA side

Until the conversation view can display an arbitrary column under the bubble, recovered rows render there like any other message. The columns are visible immediately in the HTML report, the TSV and the LAVA table view, so only the bubble lags. James is building the picker that closes it.

Not covered

One image. The zero case is proven by removing the sidecar rather than against a second corpus whose WAL is genuinely checkpointed, and that is the corpus I would most like to run this against.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

[abrignoni]

Updated with the recovery boundary stated where an examiner will actually see it.

Checked where each field surfaces before relying on it, which changed the plan:

field	HTML report	LAVA
description	yes, on the artifact page	yes
notes	no	yes
logfunc	yes, on index.html and Script Logs	n/a

notes never reaches the HTML report in ALEAPP. The pre-existing MEO note behaves the same way, so this is how the tool works rather than anything specific here. Putting the warning only in notes would have left an HTML reader with no warning at all, which is the same silent gap this change exists to close.

So it goes in three places:

1. Both descriptions now name the Record Origin column and end with "WAL frames are not parsed, so absence of a message here is not evidence it did not exist." That renders directly above the table.

2. A per-image log line on the report index, read from the WAL header and 24-byte frame headers only, no page images loaded:

   Snapchat arroyo.db-wal holds 1011 frames of 4096 bytes (386 in the current log generation, 625 from previous generations). This artifact does not parse WAL frames, so records held only in them are not reported and absence of a message from the Snapchat arroyo.db artifacts is not evidence that it did not exist.

   The salt split matters: a frame whose salt pair does not match the header is from a previous log generation the current one has cycled past, so it tells the examiner how much older content is sitting there. Silent when no -wal accompanies the database or the header is not a WAL, both tested.

3. notes carries the measured size of the gap for LAVA readers, plus a sqlite3 command to re-derive a Recovered row without this tool.

Why the gap is quantified rather than hand-waved. A one-off frame parser written during development read a further 29 conversation_message rows that neither view reports, across 10 conversations, 9 of which appear in neither view of the conversation table, with timestamps spanning 2025-11-18 to 2026-07-24. Investigation-grade and a lower bound: only live cell arrays on table-leaf pages were parsed, so freeblocks and unallocated space inside those pages may hold more.

Frame parsing is deliberately not added here. Of the 158 records that parser matched, 111 had 32 columns and 47 had 33 — schema drift inside a single WAL file. A per-artifact parser assuming the current 33-column schema would misalign 111 of them and emit confidently wrong values, which is the worst failure mode available. That belongs in the shared capability, not in a third independent implementation after L360noshowalerts.py and Honeyboard. Tracked in #957.

Counts unchanged: 16 message rows (8 Live, 8 Recovered) and 5 conversation rows (4 Live, 1 Recovered) on hc_pixel8pro_a17. Conversation view kept. All four checks green.