Repository navigation
feat(*): the files a main-conversation command wrote reach the desk diff - #692
Conversation
gloryfromca
left a comment
There was a problem hiding this comment.
Blocking: commands using exec's supported working_dir override are still absent from the desk diff.
I found one blocking issue, marked inline. I covered the repository rules and canonical Runtime/Web UI terms, the full diff, the runtime-to-RPC-to-live/replay web callers, the relevant #670 history, backward compatibility of optional wire fields, test changes for weakening, and the documented Runtime/Web UI boundary.
Verification: uv run pytest tests/test_workdir_snapshot.py tests/test_agent_loop_session_stamps.py tests/test_rpc_session.py tests/test_rpc_spine.py -q (257 passed); affected ui-web Vitest files (371 passed after npm ci --prefix ui-web; the first attempt could not start because Vitest was not installed); npm run type-check --prefix ui-web and npm run gen:check --prefix ui-web passed; focused Ruff checks passed.
…wrote reach the desk diff The desk builds the main conversation's changed-files rows from the file tools' own arguments and results, so a file a shell command created, rewrote or removed with a glob left no row: a command returns its output and names no file. #670 closed that gap for every sub-agent lane by listing the working directory either side of a command; the main lane now does the same, around its own `exec` and only that, off the event loop. What the listing finds and the call did not name itself travels as a new `tool.complete.file_written` list of `{path, created, size, lines}`, stored on the tool entry under the same name so `session.resume` replays it; a deletion the listing finds joins `file_removed` with no body, since the file was gone before anything read it. The page draws a created file as an `A` row with its line count and a rewritten one as a bare `M`, neither with a patch, and a row with no patch opens the file itself rather than an empty pane. A row a file tool already made this turn is left alone. Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
…listing off the loop Review round. A file tool that touched a file a command had already left behind this turn opened a second row: the listing keys its row by the path the runtime resolved and the tool by the path the model typed, so the exact-key lookup missed. The tool now takes over the listing's row under its own spelling, keeping the verdict that the file was new; a removal's bare row is left alone. The line count of every created file was read on the event loop right after the two walks had been moved off it; the counting now runs in the same worker. The listing also skips the agent's own `.raven` directory inside the working directory, where the checkpoint keeps a shadow git repo that another turn's commit would otherwise surface as this command's files. Tests: the real ExecTool through the listing, a non-text and an over-cap created file, the served ToolEvent hop, the counting thread, the skipped directory, the live fixture frame. Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
…rn's Review round two. The listing around a command walked the turn's working directory, but exec takes a working_dir of its own and a command sent to one writes its files there: the call succeeded and the desk showed nothing for it. The tool now says where a call's files land (ExecTool.listing_root, resolved the way execute resolves its cwd), and both lanes that list around a command -- the main conversation and the in-process sub-agent -- ask it through snapshot.root_for. A command sent to a registered machine leaves nothing on this disk, so it takes no listing at all rather than an empty one. Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
The effective-cwd fix resolves the prior blocker, including the matching in-process subagent path and the remote-machine no-listing case. I found no new issue in this revision.
I covered the repository rules and canonical Runtime terms, the full change with the new delta, the shell/snapshot/main-loop/subagent callers and relevant history, backward compatibility, tests for weakening, and the Runtime layer boundaries. Verification: uv run pytest tests/test_workdir_snapshot.py tests/test_agent_loop_session_stamps.py tests/test_shell_approval.py tests/test_subagent_manager.py tests/test_rpc_session.py tests/test_rpc_spine.py -q (728 passed); focused Ruff and git diff --check passed. The original finding thread was replied to and resolved.
f204814 to
de4dcbf
Compare
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
This revision is the reviewed change rebased onto current main. git range-diff shows the three PR commits unchanged except the generated RPC component-count header required by the newer base, and the resolved effective-cwd fix remains intact.
I covered the repository rules and current Runtime/Web UI context, the full github/main...HEAD diff, the rebase delta and commit history, runtime/RPC/live-and-replay callers, backward compatibility, tests for weakening, and the documented architecture boundaries. Verification on the rebased head: focused Python suite (728 passed), affected web suite (371 passed), web type-check and RPC generation drift check passed, focused Ruff and git diff --check passed. The thread I opened remains resolved.
## Summary A file an `exec` command wrote reached the desk diff as a bare row. The loop listed the working directory on either side of the command (#692), so it knew a file had changed but never what it changed from: a rewrite showed a bare `M` with no counts, and opening any such row fell back to the file viewer instead of the diff pane. File-tool edits were unaffected; this only concerned commands (`python3 gen.py > out.json`, `echo x >> notes.md`, `sed -i ...`). This supersedes #826. That PR measured the change in the loop, around a call named `exec`. Here the exec tool measures it itself, around the command it runs, and hands it back the way it already hands back the files a command removed; the loop only forwards it. How it works: - **The tool measures.** `raven/agent/tools/command_writes.py` lists the directory before and after the command and, where the checkpoint's shadow repo covers it, stages the tree first, so a rewritten or removed file is shown against what it held. The result travels on a new `ToolResult.written` / `ToolOutput.written` (`FileWrite` on the tool paper, contract version 32 -> 33, and 33 -> 34 for `FileRemoval.withheld`), beside the existing `removed`. - **The repo is handed in.** Tools do not import the loop shell (import-linter), so `ExecTool` holds the shadow repo through a `ShadowTree` protocol and a resolver the loop passes in (`AgentLoop._command_shadow`: the turn's repo, only where it covers the command's directory). The built-in is built with it, and a plugin's same-name `exec` replacement (raven-code's `CodeExecTool`) is handed it through `measure_writes` before it is registered, so it measures as the built-in does, as the loop-level listing it replaces did for any tool named `exec`. Both happen before the registry admits the tool, because the registry reads the ceiling off the spec it admits. - **Every command stages afresh inside its own call**, into a per-process index of its own in the shadow repo (never the index the per-turn commit reads), and waits up to 120 s. Whenever there is no snapshot to measure against -- the tree cannot be staged at all (checkpoint off, `working_dir` outside the repo, git error) or it did not finish within the wait -- the command runs all the same and is still listed, so its files are reported without the text they held; where a repo covers the directory, its rules still decide what text may be shown. A late staging is left running, so a later command usually finds it done. What follows the command is bounded too: the shadow repo's reads get 60 s (past it every file goes out without text) and the whole post-command step 120 s (past it the call returns the command's output with no record of its files), so a slow measurement never costs the output. `measure_writes` raises the tool's registry ceiling by what measuring may take (the 120 s wait and the 120 s bound), so the built-in's goes from 660 s to 900 s and a replacement's own ceiling keeps its margin. A git read whose caller stops waiting kills its process. - **Warm-up as a session opens.** The first staging of a directory hashes every file. `session.create` / `session.resume` call `ExecTool.warm` for the directory the session resolves to, and a session opened by its first message (a channel's) is warmed at the start of that turn. `warm` returns at once and runs on the staging thread, so it overlaps the model's first reply. - **One rule for what text may be shown.** Every file a command created, rewrote or removed, including the ones its own removal watch caught by name, carries text only where the shadow repo's rules would store it (`git check-ignore --no-index` against its excludes and the user's `.gitignore`, the rules alone, not what the index holds). So a `.env` or a key never enters the event or the stored session, even one the repo staged before an ignore rule named it, and even when the staging failed: the rules need no tree. Where no shadow repo can vouch for a file, a created file carries its counts only. - **Loop.** The exec-only listing branch in `turn_path.py` is gone; the loop turns `result.written` into `file_written`. Besides that it has the resolver above and the one-line warm-up at turn start. The registry now keeps `removed` / `written` on a result it rewrites as an error (a command whose output begins with "Error" still wrote what it wrote). A removal whose body the rules kept back goes out marked `withheld` (a new defaulted field on `FileRemoval`), and the turn's removal watch fills in a bare removal from what the turn wrote only when it is not: with the checkpoint off every bare removal stays fillable, as before, and a withheld body is never put back. - **Wire and page** (unchanged from #826). `file_written` entries gain optional `added` / `removed` / `diff`; the diffs share a 512 KiB budget per call, past which the counts still go. Contract updated in `rpc-schema/openrpc.json`, `raven/rpc/models.py` and both generated clients. ui-web draws these rows with their hunks, so they open in the diff pane with +/- in the desk list, and `fromUnified` now skips only a real `---`/`+++` header pair, so a removed `-- note` line is no longer dropped. - **Robustness.** Stagings run on daemon threads with blocking git calls (an asyncio subprocess still starting when its loop closes hangs the close on CPython 3.12 macOS). `add` and `write-tree` run under one lock per index; the staging `add` has its own 600 s ceiling; the turn commit waits for a warm-up's repo setup instead of racing it on the config lock. The setup's future and every staging future are running from creation, so a cancelled waiter cannot cancel them, and the warm-up and staging threads settle them whatever happens, so no staging is left published for nobody to finish (the cancellation 0xKT and gloryfromca measured on #826). Git queries now encode paths with `surrogateescape`, so a POSIX file name that is not UTF-8 no longer raises after the command has run (the follow-up gloryfromca raised on #826). Behaviour that moved with the measurement: staging now happens after the permission gate has ruled, so a denied command is never staged; a background command (`run_in_background`) takes no listing, since its files land after the call returns; a sub-agent's `exec` does not list (its runner lists every call itself, as before). Not covered: files the shadow repo excludes (`*.log`, `.env`, the user's `.gitignore`) get no diff, text files over 256 KB get no counts, a same-name `exec` that is not an `ExecTool` (none ships) reports no files, and existing sessions keep the rows they stored. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed - `make test-python` -- 27077 passed, 108 skipped, 3 failed (on the previous head; the focused set below is on this one). The 3 failures (`test_rpc_files.py::test_a_host_without_libreoffice_says_so`, `test_simulation_scenario.py::test_a_fresh_trial_refreshes_the_subagent_homes_before_it_runs`, `test_simulation_suite.py::test_a_stopped_run_is_marked_in_its_record_and_finished_with_its_evidence`) fail the same way on `github/main` on this machine. - `uv run --frozen --all-extras pytest tests/test_runtime_checkpoint.py tests/test_runtime_checkpoint_deep.py tests/test_agent_loop_session_stamps.py tests/test_agent_loop_workdir.py tests/test_rpc_session.py tests/test_shell_command_writes.py tests/test_shell_file_removals.py tests/test_tool_registry_execute.py tests/test_tool_registry_channels.py tests/test_tool_registry_session_overlay.py tests/test_rpc_spine.py tests/test_workdir_snapshot.py tests/test_shell_approval.py tests/test_contracts_two_tier_ledger.py tests/test_kernel_budget.py tests/test_removal_watch.py tests/test_agents_code_flow_exec.py tests/test_agents_code_tools_plugin.py tests/test_agents_code_launcher.py tests/test_subagent_dag_runner.py tests/test_agents_research_ask_user_text.py tests/test_living_docs.py tests/test_tracing_compact.py -q --idle-ceiling-strict` -- 1231 passed. - `make lint-python lint-types lint-imports` -- clean (10 import contracts kept). `npm run gen:check --prefix ui-web`, `npm run type-check --prefix ui-web`, `npm run lint --prefix ui-web` (0 errors, 4 pre-existing warnings), `npm run lint:rpc --prefix ui-tui` -- clean. `npm test --prefix ui-web` -- 3107 passed. - Mutation checks, each restored afterwards and each turning at least one test red (48): a staging timeout not falling back to running the command without a diff; the wiring not asking the tool to record writes, or not handing it the repo; the resolver dropping its `covers` check, or ignoring the turn's binding; the registry dropping `written` on unwrap or on an error result; the loop ignoring `written`; no turn-start warm-up; `session.create` / `session.resume` not warming; a failing warm-up failing the open; created-file text ignoring `trackable`; the diff budget ignored; removals losing the staged text; staging a directory too large to list; reading the written files on the event loop; a named removal reported twice; the ceiling under the sum of the wait, the command cap and the measuring bound; no bound on the shadow repo's reads after the command, or on the whole post-command step; a late read keeping a named removal's text; a cancelled git read left running; the checkpoint's timeout not being a `TimeoutError`; `warm` ignoring `record_writes`; a background command measured; the setup future not marked running; the warm-up or a command's staging not settling its future on an unexpected error; a rewritten or removed file read without the rule; a named removal's text kept without the rule; the repo dropped when the staging failed, or its rules applied only where there is a tree; a tree-less staging still read for blobs; the removal watch refilling a withheld body, or never refilling an unknown one; a named, listed or timed-out removal not marked withheld where the repo kept its body back, or a listed one marked where no repo ruled; measuring switched on only for the built-in, so a plugin's same-name `exec` reports nothing; `measure_writes` not raising the ceiling, or raising it on every call; the built-in or the replacement measured only after the registry admitted it, so the admitted ceiling stays at 660 s; a failed staging not warned about, or warned about on every command. - Real gateway (`raven serve` from this branch, isolated `RAVEN_HOME`, real model), in a cold 12,775-file / 428 MB directory on a machine running endpoint protection at load average 13-14. The first message opened the session and the warm-up started with it; the model's first `exec` (append to a `notes.md` Raven had never touched, and create `made_by_exec.txt`) waited out the cold staging and took 37.8 s in total. The second `exec` took 3.7 s, with the staging under the 1 s sampling interval. The desk lists `M notes.md +2` and `A made_by_exec.txt +3`; `notes.md` opens in the diff pane with both commands' hunks, and a reload draws the same rows. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes User-visible changes: - An `exec` now runs only after the working directory has been staged (or the 120 s wait has passed). On a warm directory that is well under a second; the first command in a large directory the shadow repo has never indexed waits for the first staging (tens of seconds on a slow disk), which the session-open warm-up usually hides. Past 120 s the command runs without a diff. - Opening a session starts one background `git add` per directory per process, writing into the same `.raven/shadow.git` the checkpoint already uses. Each process keeps its own `exec-<pid>.index` there; files older than 7 days are pruned. - No file content is stored beyond what the checkpoint already commits, and the same excludes apply. With `runtime.checkpoint.policy = "never"` a command's rewrite is reported bare, as today, and a created file carries counts only. - The new `FileWritten` fields are optional, so older clients ignore them. `ToolResult` / `ToolOutput` gain a defaulted `written` field and `FileRemoval` a defaulted `withheld`; contract version 34. Rollback: revert the commit. The staging indexes in `.raven/shadow.git` are inert without the code and are pruned by age. ## Related Issues Supersedes #826 --------- Co-authored-by: arelchan <204152633+arelchan@users.noreply.github.com> Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
…iff (#692) ## Summary The desk builds the main conversation's changed-files rows on the client, from the file tools' own arguments and results. A file a shell command created (`python3 build.py > out.txt`), rewrote (`echo more >> notes.md`) or removed through a glob left no row, because a command returns its output and names no file. #670 closed that gap for every sub-agent lane by listing the working directory on either side of a command; this does the same for the main conversation, which needs a wire field because its rows are drawn on the page rather than read from a run record. - `raven/agent/loop/turn_path.py`: around a call named `exec` and only that, the directory the command runs in is listed before and after the call with `snapshot.take`, off the event loop. The tool says where that is (`ExecTool.listing_root`, resolved the way `execute` resolves its cwd): the turn's working directory unless the call names a `working_dir`, and no listing at all for a command sent to a registered `machine`, whose files are not on this disk. The in-process sub-agent lane asks the same question through `snapshot.root_for`, so a sub-agent's `working_dir` command is recorded too. Created and modified paths the call did not name itself (its `file_change`, its removals) become the new `tool.complete.file_written` list of `{path, created, size, lines}`; a deletion the listing finds joins `file_removed` with no body, since the file was gone before anything read it. Stored on the tool entry under the same name and passed through `session.resume`, so a reloaded page draws what the live one drew. - Wire: `FileWritten` in `raven/rpc/models.py`, on `ToolCompletePayload` and `TranscriptMessage`; `rpc-schema/openrpc.json` and both generated clients regenerated. `lines` is an integer for a created text file of 256 KB or less and `null` otherwise, including every rewrite, whose old contents the listing never held. - Page: a created file is an `A` row with its line count, a rewritten one a bare `M`, neither with a patch; a row a file tool already made this turn is left alone (its arguments say more than a listing can); the reload path replays the stored field. A change row with no patch to show opens the file itself instead of an empty patch pane (a removal keeps its pane; a task file's row, keyed by node rather than path, is untouched). The offline fixture's GTM run gains one such command so the demo desk shows both rows and the fixture-shape gate sees the field sent. Not in this PR: the ACP translator still surfaces no file changes to an ACP client, the same as for `file_removed`. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run from the worktree on top of `0133c67c` (main with #656, #657 and #670; the branch was rebased onto it when main's own contract additions collided with the generated client, which was regenerated from the merged schema): - `.venv/bin/ruff check raven tests`; `.venv/bin/ruff format --check raven tests` -- clean - `.venv/bin/python -m pytest tests/test_agent_loop_session_stamps.py tests/test_rpc_spine.py tests/test_rpc_session.py tests/test_rpc_schema_match.py tests/test_turn_wire_shape_conformance.py tests/test_spine_events.py tests/test_contracts_two_tier_ledger.py tests/test_kernel_budget.py tests/test_workdir_snapshot.py tests/test_subagent_activity.py tests/test_shell_file_removals.py tests/test_acp_updates.py -q` -- 952 passed - `make test-python` (`uv run --frozen --all-extras pytest -q`) -- 25798 passed, 109 skipped - `npm test --prefix ui-web` (the whole suite, including the fixture-shape gate) -- 200 files, 2767 tests passed; `npm run lint --prefix ui-web` -- 0 errors (4 pre-existing hook-dependency warnings in files this change does not touch); `npm run type-check --prefix ui-web`, `npm run gen:check --prefix ui-web`, `npm run lint:rpc --prefix ui-tui` -- clean - Mutation checks, each restored afterwards: the listing taken around every tool; a created file reported as a rewrite; the held-out result path dropped; the storage rename disabled; `file_written` emitted when empty; the key dropped from `session.resume`; `file_written=None` on the served `ToolEvent`; a count of 0 on a decode failure; the 256 KB cap dropped; the created row drawn as `M`; the existing-row guard dropped; the ninth argument dropped in the stage; the file door removed, ignoring the kind, ignoring a task key; the fixture no longer sending the field (stored, then live); the row takeover removed and its removal guard dropped; `.raven` back in the walk; the count back on the loop; the shell ignoring its bound directory -- every one turned at least one new test red. - Review round two (reviewer's blocker): a command with a `working_dir` of its own is listed there, in both lanes, and a `machine` command takes no listing -- `tests/test_shell_approval.py` (the root follows the arguments), `tests/test_workdir_snapshot.py` (`root_for`), `tests/test_agent_loop_session_stamps.py` (the real `ExecTool` with `working_dir`; a remote command takes no listing) and `tests/test_subagent_manager.py` (a sub-agent command outside its workspace); each red with the root pinned back to the lane's directory or the machine guard dropped. - Review round: one row per file when a file tool follows a command that already left the file behind (the listing keys by the resolved path, the tool by the path the model typed); the listing skips `.raven`, where the checkpoint keeps a shadow git repo inside the working directory; the line counting moved into the worker beside the walks; a test through the real `ExecTool` (its cwd is the directory the listing walks), the served `ToolEvent` hop, a non-text and an over-cap created file, and the live fixture frame. - Real gateway (`raven serve` from this branch, isolated `RAVEN_HOME`, default model `deepseek/deepseek-v4-flash`): the main agent runs `printf "gen one\ngen two\n" > work/main_generated.txt` and `echo more >> work/main_shell_mod.md` beside its file-tool steps. Before this change neither command left a row. After: the desk diff tab draws `A work/main_generated.txt +2` and `M work/main_shell_mod.md` beside `M work/main_keep.md +1-1` (edit_file), a red `D work/main_old.md -2` (`rm`) and `A work/e2e_main_new.md +2` (write_file); a file one command created and the next removed draws nothing; reopening the page (`session.resume`) draws the identical five rows, and the stored tool entries carry `file_written` with `created: true, lines: 2` and `created: false, lines: null`. The same run repeated on the final tree (after the review round) and its reload draw the same rows. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk Behaviour change: every main-conversation `exec` now walks the turn's working directory twice, bounded at 20000 entries with the machinery directories (`.git`, `node_modules`, `.venv`, caches) skipped and run off the event loop; a tree past the bound records nothing from listings, and the file tools' own rows stand as before. A created file's line count reads the file (up to 256 KB). Wire: one new optional key on `tool.complete` and on `session.resume` tool entries, absent on every call that is not a command; a client that does not read it sees what it saw before. Rollback is reverting the commits. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
## Summary A file an `exec` command wrote reached the desk diff as a bare row. The loop listed the working directory on either side of the command (#692), so it knew a file had changed but never what it changed from: a rewrite showed a bare `M` with no counts, and opening any such row fell back to the file viewer instead of the diff pane. File-tool edits were unaffected; this only concerned commands (`python3 gen.py > out.json`, `echo x >> notes.md`, `sed -i ...`). This supersedes #826. That PR measured the change in the loop, around a call named `exec`. Here the exec tool measures it itself, around the command it runs, and hands it back the way it already hands back the files a command removed; the loop only forwards it. How it works: - **The tool measures.** `raven/agent/tools/command_writes.py` lists the directory before and after the command and, where the checkpoint's shadow repo covers it, stages the tree first, so a rewritten or removed file is shown against what it held. The result travels on a new `ToolResult.written` / `ToolOutput.written` (`FileWrite` on the tool paper, contract version 32 -> 33, and 33 -> 34 for `FileRemoval.withheld`), beside the existing `removed`. - **The repo is handed in.** Tools do not import the loop shell (import-linter), so `ExecTool` holds the shadow repo through a `ShadowTree` protocol and a resolver the loop passes in (`AgentLoop._command_shadow`: the turn's repo, only where it covers the command's directory). The built-in is built with it, and a plugin's same-name `exec` replacement (raven-code's `CodeExecTool`) is handed it through `measure_writes` before it is registered, so it measures as the built-in does, as the loop-level listing it replaces did for any tool named `exec`. Both happen before the registry admits the tool, because the registry reads the ceiling off the spec it admits. - **Every command stages afresh inside its own call**, into a per-process index of its own in the shadow repo (never the index the per-turn commit reads), and waits up to 120 s. Whenever there is no snapshot to measure against -- the tree cannot be staged at all (checkpoint off, `working_dir` outside the repo, git error) or it did not finish within the wait -- the command runs all the same and is still listed, so its files are reported without the text they held; where a repo covers the directory, its rules still decide what text may be shown. A late staging is left running, so a later command usually finds it done. What follows the command is bounded too: the shadow repo's reads get 60 s (past it every file goes out without text) and the whole post-command step 120 s (past it the call returns the command's output with no record of its files), so a slow measurement never costs the output. `measure_writes` raises the tool's registry ceiling by what measuring may take (the 120 s wait and the 120 s bound), so the built-in's goes from 660 s to 900 s and a replacement's own ceiling keeps its margin. A git read whose caller stops waiting kills its process. - **Warm-up as a session opens.** The first staging of a directory hashes every file. `session.create` / `session.resume` call `ExecTool.warm` for the directory the session resolves to, and a session opened by its first message (a channel's) is warmed at the start of that turn. `warm` returns at once and runs on the staging thread, so it overlaps the model's first reply. - **One rule for what text may be shown.** Every file a command created, rewrote or removed, including the ones its own removal watch caught by name, carries text only where the shadow repo's rules would store it (`git check-ignore --no-index` against its excludes and the user's `.gitignore`, the rules alone, not what the index holds). So a `.env` or a key never enters the event or the stored session, even one the repo staged before an ignore rule named it, and even when the staging failed: the rules need no tree. Where no shadow repo can vouch for a file, a created file carries its counts only. - **Loop.** The exec-only listing branch in `turn_path.py` is gone; the loop turns `result.written` into `file_written`. Besides that it has the resolver above and the one-line warm-up at turn start. The registry now keeps `removed` / `written` on a result it rewrites as an error (a command whose output begins with "Error" still wrote what it wrote). A removal whose body the rules kept back goes out marked `withheld` (a new defaulted field on `FileRemoval`), and the turn's removal watch fills in a bare removal from what the turn wrote only when it is not: with the checkpoint off every bare removal stays fillable, as before, and a withheld body is never put back. - **Wire and page** (unchanged from #826). `file_written` entries gain optional `added` / `removed` / `diff`; the diffs share a 512 KiB budget per call, past which the counts still go. Contract updated in `rpc-schema/openrpc.json`, `raven/rpc/models.py` and both generated clients. ui-web draws these rows with their hunks, so they open in the diff pane with +/- in the desk list, and `fromUnified` now skips only a real `---`/`+++` header pair, so a removed `-- note` line is no longer dropped. - **Robustness.** Stagings run on daemon threads with blocking git calls (an asyncio subprocess still starting when its loop closes hangs the close on CPython 3.12 macOS). `add` and `write-tree` run under one lock per index; the staging `add` has its own 600 s ceiling; the turn commit waits for a warm-up's repo setup instead of racing it on the config lock. The setup's future and every staging future are running from creation, so a cancelled waiter cannot cancel them, and the warm-up and staging threads settle them whatever happens, so no staging is left published for nobody to finish (the cancellation 0xKT and gloryfromca measured on #826). Git queries now encode paths with `surrogateescape`, so a POSIX file name that is not UTF-8 no longer raises after the command has run (the follow-up gloryfromca raised on #826). Behaviour that moved with the measurement: staging now happens after the permission gate has ruled, so a denied command is never staged; a background command (`run_in_background`) takes no listing, since its files land after the call returns; a sub-agent's `exec` does not list (its runner lists every call itself, as before). Not covered: files the shadow repo excludes (`*.log`, `.env`, the user's `.gitignore`) get no diff, text files over 256 KB get no counts, a same-name `exec` that is not an `ExecTool` (none ships) reports no files, and existing sessions keep the rows they stored. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed - `make test-python` -- 27077 passed, 108 skipped, 3 failed (on the previous head; the focused set below is on this one). The 3 failures (`test_rpc_files.py::test_a_host_without_libreoffice_says_so`, `test_simulation_scenario.py::test_a_fresh_trial_refreshes_the_subagent_homes_before_it_runs`, `test_simulation_suite.py::test_a_stopped_run_is_marked_in_its_record_and_finished_with_its_evidence`) fail the same way on `github/main` on this machine. - `uv run --frozen --all-extras pytest tests/test_runtime_checkpoint.py tests/test_runtime_checkpoint_deep.py tests/test_agent_loop_session_stamps.py tests/test_agent_loop_workdir.py tests/test_rpc_session.py tests/test_shell_command_writes.py tests/test_shell_file_removals.py tests/test_tool_registry_execute.py tests/test_tool_registry_channels.py tests/test_tool_registry_session_overlay.py tests/test_rpc_spine.py tests/test_workdir_snapshot.py tests/test_shell_approval.py tests/test_contracts_two_tier_ledger.py tests/test_kernel_budget.py tests/test_removal_watch.py tests/test_agents_code_flow_exec.py tests/test_agents_code_tools_plugin.py tests/test_agents_code_launcher.py tests/test_subagent_dag_runner.py tests/test_agents_research_ask_user_text.py tests/test_living_docs.py tests/test_tracing_compact.py -q --idle-ceiling-strict` -- 1231 passed. - `make lint-python lint-types lint-imports` -- clean (10 import contracts kept). `npm run gen:check --prefix ui-web`, `npm run type-check --prefix ui-web`, `npm run lint --prefix ui-web` (0 errors, 4 pre-existing warnings), `npm run lint:rpc --prefix ui-tui` -- clean. `npm test --prefix ui-web` -- 3107 passed. - Mutation checks, each restored afterwards and each turning at least one test red (48): a staging timeout not falling back to running the command without a diff; the wiring not asking the tool to record writes, or not handing it the repo; the resolver dropping its `covers` check, or ignoring the turn's binding; the registry dropping `written` on unwrap or on an error result; the loop ignoring `written`; no turn-start warm-up; `session.create` / `session.resume` not warming; a failing warm-up failing the open; created-file text ignoring `trackable`; the diff budget ignored; removals losing the staged text; staging a directory too large to list; reading the written files on the event loop; a named removal reported twice; the ceiling under the sum of the wait, the command cap and the measuring bound; no bound on the shadow repo's reads after the command, or on the whole post-command step; a late read keeping a named removal's text; a cancelled git read left running; the checkpoint's timeout not being a `TimeoutError`; `warm` ignoring `record_writes`; a background command measured; the setup future not marked running; the warm-up or a command's staging not settling its future on an unexpected error; a rewritten or removed file read without the rule; a named removal's text kept without the rule; the repo dropped when the staging failed, or its rules applied only where there is a tree; a tree-less staging still read for blobs; the removal watch refilling a withheld body, or never refilling an unknown one; a named, listed or timed-out removal not marked withheld where the repo kept its body back, or a listed one marked where no repo ruled; measuring switched on only for the built-in, so a plugin's same-name `exec` reports nothing; `measure_writes` not raising the ceiling, or raising it on every call; the built-in or the replacement measured only after the registry admitted it, so the admitted ceiling stays at 660 s; a failed staging not warned about, or warned about on every command. - Real gateway (`raven serve` from this branch, isolated `RAVEN_HOME`, real model), in a cold 12,775-file / 428 MB directory on a machine running endpoint protection at load average 13-14. The first message opened the session and the warm-up started with it; the model's first `exec` (append to a `notes.md` Raven had never touched, and create `made_by_exec.txt`) waited out the cold staging and took 37.8 s in total. The second `exec` took 3.7 s, with the staging under the 1 s sampling interval. The desk lists `M notes.md +2` and `A made_by_exec.txt +3`; `notes.md` opens in the diff pane with both commands' hunks, and a reload draws the same rows. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes User-visible changes: - An `exec` now runs only after the working directory has been staged (or the 120 s wait has passed). On a warm directory that is well under a second; the first command in a large directory the shadow repo has never indexed waits for the first staging (tens of seconds on a slow disk), which the session-open warm-up usually hides. Past 120 s the command runs without a diff. - Opening a session starts one background `git add` per directory per process, writing into the same `.raven/shadow.git` the checkpoint already uses. Each process keeps its own `exec-<pid>.index` there; files older than 7 days are pruned. - No file content is stored beyond what the checkpoint already commits, and the same excludes apply. With `runtime.checkpoint.policy = "never"` a command's rewrite is reported bare, as today, and a created file carries counts only. - The new `FileWritten` fields are optional, so older clients ignore them. `ToolResult` / `ToolOutput` gain a defaulted `written` field and `FileRemoval` a defaulted `withheld`; contract version 34. Rollback: revert the commit. The staging indexes in `.raven/shadow.git` are inert without the code and are pruned by age. ## Related Issues Supersedes #826 --------- Co-authored-by: arelchan <204152633+arelchan@users.noreply.github.com> Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
Summary
The desk builds the main conversation's changed-files rows on the client, from the file tools' own arguments and results. A file a shell command created (
python3 build.py > out.txt), rewrote (echo more >> notes.md) or removed through a glob left no row, because a command returns its output and names no file. #670 closed that gap for every sub-agent lane by listing the working directory on either side of a command; this does the same for the main conversation, which needs a wire field because its rows are drawn on the page rather than read from a run record.raven/agent/loop/turn_path.py: around a call namedexecand only that, the directory the command runs in is listed before and after the call withsnapshot.take, off the event loop. The tool says where that is (ExecTool.listing_root, resolved the wayexecuteresolves its cwd): the turn's working directory unless the call names aworking_dir, and no listing at all for a command sent to a registeredmachine, whose files are not on this disk. The in-process sub-agent lane asks the same question throughsnapshot.root_for, so a sub-agent'sworking_dircommand is recorded too. Created and modified paths the call did not name itself (itsfile_change, its removals) become the newtool.complete.file_writtenlist of{path, created, size, lines}; a deletion the listing finds joinsfile_removedwith no body, since the file was gone before anything read it. Stored on the tool entry under the same name and passed throughsession.resume, so a reloaded page draws what the live one drew.FileWritteninraven/rpc/models.py, onToolCompletePayloadandTranscriptMessage;rpc-schema/openrpc.jsonand both generated clients regenerated.linesis an integer for a created text file of 256 KB or less andnullotherwise, including every rewrite, whose old contents the listing never held.Arow with its line count, a rewritten one a bareM, neither with a patch; a row a file tool already made this turn is left alone (its arguments say more than a listing can); the reload path replays the stored field. A change row with no patch to show opens the file itself instead of an empty patch pane (a removal keeps its pane; a task file's row, keyed by node rather than path, is untouched). The offline fixture's GTM run gains one such command so the demo desk shows both rows and the fixture-shape gate sees the field sent.Not in this PR: the ACP translator still surfaces no file changes to an ACP client, the same as for
file_removed.Type
Verification
Run from the worktree on top of
0133c67c(main with #656, #657 and #670; the branch was rebased onto it when main's own contract additions collided with the generated client, which was regenerated from the merged schema):.venv/bin/ruff check raven tests;.venv/bin/ruff format --check raven tests-- clean.venv/bin/python -m pytest tests/test_agent_loop_session_stamps.py tests/test_rpc_spine.py tests/test_rpc_session.py tests/test_rpc_schema_match.py tests/test_turn_wire_shape_conformance.py tests/test_spine_events.py tests/test_contracts_two_tier_ledger.py tests/test_kernel_budget.py tests/test_workdir_snapshot.py tests/test_subagent_activity.py tests/test_shell_file_removals.py tests/test_acp_updates.py -q-- 952 passedmake test-python(uv run --frozen --all-extras pytest -q) -- 25798 passed, 109 skippednpm test --prefix ui-web(the whole suite, including the fixture-shape gate) -- 200 files, 2767 tests passed;npm run lint --prefix ui-web-- 0 errors (4 pre-existing hook-dependency warnings in files this change does not touch);npm run type-check --prefix ui-web,npm run gen:check --prefix ui-web,npm run lint:rpc --prefix ui-tui-- cleanMutation checks, each restored afterwards: the listing taken around every tool; a created file reported as a rewrite; the held-out result path dropped; the storage rename disabled;
file_writtenemitted when empty; the key dropped fromsession.resume;file_written=Noneon the servedToolEvent; a count of 0 on a decode failure; the 256 KB cap dropped; the created row drawn asM; the existing-row guard dropped; the ninth argument dropped in the stage; the file door removed, ignoring the kind, ignoring a task key; the fixture no longer sending the field (stored, then live); the row takeover removed and its removal guard dropped;.ravenback in the walk; the count back on the loop; the shell ignoring its bound directory -- every one turned at least one new test red.Review round two (reviewer's blocker): a command with a
working_dirof its own is listed there, in both lanes, and amachinecommand takes no listing --tests/test_shell_approval.py(the root follows the arguments),tests/test_workdir_snapshot.py(root_for),tests/test_agent_loop_session_stamps.py(the realExecToolwithworking_dir; a remote command takes no listing) andtests/test_subagent_manager.py(a sub-agent command outside its workspace); each red with the root pinned back to the lane's directory or the machine guard dropped.Review round: one row per file when a file tool follows a command that already left the file behind (the listing keys by the resolved path, the tool by the path the model typed); the listing skips
.raven, where the checkpoint keeps a shadow git repo inside the working directory; the line counting moved into the worker beside the walks; a test through the realExecTool(its cwd is the directory the listing walks), the servedToolEventhop, a non-text and an over-cap created file, and the live fixture frame.Real gateway (
raven servefrom this branch, isolatedRAVEN_HOME, default modeldeepseek/deepseek-v4-flash): the main agent runsprintf "gen one\ngen two\n" > work/main_generated.txtandecho more >> work/main_shell_mod.mdbeside its file-tool steps. Before this change neither command left a row. After: the desk diff tab drawsA work/main_generated.txt +2andM work/main_shell_mod.mdbesideM work/main_keep.md +1-1(edit_file), a redD work/main_old.md -2(rm) andA work/e2e_main_new.md +2(write_file); a file one command created and the next removed draws nothing; reopening the page (session.resume) draws the identical five rows, and the stored tool entries carryfile_writtenwithcreated: true, lines: 2andcreated: false, lines: null. The same run repeated on the final tree (after the review round) and its reload draw the same rows.Relevant tests pass locally
Relevant lint / type checks pass locally
User-facing docs or screenshots are updated when needed
Risk
Behaviour change: every main-conversation
execnow walks the turn's working directory twice, bounded at 20000 entries with the machinery directories (.git,node_modules,.venv, caches) skipped and run off the event loop; a tree past the bound records nothing from listings, and the file tools' own rows stand as before. A created file's line count reads the file (up to 256 KB). Wire: one new optional key ontool.completeand onsession.resumetool entries, absent on every call that is not a command; a client that does not read it sees what it saw before. Rollback is reverting the commits.Related Issues
N/A