Skip to content

feat(*): the files a main-conversation command wrote reach the desk diff - #692

Merged
gloryfromca merged 3 commits into
mainfrom
feat/main_lane_exec_files
Sep 23, 2026
Merged

gloryfromca merged 3 commits into
mainfrom
feat/main_lane_exec_files

Conversation

@0xKT

@0xKT 0xKT commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Summary

The desk builds the main conversation's changed-files rows on the client, from the file tools' own arguments and results. A file a shell command created (python3 build.py > out.txt), rewrote (echo more >> notes.md) or removed through a glob left no row, because a command returns its output and names no file. #670 closed that gap for every sub-agent lane by listing the working directory on either side of a command; this does the same for the main conversation, which needs a wire field because its rows are drawn on the page rather than read from a run record.

  • raven/agent/loop/turn_path.py: around a call named exec and only that, the directory the command runs in is listed before and after the call with snapshot.take, off the event loop. The tool says where that is (ExecTool.listing_root, resolved the way execute resolves its cwd): the turn's working directory unless the call names a working_dir, and no listing at all for a command sent to a registered machine, whose files are not on this disk. The in-process sub-agent lane asks the same question through snapshot.root_for, so a sub-agent's working_dir command is recorded too. Created and modified paths the call did not name itself (its file_change, its removals) become the new tool.complete.file_written list of {path, created, size, lines}; a deletion the listing finds joins file_removed with no body, since the file was gone before anything read it. Stored on the tool entry under the same name and passed through session.resume, so a reloaded page draws what the live one drew.
  • Wire: FileWritten in raven/rpc/models.py, on ToolCompletePayload and TranscriptMessage; rpc-schema/openrpc.json and both generated clients regenerated. lines is an integer for a created text file of 256 KB or less and null otherwise, including every rewrite, whose old contents the listing never held.
  • Page: a created file is an A row with its line count, a rewritten one a bare M, neither with a patch; a row a file tool already made this turn is left alone (its arguments say more than a listing can); the reload path replays the stored field. A change row with no patch to show opens the file itself instead of an empty patch pane (a removal keeps its pane; a task file's row, keyed by node rather than path, is untouched). The offline fixture's GTM run gains one such command so the demo desk shows both rows and the fixture-shape gate sees the field sent.

Not in this PR: the ACP translator still surfaces no file changes to an ACP client, the same as for file_removed.

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

Run from the worktree on top of 0133c67c (main with #656, #657 and #670; the branch was rebased onto it when main's own contract additions collided with the generated client, which was regenerated from the merged schema):

  • .venv/bin/ruff check raven tests; .venv/bin/ruff format --check raven tests -- clean

  • .venv/bin/python -m pytest tests/test_agent_loop_session_stamps.py tests/test_rpc_spine.py tests/test_rpc_session.py tests/test_rpc_schema_match.py tests/test_turn_wire_shape_conformance.py tests/test_spine_events.py tests/test_contracts_two_tier_ledger.py tests/test_kernel_budget.py tests/test_workdir_snapshot.py tests/test_subagent_activity.py tests/test_shell_file_removals.py tests/test_acp_updates.py -q -- 952 passed

  • make test-python (uv run --frozen --all-extras pytest -q) -- 25798 passed, 109 skipped

  • npm test --prefix ui-web (the whole suite, including the fixture-shape gate) -- 200 files, 2767 tests passed; npm run lint --prefix ui-web -- 0 errors (4 pre-existing hook-dependency warnings in files this change does not touch); npm run type-check --prefix ui-web, npm run gen:check --prefix ui-web, npm run lint:rpc --prefix ui-tui -- clean

  • Mutation checks, each restored afterwards: the listing taken around every tool; a created file reported as a rewrite; the held-out result path dropped; the storage rename disabled; file_written emitted when empty; the key dropped from session.resume; file_written=None on the served ToolEvent; a count of 0 on a decode failure; the 256 KB cap dropped; the created row drawn as M; the existing-row guard dropped; the ninth argument dropped in the stage; the file door removed, ignoring the kind, ignoring a task key; the fixture no longer sending the field (stored, then live); the row takeover removed and its removal guard dropped; .raven back in the walk; the count back on the loop; the shell ignoring its bound directory -- every one turned at least one new test red.

  • Review round two (reviewer's blocker): a command with a working_dir of its own is listed there, in both lanes, and a machine command takes no listing -- tests/test_shell_approval.py (the root follows the arguments), tests/test_workdir_snapshot.py (root_for), tests/test_agent_loop_session_stamps.py (the real ExecTool with working_dir; a remote command takes no listing) and tests/test_subagent_manager.py (a sub-agent command outside its workspace); each red with the root pinned back to the lane's directory or the machine guard dropped.

  • Review round: one row per file when a file tool follows a command that already left the file behind (the listing keys by the resolved path, the tool by the path the model typed); the listing skips .raven, where the checkpoint keeps a shadow git repo inside the working directory; the line counting moved into the worker beside the walks; a test through the real ExecTool (its cwd is the directory the listing walks), the served ToolEvent hop, a non-text and an over-cap created file, and the live fixture frame.

  • Real gateway (raven serve from this branch, isolated RAVEN_HOME, default model deepseek/deepseek-v4-flash): the main agent runs printf "gen one\ngen two\n" > work/main_generated.txt and echo more >> work/main_shell_mod.md beside its file-tool steps. Before this change neither command left a row. After: the desk diff tab draws A work/main_generated.txt +2 and M work/main_shell_mod.md beside M work/main_keep.md +1-1 (edit_file), a red D work/main_old.md -2 (rm) and A work/e2e_main_new.md +2 (write_file); a file one command created and the next removed draws nothing; reopening the page (session.resume) draws the identical five rows, and the stored tool entries carry file_written with created: true, lines: 2 and created: false, lines: null. The same run repeated on the final tree (after the review round) and its reload draw the same rows.

  • Relevant tests pass locally

  • Relevant lint / type checks pass locally

  • User-facing docs or screenshots are updated when needed

Risk

Behaviour change: every main-conversation exec now walks the turn's working directory twice, bounded at 20000 entries with the machinery directories (.git, node_modules, .venv, caches) skipped and run off the event loop; a tree past the bound records nothing from listings, and the file tools' own rows stand as before. A created file's line count reads the file (up to 256 KB). Wire: one new optional key on tool.complete and on session.resume tool entries, absent on every call that is not a command; a client that does not read it sees what it saw before. Rollback is reverting the commits.

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

Related Issues

N/A

@0xKT
0xKT requested a review from LivXue as a code owner September 23, 2026 04:20
@0xKT
0xKT requested a review from gloryfromca September 23, 2026 04:20

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: commands using exec's supported working_dir override are still absent from the desk diff.

I found one blocking issue, marked inline. I covered the repository rules and canonical Runtime/Web UI terms, the full diff, the runtime-to-RPC-to-live/replay web callers, the relevant #670 history, backward compatibility of optional wire fields, test changes for weakening, and the documented Runtime/Web UI boundary.

Verification: uv run pytest tests/test_workdir_snapshot.py tests/test_agent_loop_session_stamps.py tests/test_rpc_session.py tests/test_rpc_spine.py -q (257 passed); affected ui-web Vitest files (371 passed after npm ci --prefix ui-web; the first attempt could not start because Vitest was not installed); npm run type-check --prefix ui-web and npm run gen:check --prefix ui-web passed; focused Ruff checks passed.

Comment thread raven/agent/loop/turn_path.py Outdated
@0xKT
0xKT requested a review from gloryfromca September 23, 2026 04:42
0xKT and others added 3 commits September 23, 2026 12:44
…wrote reach the desk diff

The desk builds the main conversation's changed-files rows from the file
tools' own arguments and results, so a file a shell command created, rewrote
or removed with a glob left no row: a command returns its output and names no
file. #670 closed that gap for every sub-agent lane by listing the working
directory either side of a command; the main lane now does the same, around
its own `exec` and only that, off the event loop.

What the listing finds and the call did not name itself travels as a new
`tool.complete.file_written` list of `{path, created, size, lines}`, stored on
the tool entry under the same name so `session.resume` replays it; a deletion
the listing finds joins `file_removed` with no body, since the file was gone
before anything read it. The page draws a created file as an `A` row with its
line count and a rewritten one as a bare `M`, neither with a patch, and a row
with no patch opens the file itself rather than an empty pane. A row a file
tool already made this turn is left alone.

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
…listing off the loop

Review round. A file tool that touched a file a command had already left
behind this turn opened a second row: the listing keys its row by the path the
runtime resolved and the tool by the path the model typed, so the exact-key
lookup missed. The tool now takes over the listing's row under its own
spelling, keeping the verdict that the file was new; a removal's bare row is
left alone.

The line count of every created file was read on the event loop right after
the two walks had been moved off it; the counting now runs in the same worker.
The listing also skips the agent's own `.raven` directory inside the working
directory, where the checkpoint keeps a shadow git repo that another turn's
commit would otherwise surface as this command's files.

Tests: the real ExecTool through the listing, a non-text and an over-cap
created file, the served ToolEvent hop, the counting thread, the skipped
directory, the live fixture frame.

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
…rn's

Review round two. The listing around a command walked the turn's working
directory, but exec takes a working_dir of its own and a command sent to one
writes its files there: the call succeeded and the desk showed nothing for
it. The tool now says where a call's files land (ExecTool.listing_root,
resolved the way execute resolves its cwd), and both lanes that list around a
command -- the main conversation and the in-process sub-agent -- ask it
through snapshot.root_for. A command sent to a registered machine leaves
nothing on this disk, so it takes no listing at all rather than an empty one.

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

The effective-cwd fix resolves the prior blocker, including the matching in-process subagent path and the remote-machine no-listing case. I found no new issue in this revision.

I covered the repository rules and canonical Runtime terms, the full change with the new delta, the shell/snapshot/main-loop/subagent callers and relevant history, backward compatibility, tests for weakening, and the Runtime layer boundaries. Verification: uv run pytest tests/test_workdir_snapshot.py tests/test_agent_loop_session_stamps.py tests/test_shell_approval.py tests/test_subagent_manager.py tests/test_rpc_session.py tests/test_rpc_spine.py -q (728 passed); focused Ruff and git diff --check passed. The original finding thread was replied to and resolved.

@0xKT
0xKT force-pushed the feat/main_lane_exec_files branch from f204814 to de4dcbf Compare September 23, 2026 04:51

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

This revision is the reviewed change rebased onto current main. git range-diff shows the three PR commits unchanged except the generated RPC component-count header required by the newer base, and the resolved effective-cwd fix remains intact.

I covered the repository rules and current Runtime/Web UI context, the full github/main...HEAD diff, the rebase delta and commit history, runtime/RPC/live-and-replay callers, backward compatibility, tests for weakening, and the documented architecture boundaries. Verification on the rebased head: focused Python suite (728 passed), affected web suite (371 passed), web type-check and RPC generation drift check passed, focused Ruff and git diff --check passed. The thread I opened remains resolved.

@gloryfromca
gloryfromca merged commit 5854582 into main Sep 23, 2026
29 checks passed
@gloryfromca
gloryfromca deleted the feat/main_lane_exec_files branch September 23, 2026 05:35
0xKT pushed a commit that referenced this pull request Sep 30, 2026
## Summary

A file an `exec` command wrote reached the desk diff as a bare row. The
loop listed the working directory on either side of the command (#692),
so it knew a file had changed but never what it changed from: a rewrite
showed a bare `M` with no counts, and opening any such row fell back to
the file viewer instead of the diff pane. File-tool edits were
unaffected; this only concerned commands (`python3 gen.py > out.json`,
`echo x >> notes.md`, `sed -i ...`).

This supersedes #826. That PR measured the change in the loop, around a
call named `exec`. Here the exec tool measures it itself, around the
command it runs, and hands it back the way it already hands back the
files a command removed; the loop only forwards it.

How it works:

- **The tool measures.** `raven/agent/tools/command_writes.py` lists the
directory before and after the command and, where the checkpoint's
shadow repo covers it, stages the tree first, so a rewritten or removed
file is shown against what it held. The result travels on a new
`ToolResult.written` / `ToolOutput.written` (`FileWrite` on the tool
paper, contract version 32 -> 33, and 33 -> 34 for
`FileRemoval.withheld`), beside the existing `removed`.
- **The repo is handed in.** Tools do not import the loop shell
(import-linter), so `ExecTool` holds the shadow repo through a
`ShadowTree` protocol and a resolver the loop passes in
(`AgentLoop._command_shadow`: the turn's repo, only where it covers the
command's directory). The built-in is built with it, and a plugin's
same-name `exec` replacement (raven-code's `CodeExecTool`) is handed it
through `measure_writes` before it is registered, so it measures as the
built-in does, as the loop-level listing it replaces did for any tool
named `exec`. Both happen before the registry admits the tool, because
the registry reads the ceiling off the spec it admits.
- **Every command stages afresh inside its own call**, into a
per-process index of its own in the shadow repo (never the index the
per-turn commit reads), and waits up to 120 s. Whenever there is no
snapshot to measure against -- the tree cannot be staged at all
(checkpoint off, `working_dir` outside the repo, git error) or it did
not finish within the wait -- the command runs all the same and is still
listed, so its files are reported without the text they held; where a
repo covers the directory, its rules still decide what text may be
shown. A late staging is left running, so a later command usually finds
it done. What follows the command is bounded too: the shadow repo's
reads get 60 s (past it every file goes out without text) and the whole
post-command step 120 s (past it the call returns the command's output
with no record of its files), so a slow measurement never costs the
output. `measure_writes` raises the tool's registry ceiling by what
measuring may take (the 120 s wait and the 120 s bound), so the
built-in's goes from 660 s to 900 s and a replacement's own ceiling
keeps its margin. A git read whose caller stops waiting kills its
process.
- **Warm-up as a session opens.** The first staging of a directory
hashes every file. `session.create` / `session.resume` call
`ExecTool.warm` for the directory the session resolves to, and a session
opened by its first message (a channel's) is warmed at the start of that
turn. `warm` returns at once and runs on the staging thread, so it
overlaps the model's first reply.
- **One rule for what text may be shown.** Every file a command created,
rewrote or removed, including the ones its own removal watch caught by
name, carries text only where the shadow repo's rules would store it
(`git check-ignore --no-index` against its excludes and the user's
`.gitignore`, the rules alone, not what the index holds). So a `.env` or
a key never enters the event or the stored session, even one the repo
staged before an ignore rule named it, and even when the staging failed:
the rules need no tree. Where no shadow repo can vouch for a file, a
created file carries its counts only.
- **Loop.** The exec-only listing branch in `turn_path.py` is gone; the
loop turns `result.written` into `file_written`. Besides that it has the
resolver above and the one-line warm-up at turn start. The registry now
keeps `removed` / `written` on a result it rewrites as an error (a
command whose output begins with "Error" still wrote what it wrote). A
removal whose body the rules kept back goes out marked `withheld` (a new
defaulted field on `FileRemoval`), and the turn's removal watch fills in
a bare removal from what the turn wrote only when it is not: with the
checkpoint off every bare removal stays fillable, as before, and a
withheld body is never put back.
- **Wire and page** (unchanged from #826). `file_written` entries gain
optional `added` / `removed` / `diff`; the diffs share a 512 KiB budget
per call, past which the counts still go. Contract updated in
`rpc-schema/openrpc.json`, `raven/rpc/models.py` and both generated
clients. ui-web draws these rows with their hunks, so they open in the
diff pane with +/- in the desk list, and `fromUnified` now skips only a
real `---`/`+++` header pair, so a removed `-- note` line is no longer
dropped.
- **Robustness.** Stagings run on daemon threads with blocking git calls
(an asyncio subprocess still starting when its loop closes hangs the
close on CPython 3.12 macOS). `add` and `write-tree` run under one lock
per index; the staging `add` has its own 600 s ceiling; the turn commit
waits for a warm-up's repo setup instead of racing it on the config
lock. The setup's future and every staging future are running from
creation, so a cancelled waiter cannot cancel them, and the warm-up and
staging threads settle them whatever happens, so no staging is left
published for nobody to finish (the cancellation 0xKT and gloryfromca
measured on #826). Git queries now encode paths with `surrogateescape`,
so a POSIX file name that is not UTF-8 no longer raises after the
command has run (the follow-up gloryfromca raised on #826).

Behaviour that moved with the measurement: staging now happens after the
permission gate has ruled, so a denied command is never staged; a
background command (`run_in_background`) takes no listing, since its
files land after the call returns; a sub-agent's `exec` does not list
(its runner lists every call itself, as before).

Not covered: files the shadow repo excludes (`*.log`, `.env`, the user's
`.gitignore`) get no diff, text files over 256 KB get no counts, a
same-name `exec` that is not an `ExecTool` (none ships) reports no
files, and existing sessions keep the rows they stored.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

- `make test-python` -- 27077 passed, 108 skipped, 3 failed (on the
previous head; the focused set below is on this one). The 3 failures
(`test_rpc_files.py::test_a_host_without_libreoffice_says_so`,
`test_simulation_scenario.py::test_a_fresh_trial_refreshes_the_subagent_homes_before_it_runs`,
`test_simulation_suite.py::test_a_stopped_run_is_marked_in_its_record_and_finished_with_its_evidence`)
fail the same way on `github/main` on this machine.
- `uv run --frozen --all-extras pytest tests/test_runtime_checkpoint.py
tests/test_runtime_checkpoint_deep.py
tests/test_agent_loop_session_stamps.py tests/test_agent_loop_workdir.py
tests/test_rpc_session.py tests/test_shell_command_writes.py
tests/test_shell_file_removals.py tests/test_tool_registry_execute.py
tests/test_tool_registry_channels.py
tests/test_tool_registry_session_overlay.py tests/test_rpc_spine.py
tests/test_workdir_snapshot.py tests/test_shell_approval.py
tests/test_contracts_two_tier_ledger.py tests/test_kernel_budget.py
tests/test_removal_watch.py tests/test_agents_code_flow_exec.py
tests/test_agents_code_tools_plugin.py
tests/test_agents_code_launcher.py tests/test_subagent_dag_runner.py
tests/test_agents_research_ask_user_text.py tests/test_living_docs.py
tests/test_tracing_compact.py -q --idle-ceiling-strict` -- 1231 passed.
- `make lint-python lint-types lint-imports` -- clean (10 import
contracts kept). `npm run gen:check --prefix ui-web`, `npm run
type-check --prefix ui-web`, `npm run lint --prefix ui-web` (0 errors, 4
pre-existing warnings), `npm run lint:rpc --prefix ui-tui` -- clean.
`npm test --prefix ui-web` -- 3107 passed.
- Mutation checks, each restored afterwards and each turning at least
one test red (48): a staging timeout not falling back to running the
command without a diff; the wiring not asking the tool to record writes,
or not handing it the repo; the resolver dropping its `covers` check, or
ignoring the turn's binding; the registry dropping `written` on unwrap
or on an error result; the loop ignoring `written`; no turn-start
warm-up; `session.create` / `session.resume` not warming; a failing
warm-up failing the open; created-file text ignoring `trackable`; the
diff budget ignored; removals losing the staged text; staging a
directory too large to list; reading the written files on the event
loop; a named removal reported twice; the ceiling under the sum of the
wait, the command cap and the measuring bound; no bound on the shadow
repo's reads after the command, or on the whole post-command step; a
late read keeping a named removal's text; a cancelled git read left
running; the checkpoint's timeout not being a `TimeoutError`; `warm`
ignoring `record_writes`; a background command measured; the setup
future not marked running; the warm-up or a command's staging not
settling its future on an unexpected error; a rewritten or removed file
read without the rule; a named removal's text kept without the rule; the
repo dropped when the staging failed, or its rules applied only where
there is a tree; a tree-less staging still read for blobs; the removal
watch refilling a withheld body, or never refilling an unknown one; a
named, listed or timed-out removal not marked withheld where the repo
kept its body back, or a listed one marked where no repo ruled;
measuring switched on only for the built-in, so a plugin's same-name
`exec` reports nothing; `measure_writes` not raising the ceiling, or
raising it on every call; the built-in or the replacement measured only
after the registry admitted it, so the admitted ceiling stays at 660 s;
a failed staging not warned about, or warned about on every command.
- Real gateway (`raven serve` from this branch, isolated `RAVEN_HOME`,
real model), in a cold 12,775-file / 428 MB directory on a machine
running endpoint protection at load average 13-14. The first message
opened the session and the warm-up started with it; the model's first
`exec` (append to a `notes.md` Raven had never touched, and create
`made_by_exec.txt`) waited out the cold staging and took 37.8 s in
total. The second `exec` took 3.7 s, with the staging under the 1 s
sampling interval. The desk lists `M notes.md +2` and `A
made_by_exec.txt +3`; `notes.md` opens in the diff pane with both
commands' hunks, and a reload draws the same rows.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

User-visible changes:
- An `exec` now runs only after the working directory has been staged
(or the 120 s wait has passed). On a warm directory that is well under a
second; the first command in a large directory the shadow repo has never
indexed waits for the first staging (tens of seconds on a slow disk),
which the session-open warm-up usually hides. Past 120 s the command
runs without a diff.
- Opening a session starts one background `git add` per directory per
process, writing into the same `.raven/shadow.git` the checkpoint
already uses. Each process keeps its own `exec-<pid>.index` there; files
older than 7 days are pruned.
- No file content is stored beyond what the checkpoint already commits,
and the same excludes apply. With `runtime.checkpoint.policy = "never"`
a command's rewrite is reported bare, as today, and a created file
carries counts only.
- The new `FileWritten` fields are optional, so older clients ignore
them. `ToolResult` / `ToolOutput` gain a defaulted `written` field and
`FileRemoval` a defaulted `withheld`; contract version 34.

Rollback: revert the commit. The staging indexes in `.raven/shadow.git`
are inert without the code and are pruned by age.

## Related Issues

Supersedes #826

---------

Co-authored-by: arelchan <204152633+arelchan@users.noreply.github.com>
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
LivXue pushed a commit that referenced this pull request Oct 12, 2026
…iff (#692)

## Summary

The desk builds the main conversation's changed-files rows on the
client, from the file tools' own arguments and results. A file a shell
command created (`python3 build.py > out.txt`), rewrote (`echo more >>
notes.md`) or removed through a glob left no row, because a command
returns its output and names no file. #670 closed that gap for every
sub-agent lane by listing the working directory on either side of a
command; this does the same for the main conversation, which needs a
wire field because its rows are drawn on the page rather than read from
a run record.

- `raven/agent/loop/turn_path.py`: around a call named `exec` and only
that, the directory the command runs in is listed before and after the
call with `snapshot.take`, off the event loop. The tool says where that
is (`ExecTool.listing_root`, resolved the way `execute` resolves its
cwd): the turn's working directory unless the call names a
`working_dir`, and no listing at all for a command sent to a registered
`machine`, whose files are not on this disk. The in-process sub-agent
lane asks the same question through `snapshot.root_for`, so a
sub-agent's `working_dir` command is recorded too. Created and modified
paths the call did not name itself (its `file_change`, its removals)
become the new `tool.complete.file_written` list of `{path, created,
size, lines}`; a deletion the listing finds joins `file_removed` with no
body, since the file was gone before anything read it. Stored on the
tool entry under the same name and passed through `session.resume`, so a
reloaded page draws what the live one drew.
- Wire: `FileWritten` in `raven/rpc/models.py`, on `ToolCompletePayload`
and `TranscriptMessage`; `rpc-schema/openrpc.json` and both generated
clients regenerated. `lines` is an integer for a created text file of
256 KB or less and `null` otherwise, including every rewrite, whose old
contents the listing never held.
- Page: a created file is an `A` row with its line count, a rewritten
one a bare `M`, neither with a patch; a row a file tool already made
this turn is left alone (its arguments say more than a listing can); the
reload path replays the stored field. A change row with no patch to show
opens the file itself instead of an empty patch pane (a removal keeps
its pane; a task file's row, keyed by node rather than path, is
untouched). The offline fixture's GTM run gains one such command so the
demo desk shows both rows and the fixture-shape gate sees the field
sent.

Not in this PR: the ACP translator still surfaces no file changes to an
ACP client, the same as for `file_removed`.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run from the worktree on top of `0133c67c` (main with #656, #657 and
#670; the branch was rebased onto it when main's own contract additions
collided with the generated client, which was regenerated from the
merged schema):

- `.venv/bin/ruff check raven tests`; `.venv/bin/ruff format --check
raven tests` -- clean
- `.venv/bin/python -m pytest tests/test_agent_loop_session_stamps.py
tests/test_rpc_spine.py tests/test_rpc_session.py
tests/test_rpc_schema_match.py tests/test_turn_wire_shape_conformance.py
tests/test_spine_events.py tests/test_contracts_two_tier_ledger.py
tests/test_kernel_budget.py tests/test_workdir_snapshot.py
tests/test_subagent_activity.py tests/test_shell_file_removals.py
tests/test_acp_updates.py -q` -- 952 passed
- `make test-python` (`uv run --frozen --all-extras pytest -q`) -- 25798
passed, 109 skipped
- `npm test --prefix ui-web` (the whole suite, including the
fixture-shape gate) -- 200 files, 2767 tests passed; `npm run lint
--prefix ui-web` -- 0 errors (4 pre-existing hook-dependency warnings in
files this change does not touch); `npm run type-check --prefix ui-web`,
`npm run gen:check --prefix ui-web`, `npm run lint:rpc --prefix ui-tui`
-- clean
- Mutation checks, each restored afterwards: the listing taken around
every tool; a created file reported as a rewrite; the held-out result
path dropped; the storage rename disabled; `file_written` emitted when
empty; the key dropped from `session.resume`; `file_written=None` on the
served `ToolEvent`; a count of 0 on a decode failure; the 256 KB cap
dropped; the created row drawn as `M`; the existing-row guard dropped;
the ninth argument dropped in the stage; the file door removed, ignoring
the kind, ignoring a task key; the fixture no longer sending the field
(stored, then live); the row takeover removed and its removal guard
dropped; `.raven` back in the walk; the count back on the loop; the
shell ignoring its bound directory -- every one turned at least one new
test red.
- Review round two (reviewer's blocker): a command with a `working_dir`
of its own is listed there, in both lanes, and a `machine` command takes
no listing -- `tests/test_shell_approval.py` (the root follows the
arguments), `tests/test_workdir_snapshot.py` (`root_for`),
`tests/test_agent_loop_session_stamps.py` (the real `ExecTool` with
`working_dir`; a remote command takes no listing) and
`tests/test_subagent_manager.py` (a sub-agent command outside its
workspace); each red with the root pinned back to the lane's directory
or the machine guard dropped.
- Review round: one row per file when a file tool follows a command that
already left the file behind (the listing keys by the resolved path, the
tool by the path the model typed); the listing skips `.raven`, where the
checkpoint keeps a shadow git repo inside the working directory; the
line counting moved into the worker beside the walks; a test through the
real `ExecTool` (its cwd is the directory the listing walks), the served
`ToolEvent` hop, a non-text and an over-cap created file, and the live
fixture frame.
- Real gateway (`raven serve` from this branch, isolated `RAVEN_HOME`,
default model `deepseek/deepseek-v4-flash`): the main agent runs `printf
"gen one\ngen two\n" > work/main_generated.txt` and `echo more >>
work/main_shell_mod.md` beside its file-tool steps. Before this change
neither command left a row. After: the desk diff tab draws `A
work/main_generated.txt +2` and `M work/main_shell_mod.md` beside `M
work/main_keep.md +1-1` (edit_file), a red `D work/main_old.md -2`
(`rm`) and `A work/e2e_main_new.md +2` (write_file); a file one command
created and the next removed draws nothing; reopening the page
(`session.resume`) draws the identical five rows, and the stored tool
entries carry `file_written` with `created: true, lines: 2` and
`created: false, lines: null`. The same run repeated on the final tree
(after the review round) and its reload draw the same rows.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

Behaviour change: every main-conversation `exec` now walks the turn's
working directory twice, bounded at 20000 entries with the machinery
directories (`.git`, `node_modules`, `.venv`, caches) skipped and run
off the event loop; a tree past the bound records nothing from listings,
and the file tools' own rows stand as before. A created file's line
count reads the file (up to 256 KB). Wire: one new optional key on
`tool.complete` and on `session.resume` tool entries, absent on every
call that is not a command; a client that does not read it sees what it
saw before. Rollback is reverting the commits.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
LivXue pushed a commit that referenced this pull request Oct 12, 2026
## Summary

A file an `exec` command wrote reached the desk diff as a bare row. The
loop listed the working directory on either side of the command (#692),
so it knew a file had changed but never what it changed from: a rewrite
showed a bare `M` with no counts, and opening any such row fell back to
the file viewer instead of the diff pane. File-tool edits were
unaffected; this only concerned commands (`python3 gen.py > out.json`,
`echo x >> notes.md`, `sed -i ...`).

This supersedes #826. That PR measured the change in the loop, around a
call named `exec`. Here the exec tool measures it itself, around the
command it runs, and hands it back the way it already hands back the
files a command removed; the loop only forwards it.

How it works:

- **The tool measures.** `raven/agent/tools/command_writes.py` lists the
directory before and after the command and, where the checkpoint's
shadow repo covers it, stages the tree first, so a rewritten or removed
file is shown against what it held. The result travels on a new
`ToolResult.written` / `ToolOutput.written` (`FileWrite` on the tool
paper, contract version 32 -> 33, and 33 -> 34 for
`FileRemoval.withheld`), beside the existing `removed`.
- **The repo is handed in.** Tools do not import the loop shell
(import-linter), so `ExecTool` holds the shadow repo through a
`ShadowTree` protocol and a resolver the loop passes in
(`AgentLoop._command_shadow`: the turn's repo, only where it covers the
command's directory). The built-in is built with it, and a plugin's
same-name `exec` replacement (raven-code's `CodeExecTool`) is handed it
through `measure_writes` before it is registered, so it measures as the
built-in does, as the loop-level listing it replaces did for any tool
named `exec`. Both happen before the registry admits the tool, because
the registry reads the ceiling off the spec it admits.
- **Every command stages afresh inside its own call**, into a
per-process index of its own in the shadow repo (never the index the
per-turn commit reads), and waits up to 120 s. Whenever there is no
snapshot to measure against -- the tree cannot be staged at all
(checkpoint off, `working_dir` outside the repo, git error) or it did
not finish within the wait -- the command runs all the same and is still
listed, so its files are reported without the text they held; where a
repo covers the directory, its rules still decide what text may be
shown. A late staging is left running, so a later command usually finds
it done. What follows the command is bounded too: the shadow repo's
reads get 60 s (past it every file goes out without text) and the whole
post-command step 120 s (past it the call returns the command's output
with no record of its files), so a slow measurement never costs the
output. `measure_writes` raises the tool's registry ceiling by what
measuring may take (the 120 s wait and the 120 s bound), so the
built-in's goes from 660 s to 900 s and a replacement's own ceiling
keeps its margin. A git read whose caller stops waiting kills its
process.
- **Warm-up as a session opens.** The first staging of a directory
hashes every file. `session.create` / `session.resume` call
`ExecTool.warm` for the directory the session resolves to, and a session
opened by its first message (a channel's) is warmed at the start of that
turn. `warm` returns at once and runs on the staging thread, so it
overlaps the model's first reply.
- **One rule for what text may be shown.** Every file a command created,
rewrote or removed, including the ones its own removal watch caught by
name, carries text only where the shadow repo's rules would store it
(`git check-ignore --no-index` against its excludes and the user's
`.gitignore`, the rules alone, not what the index holds). So a `.env` or
a key never enters the event or the stored session, even one the repo
staged before an ignore rule named it, and even when the staging failed:
the rules need no tree. Where no shadow repo can vouch for a file, a
created file carries its counts only.
- **Loop.** The exec-only listing branch in `turn_path.py` is gone; the
loop turns `result.written` into `file_written`. Besides that it has the
resolver above and the one-line warm-up at turn start. The registry now
keeps `removed` / `written` on a result it rewrites as an error (a
command whose output begins with "Error" still wrote what it wrote). A
removal whose body the rules kept back goes out marked `withheld` (a new
defaulted field on `FileRemoval`), and the turn's removal watch fills in
a bare removal from what the turn wrote only when it is not: with the
checkpoint off every bare removal stays fillable, as before, and a
withheld body is never put back.
- **Wire and page** (unchanged from #826). `file_written` entries gain
optional `added` / `removed` / `diff`; the diffs share a 512 KiB budget
per call, past which the counts still go. Contract updated in
`rpc-schema/openrpc.json`, `raven/rpc/models.py` and both generated
clients. ui-web draws these rows with their hunks, so they open in the
diff pane with +/- in the desk list, and `fromUnified` now skips only a
real `---`/`+++` header pair, so a removed `-- note` line is no longer
dropped.
- **Robustness.** Stagings run on daemon threads with blocking git calls
(an asyncio subprocess still starting when its loop closes hangs the
close on CPython 3.12 macOS). `add` and `write-tree` run under one lock
per index; the staging `add` has its own 600 s ceiling; the turn commit
waits for a warm-up's repo setup instead of racing it on the config
lock. The setup's future and every staging future are running from
creation, so a cancelled waiter cannot cancel them, and the warm-up and
staging threads settle them whatever happens, so no staging is left
published for nobody to finish (the cancellation 0xKT and gloryfromca
measured on #826). Git queries now encode paths with `surrogateescape`,
so a POSIX file name that is not UTF-8 no longer raises after the
command has run (the follow-up gloryfromca raised on #826).

Behaviour that moved with the measurement: staging now happens after the
permission gate has ruled, so a denied command is never staged; a
background command (`run_in_background`) takes no listing, since its
files land after the call returns; a sub-agent's `exec` does not list
(its runner lists every call itself, as before).

Not covered: files the shadow repo excludes (`*.log`, `.env`, the user's
`.gitignore`) get no diff, text files over 256 KB get no counts, a
same-name `exec` that is not an `ExecTool` (none ships) reports no
files, and existing sessions keep the rows they stored.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

- `make test-python` -- 27077 passed, 108 skipped, 3 failed (on the
previous head; the focused set below is on this one). The 3 failures
(`test_rpc_files.py::test_a_host_without_libreoffice_says_so`,
`test_simulation_scenario.py::test_a_fresh_trial_refreshes_the_subagent_homes_before_it_runs`,
`test_simulation_suite.py::test_a_stopped_run_is_marked_in_its_record_and_finished_with_its_evidence`)
fail the same way on `github/main` on this machine.
- `uv run --frozen --all-extras pytest tests/test_runtime_checkpoint.py
tests/test_runtime_checkpoint_deep.py
tests/test_agent_loop_session_stamps.py tests/test_agent_loop_workdir.py
tests/test_rpc_session.py tests/test_shell_command_writes.py
tests/test_shell_file_removals.py tests/test_tool_registry_execute.py
tests/test_tool_registry_channels.py
tests/test_tool_registry_session_overlay.py tests/test_rpc_spine.py
tests/test_workdir_snapshot.py tests/test_shell_approval.py
tests/test_contracts_two_tier_ledger.py tests/test_kernel_budget.py
tests/test_removal_watch.py tests/test_agents_code_flow_exec.py
tests/test_agents_code_tools_plugin.py
tests/test_agents_code_launcher.py tests/test_subagent_dag_runner.py
tests/test_agents_research_ask_user_text.py tests/test_living_docs.py
tests/test_tracing_compact.py -q --idle-ceiling-strict` -- 1231 passed.
- `make lint-python lint-types lint-imports` -- clean (10 import
contracts kept). `npm run gen:check --prefix ui-web`, `npm run
type-check --prefix ui-web`, `npm run lint --prefix ui-web` (0 errors, 4
pre-existing warnings), `npm run lint:rpc --prefix ui-tui` -- clean.
`npm test --prefix ui-web` -- 3107 passed.
- Mutation checks, each restored afterwards and each turning at least
one test red (48): a staging timeout not falling back to running the
command without a diff; the wiring not asking the tool to record writes,
or not handing it the repo; the resolver dropping its `covers` check, or
ignoring the turn's binding; the registry dropping `written` on unwrap
or on an error result; the loop ignoring `written`; no turn-start
warm-up; `session.create` / `session.resume` not warming; a failing
warm-up failing the open; created-file text ignoring `trackable`; the
diff budget ignored; removals losing the staged text; staging a
directory too large to list; reading the written files on the event
loop; a named removal reported twice; the ceiling under the sum of the
wait, the command cap and the measuring bound; no bound on the shadow
repo's reads after the command, or on the whole post-command step; a
late read keeping a named removal's text; a cancelled git read left
running; the checkpoint's timeout not being a `TimeoutError`; `warm`
ignoring `record_writes`; a background command measured; the setup
future not marked running; the warm-up or a command's staging not
settling its future on an unexpected error; a rewritten or removed file
read without the rule; a named removal's text kept without the rule; the
repo dropped when the staging failed, or its rules applied only where
there is a tree; a tree-less staging still read for blobs; the removal
watch refilling a withheld body, or never refilling an unknown one; a
named, listed or timed-out removal not marked withheld where the repo
kept its body back, or a listed one marked where no repo ruled;
measuring switched on only for the built-in, so a plugin's same-name
`exec` reports nothing; `measure_writes` not raising the ceiling, or
raising it on every call; the built-in or the replacement measured only
after the registry admitted it, so the admitted ceiling stays at 660 s;
a failed staging not warned about, or warned about on every command.
- Real gateway (`raven serve` from this branch, isolated `RAVEN_HOME`,
real model), in a cold 12,775-file / 428 MB directory on a machine
running endpoint protection at load average 13-14. The first message
opened the session and the warm-up started with it; the model's first
`exec` (append to a `notes.md` Raven had never touched, and create
`made_by_exec.txt`) waited out the cold staging and took 37.8 s in
total. The second `exec` took 3.7 s, with the staging under the 1 s
sampling interval. The desk lists `M notes.md +2` and `A
made_by_exec.txt +3`; `notes.md` opens in the diff pane with both
commands' hunks, and a reload draws the same rows.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

User-visible changes:
- An `exec` now runs only after the working directory has been staged
(or the 120 s wait has passed). On a warm directory that is well under a
second; the first command in a large directory the shadow repo has never
indexed waits for the first staging (tens of seconds on a slow disk),
which the session-open warm-up usually hides. Past 120 s the command
runs without a diff.
- Opening a session starts one background `git add` per directory per
process, writing into the same `.raven/shadow.git` the checkpoint
already uses. Each process keeps its own `exec-<pid>.index` there; files
older than 7 days are pruned.
- No file content is stored beyond what the checkpoint already commits,
and the same excludes apply. With `runtime.checkpoint.policy = "never"`
a command's rewrite is reported bare, as today, and a created file
carries counts only.
- The new `FileWritten` fields are optional, so older clients ignore
them. `ToolResult` / `ToolOutput` gain a defaulted `written` field and
`FileRemoval` a defaulted `withheld`; contract version 34.

Rollback: revert the commit. The staging indexes in `.raven/shadow.git`
are inert without the code and are pruned by age.

## Related Issues

Supersedes #826

---------

Co-authored-by: arelchan <204152633+arelchan@users.noreply.github.com>
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants