feat(*): show how far the import is into the source it is on

[gloryfromca]

Summary

The rail's import row counted settled sources over the total, so its share stood still for the whole of a large source. On the live run a 287-message memory directory is 29 batches of 10 and about ten minutes at 42 percent, which the maintainer read as a hang.

• The orchestrator reports, per source, how many of its messages have landed after each batch (on_batch(platform, source_key, sent, total)), the way it already reports per-source outcomes.
• import.status carries it as a nullable current {platform, source_key, sent, total} beside phase, set while the message pass is on and null otherwise; additive and optional in the contract, both generated clients regenerated.
• The row adds that source's share to its count: (settled + sent/total) / total. A gateway restart clears the field and the row falls back to the per-source share, which is what it showed before.
• A source is counted or named, never both. The gateway stops naming a source the moment the run's progress event settles it, and leaves the one a retry is sending out of the settled count instead of out of sight. Without the first, a poller on a three-source pass saw 33, 67, 33, 47, 60, 67, 100, 67 percent, reaching 100 with a whole source still to send; without the second, a failed source being sent again was held by the count and named at the same time. The row also ignores a current while no run is on.
• The row stops at 99 percent until every source is settled: rounding carried the last source over 99.5 well before it was done, which is the 100-percent-then-wait the profile mirror's progress line already warns about.
• The row carries the current source's own counts beside the percentage (41% - 120/287), the way it already carries the phase's 1/3. One source's whole share is 1/N of the bar, so in a run of many the percentage alone moves a few times an hour; the counts move with every batch.

Type

• Fix
• Feature
• Docs
• CI / tooling
• Refactor
• Other

Verification

• uv run pytest tests/test_rpc_schema_match.py tests/test_rpc_import_sync.py tests/test_importer_orchestrator.py tests/test_importer_phases.py -q: 472 passed. tests/integration/test_import_e2e.py green on the first round of this branch.
• The reports are read out of a real pass rather than a set global: a two-source run over a fake backend that polls import.status from inside store sees [(k1,0,12), (k1,10,12), (k2,0,12), (k2,10,12)], and a poll taken in the window between one source settling and the next one's first batch sees current null both times.
• ui-web: 194 tests across the import feature and the gates pass; npm run type-check clean, npm run gen:check in sync, eslint clean on the touched feature.
• Mutation checks, each confirmed red: report the batch before backend.store rather than after it lands; report once per store attempt instead of once per landed batch; name every report after the first source of the run; drop on_progress from the gateway's run_import call; drop on_batch from it; drop the running guard, the clamp, or the zero guard from the row's share.
• Real host: a gateway built from this branch, its own RAVEN_HOME and a synthetic Claude Code home of three memory sources (5, 68 and 9 messages), driven through the wizard's data-sync step in a browser, storing into a real EverOS. The row, sampled from the DOM:

run  0% - 0/5        (first source)
run 33% - 0/68       (second source begins)
run 38% - 10/68
run 43% - 20/68
run 48% - 30/68
run 53% - 40/68
run 58% - 50/68
run 63% - 60/68
run 67% - 0/9        (third source begins)
done                 100%

The number moves with every batch inside the 68-message source, the bar width follows it, no step goes backwards, and 100 percent arrives only with the finished row. One apparent inversion appeared while two samplers were reading the page at once and did not reproduce with a single sampler; the row's poll is a fixed interval and does not serialise its reads, so an out-of-order answer could still show one. That is older than this branch and corrects itself on the next read.

• Relevant tests pass locally
• Relevant lint / type checks pass locally
• User-facing docs or screenshots are updated when needed

Risk

User-visible: the row's percentage now moves during a large source instead of stepping once per source. import.status gains one optional nullable field; older clients ignore it.

Rollback: revert the squash commit; no data migration is involved.

• Security impact considered (no new inputs, credentials or endpoints)
• Backward compatibility considered
• Rollback path is clear for risky changes

Related Issues

N/A

[gloryfromca]

Self-review round, three clean contexts (backend contract, web client, test strength). One blocker, three kept, four let go. Fixed in 202aeea.

Blocker: the source the pass is on was counted twice at every source boundary. _CURRENT held the last batch report of a source after run_import had marked it submitted, so settled + within drew it twice. Reproduced against the real run_import with the gateway's own callbacks and the row's own formula, three sources of 25 messages:

on_batch    a 25/25   settled=0 within=1.00 ->  33%
on_progress a submitted settled=1 within=1.00 ->  67%
on_batch    b  0/25   settled=1 within=0.00 ->  33%   <- backwards
...
on_progress b submitted settled=2 within=1.00 -> 100%  <- one whole source still to send
on_batch    c  0/25   settled=2 within=0.00 ->  67%   <- backwards

The window is every gap between a source settling and the next source's first batch, which is a file read wide, and it gets worse as the total gets smaller: a two-source run reads 100 percent halfway through. Fix: the gateway listens to the run's progress events and stops naming a settled source. Same sequence after the fix is monotonic 0 to 100 with no early 100.

Also taken:

• The row ignores current while no run is on. Unreachable from today's gateway, but the generated type allows it and the paused branch reuses the same number.
• The reports are now read out of a real pass in the RPC test. The previous test set _CURRENT directly, so removing on_batch= from the run_import call left every Python test green - the feature could be dead end to end.
• Three orchestrator cases that were unpinned: the reports name the source the pass is on (not the first of the run), a retried batch adds its messages once and only after the attempt that lands, and a source given up on never reports its last batch as landed. Each verified red under the matching mutation.

Named and let go:

• raven import (the CLI) does not pass on_batch; it has its own per-source bar.
• An on_batch that raises fails the source it is in, because the callbacks run inside run_import's per-source try. Pre-existing shape for on_progress; the gateway's callback is a dict assignment.
• import.status.current stays on the fixture-shape UNSENT ledger, so the offline canvas does not draw the within-source movement. The ledger is enforced both ways and the field is exercised by unit tests; modelling sub-source timing in the demo fixture buys a demo, not a guarantee.
• status.current (an object) sits beside status.phase.current (an integer). Renaming either is a contract change for a naming preference.

[gloryfromca]

No blockers; this can merge as far as I am concerned.

Reviewed the complete PR diff and the follow-up fix, plus the surrounding importer, RPC status lifecycle, generated clients, web row calculation, callers, and relevant history. The source-boundary fix clears current only after the state file has settled that source, so status readers see exactly one share before, during, and after the transition. Cancellation, retry/failure, empty-source, phase-transition, and stopped/restarted-gateway behavior remain coherent.

Also checked AGENTS.md/CLAUDE.md, CONTEXT-MAP.md and the Runtime/Web UI architecture terms, backward compatibility of the additive optional wire field, commit/history requirements, and that the tests were strengthened rather than weakened.

Verification:

• uv run pytest tests/test_rpc_schema_match.py tests/test_rpc_import_sync.py tests/test_importer_orchestrator.py tests/test_importer_phases.py -q - 472 passed
• npm test -- src/features/importSync/store.test.ts - 22 passed
• npx vitest run scripts/gates/fixture-shape.test.mjs - 5 passed
• npm run type-check - passed
• npm run gen:check - generated client matches the 193-method contract
• npx eslint src/features/importSync/store.ts src/features/importSync/store.test.ts - passed

[gloryfromca]

Blocking: keep batch progress visible while retrying a failed source.

The new 99-percent cap and the source-count display work for first attempts, but the retry path suppresses the data that display needs. Reviewed the revision delta, the importer/state/RPC callers and history, the web rendering path, the wire contract and backward compatibility, the project architecture/rules, and the added tests; no other blocker survived verification.

Verification:

• uv run pytest tests/test_rpc_schema_match.py tests/test_rpc_import_sync.py tests/test_importer_orchestrator.py tests/test_importer_phases.py -q - 477 passed
• npm test -- src/features/importSync/store.test.ts src/features/importSync/ImportSyncPage.test.tsx - 34 passed
• npm run type-check - passed
• npm run gen:check - generated client matches the 193-method contract
• ESLint on the five touched importSync files - passed

[gloryfromca]

Second self-review round, three fresh clean contexts (the fix itself, the row's user-visible states, test strength). Three findings taken, four named and let go. Fixed in 40d43b8.

The same double count, through the failed bucket. is_submitted is true only for "submitted", so a source that failed is sent again by the next run while its entry still says failed -- and the row counts failed as settled. During that re-send the count holds the source and current names it too. On a three-source resume the row read 94 percent, then fell back to 67 when the source landed; on this branch's base the same run is 67, 67, 100. import.status now drops a source the counts already hold.

Rounding reached 100 percent before the pass was done. 17 of 18 settled with a 3000-message source in flight reads 100 percent from sent = 2730 -- 270 messages, about nine minutes, at "Syncing data 100%". Before this branch that state read 94. The repo already carries the rule this broke, in the profile mirror's progress line (raven/importer/hermes_user_md.py): counting the call about to be made showed 3/3 while the third was still in flight, which is the 100%-then-wait this progress line exists to remove. The row now stops at 99 until every source is settled.

The number still barely moved in the case the branch exists for. One source's whole share is 1/N of the bar: a 287-message source inside a 19-source run buys 5.3 points over ten minutes, five integer steps, one per two minutes. True, but still reads as stuck. The row now carries that source's own counts beside the percentage (41% - 120/287), the way it already carries the phase's 1/3; those move every batch, about every twenty seconds, whatever N is.

Tests. The second round mutated fourteen things the first round had not; nine survived. The ones that mattered: _on_progress clearing only for submitted (so failed and skipped sources stayed named), both _CURRENT resets deletable with the suite green, the cancel exit free to report a source it never sent, a zero-message source free to report (0, 0), the web share free to run a batch ahead, and the payload never validated against the result model. All are pinned now, each verified red under its mutation: a failed source being sent again, a source given up on, a skipped source, a run a stop left half way, a crash after a source was named, a source with nothing to send, the share as an exact fraction batch by batch, the 99 cap, and the row drawing the source's counts.

Named and let go:

• The row's number drops when a run is stopped mid-source (60 percent to 0 on a single-source run). Both ends are true -- a source a stop left half sent keeps no entry and the next run sends it whole -- and pretending otherwise would misdescribe what resume does.
• The stop is not acknowledged: the x stays live and one more batch lands before the row flips to paused. The in-flight store is not interruptible, so a stopping state is its own change.
• A failed status read leaves the row on its last frame with no error shown. Pre-existing, and it has no path through RowView.
• import.status.current stays on the fixture-shape UNSENT ledger: the offline canvas would need sub-source timing modelled to draw it, which buys a demo rather than a guarantee.

One correction to the review above: retry/failure ... remain coherent does not hold -- that is the first finding here, reproduced against a real pass.

[gloryfromca]

No blockers; suggestions only, and they are marked inline.

This head rebases the reviewed import change onto the updated target without changing its import-related tree. Rechecked the full target diff, relevant callers/history, wire compatibility, tests, and project architecture/rules; no new issue survived verification.

Named follow-up: a failed source being retried is omitted from import.status.current, so that retry does not show per-batch counts. The behavior remains real but is nonblocking at this fourth round because the row still reports the run and the operator can wait for completion. The prior thread has been replied to and resolved.

Verification:

• uv run pytest tests/test_rpc_schema_match.py tests/test_rpc_import_sync.py tests/test_importer_orchestrator.py tests/test_importer_phases.py -q - 477 passed
• npm test -- src/features/importSync/store.test.ts src/features/importSync/ImportSyncPage.test.tsx scripts/gates/fixture-shape.test.mjs - 39 passed
• npm run type-check - passed
• npm run gen:check - generated client matches the 193-method contract
• ESLint on the five touched importSync files - passed

[gloryfromca]

No blockers; this can merge as far as I am concerned.

Reviewed the new retry-count delta on the actual GitHub head. It fixes the named follow-up without changing the wire shape: the in-flight source remains visible through current and is temporarily omitted from settled counts, so the row neither double-counts nor freezes. Checked the state transitions, per-platform totals, cancellation/settlement behavior, caller compatibility, project rules, and the strengthened regression test.

Verification on an isolated worktree at b4cf9751c: uv run pytest tests/test_rpc_schema_match.py tests/test_rpc_import_sync.py tests/test_importer_orchestrator.py tests/test_importer_phases.py -q - 477 passed.

The author reply was confirmed; the originating thread was already resolved and remains settled.