fix(*): smaller import batches, retried with backoff, and a dismissable failed finish

[gloryfromca]

Summary

Three follow-ups from running the web cold-start import against a real EverOS with the maintainer's full ~/.claude (19 memory-file sources, about 1,150 distinct messages).

• Import batches drop from 50 to 10 messages. With a slower extraction model (qwen/qwen3.8-flash via OpenRouter) a 50-message batch took 2.4 to 7.4 minutes and six of seven memory-file sources died on the six-minute bulk budget; the same batches took about 75 s on claude-sonnet-4-5. Ten is the maintainer's call: a batch that finishes well inside the budget on any model matters more than the fixed cost of about 7 s that every add carries on the EverOS side (which is also why it is not one message per add). Under the slow model the worst measured per-message cost puts a batch of ten at about a quarter of the budget. The 30k-character bound is unchanged.
• A refused batch is retried with backoff before its source fails. A batch the memory service refused failed its source at once, and the next source started at once; against the real service one two-minute rate-limit window at the extraction provider (OpenRouter 429, which EverOS retries only twice sub-second) took five sources down in 70 s, each one's first batch running into the same wall, and a parse failure on one model answer or one slow answer past the budget cost a source each the same way. A refused batch is now sent again after 30 s, 60 s and 120 s before the source counts as failed, whether the backend returned False or raised; the wait polls the stop file every second so a stop lands inside it.
• A finished import with failures can be dismissed. The rail row showed the failure count and a retry but no close, so a run whose failures did not clear stayed in the rail indefinitely. The close is now offered on any finish; it hides that run by signature, and the failed entries stay in the state file so a retry from the wizard or the CLI still picks up exactly those.

The batch size takes part in the EverOS message id, so a run imported under the 50-message batch is not deduplicated against a re-import under 10. No installation has completed a real-size import under either limit, so nothing on disk depends on the old id.

Type

• Fix
• Feature
• Docs
• CI / tooling
• Refactor
• Other

Verification

• uv run pytest tests/test_importer_orchestrator.py tests/integration/test_import_e2e.py tests/test_everos_backend.py tests/test_rpc_import_sync.py tests/test_cli_import_commands.py tests/test_importer_phases.py tests/test_importer_state.py -q: 313 passed. The batching tests now pin 120 messages to twelve batches of 10 and 160 messages to sixteen, is_final only on the last. New retry tests: a batch refused twice lands on the third send with the recorded waits, one refused every time fails after the last wait with after 4 attempts in the error, a raised store error is retried the same way, a stop during a wait ends the run with the source unmarked, and _pause returns as soon as the stop file appears (286 passed over the importer / RPC / CLI / EverOS backend suites).

• uv run ruff check and ruff format --check on the touched files: clean.

• ui-web npm test: 2550 passed across 190 files; npm run type-check clean; npm run lint 0 errors (5 pre-existing warnings).

• Reverse checks: with _BATCH_MSG_LIMIT set back to 50 the batching tests fail; with the retry loop reduced to a single send the retry tests fail; with the stop check removed from the wait the stop test fails; with the close restored to clean finishes only, the row test "offers a retry, and a dismiss that takes the row down" fails.

• Real host: the 50-message failure mode was measured on a live gateway (isolated RAVEN_HOME, throwaway EverOS on :18893, the maintainer's real ~/.claude): 6 of 7 sources timed out at exactly 360 s under qwen3.8-flash. Reruns on the same host at batch 10 without the retry: one 429 window took five sources, one 10-message batch still ran past the budget. A rerun with the retry is in progress and will be reported on this PR.

• Relevant tests pass locally

• Relevant lint / type checks pass locally

• User-facing docs or screenshots are updated when needed

Risk

User-visible: imports make five times as many EverOS calls, each smaller; a source now checkpoints, and a stop lands, every 10 messages instead of 50. A refused batch costs up to 210 s of waiting before its source is given up on. The rail's finished-with-failures row gains a close button.

The state file format and the RPC contract are unchanged. Rollback: revert the squash commit; no data migration is involved.

• Security impact considered (no new inputs, credentials or endpoints)
• Backward compatibility considered
• Rollback path is clear for risky changes

Related Issues

N/A

[gloryfromca]

No blockers; this can merge as far as I am concerned.

No findings survived refutation. The ten-message limit stays inside the existing count/character batching contract and preserves final-batch signaling; the failed-run close only records the finished status signature, while failed entries remain available to the existing wizard/CLI retry path. The title was also updated to say ten before this review was posted.

Coverage included AGENTS.md, CLAUDE.md, CONTEXT-MAP.md, and the Runtime/Web UI context; the full diff and the new-head delta; importer state, RPC, EverOS backend callers, and importSync store/source behavior; relevant history and blame; backward compatibility and message-id implications; test changes for weakening; and the documented architecture boundaries. No boundary or compatibility regression was found.

Verification:

• uv run pytest tests/test_importer_*.py tests/integration/test_import_e2e.py -q: 272 passed on the current head.
• npm test --prefix ui-web: 190 files, 2550 tests passed; the web tree is unchanged by the final head commit.
• Targeted web tests: 2 files, 28 tests passed.
• npm run type-check --prefix ui-web: passed.
• npm run lint --prefix ui-web -- src/features/importSync/ImportSyncPage.tsx src/features/importSync/ImportSyncPage.test.tsx: 0 errors, 5 pre-existing warnings in unrelated files.
• Ruff, source-language, commit-message, and git diff --check checks: passed.

[gloryfromca]

No blockers; this can merge as far as I am concerned.

The new retry/backoff delta is consistent with the memory-store contract and the repository's existing store-pipeline retry semantics. Explicit import metadata keeps is_final stable across replay, an exhausted retry still leaves the source failed and resumable, and a stop during backoff leaves it unmarked. No concrete failure survived refutation.

Coverage for this revision included the repository rules and Runtime/Web UI contexts; the full target diff and the delta from e44e8d600627; importer, RPC, CLI, state, EverOS backend, and store-pipeline callers and history; backward compatibility and replay behavior; tests for weakening; and architecture boundaries. The new tests add transient-success, exhaustion, raised-error, and cancellation cases while preserving the existing batching and error-isolation coverage.

Verification:

• uv run --extra dev pytest tests/test_importer_orchestrator.py tests/integration/test_import_e2e.py tests/test_everos_backend.py tests/test_rpc_import_sync.py tests/test_cli_import_commands.py tests/test_importer_phases.py tests/test_importer_state.py -q: 318 passed.
• The first run without --extra dev stopped at collection after 147 passes because the optional raven_everos package was absent; the declared dev environment above resolved that setup issue.
• Ruff check and format check on the touched Python files: passed.
• Source-language, commit-message, and git diff --check checks: passed.