feat: serialize backfill discovery writes to avoid optimistic concurrency races

[rouzwelt]

Backfill passes now serialize writes to the vault's inventory aggregate, eliminating optimistic-concurrency races that could stop the service at startup. Restarts after rollbacks find every returned receipt in one pass, and concurrent writes to the same aggregate were losing the race on every retry. Reads still run concurrently; only the aggregate writes run sequentially. Fixes RAI-2835. This is the top of a 10-PR stack.

Closes RAI-2835

Contributes to RAI-2601
Contributes to RAI-1215

Live effect: Service restarts after rollbacks succeed instead of failing on optimistic-concurrency conflicts. · Risk: Low (changes backfill write order, no money or keys). · Ships: On merge. · Blocks: None.

Decisions

• Serialize writes in collection order, not batched or with retry backoff, to eliminate aggregate contention while keeping concurrent balance reads.

Risks

• Sequential writes add latency to large passes, but balance reads (the network bottleneck) remain concurrent and writes are fast in-process.

Proof

• New test with 65 discoveries completes without any optimistic-concurrency conflicts.

Rollout

1. Deploy and watch startup logs after next rollback for absence of "optimistic-concurrency conflict" errors.
2. Rollback: Revert to previous version; large passes may fail and require manual restarts until fixed.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[linear-code]

RAI-2835

RAI-1215

RAI-2601

[rouzwelt]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: fork rehearsal of RKLB orchestrator cutover on a live Base fork #441
• feat: serialize backfill discovery writes to avoid optimistic concurrency races #440 👈 (View in Graphite)
• feat: pin receipt balance reads to pass head block #439
• feat: auto-rebroadcast retained burn when still mineable before refusing resume #438
• feat: recover exhausted orchestrator mints via own transaction classification #436
• feat: alert on orchestrator mints overdue for recipient authorization #424
• docs: expand orchestrator onboarding runbook for multi-chain cutover scope, release tag hygiene, and stuck-mint escalation #423
• feat: guard against duplicate (recipient, nonce) pairs across open mints #421
• feat: recover mined orchestrator mint from MintingFailed via its own tx receipt, never vault-direct confirm #427
• feat: orchestrator cutover leaves custody at bot wallet; rollback rediscovers receipts without confirm-custody #426
• feat: guard against double-burn when tx-free failure races an in-flight burn #419 : 1 other dependent PR (#420 )
• main

---

How to use the Graphite Merge Queue

Add the label add-to-gt-merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[rouzwelt]

@rain-marvin review

[rain-marvin]

🔎 Reviewing 2fc2f90, started by @rouzwelt. The review will appear here when it's done.

[rain-marvin]

This PR splits the receipt backfill pass into two steps. It still reads the receipt balances in parallel, at most 4 at a time and pinned to the read block. Then it writes the discoveries to the vault's single inventory aggregate one at a time, in the order the logs were collected. Before, up to 4 writers loaded and persisted the same aggregate at once. A big pass, such as the restart after a rollback, could use up the 3-attempt conflict retry budget and stop startup. The PR also adds a 65-receipt regression test and a SPEC note.

Overall read: the fix is correct and the right size. During run_startup_recovery the backfill is the only writer to the aggregate, because the live monitors, periodic backfills and redemption workers start after it returns. So serial writes remove the conflict at its source. buffered keeps the collection order, so unique_by and the ITN duplicate-deposit check now treat the earliest deposit as canonical every time. The reconciliation loop was already serial. The open thread about try_collect dropping finished reads is not a defect: no durable progress is lost, the checkpoint does not move, and the next pass redoes the same idempotent range. No new findings.

claude-opus-5-5 · high · 12 min

[rouzwelt]

@rain-marvin review

[rain-marvin]

🔎 Reviewing ed56f2f, started by @rouzwelt. The review will appear here when it's done.

[rain-marvin]

This PR splits receipt backfill discovery into two steps: the pinned balanceOf reads still run concurrently (now buffered so they keep log order, with try_collect), and the writes to the vault's single inventory aggregate (ITN duplicate check, DiscoverReceipt, ReconcileBalance, mint-recovery callback) now run one at a time. Before this, four concurrent writers on one aggregate could use up the three-attempt optimistic-concurrency retry budget, and a large pass (the restart after a rollback) would fail and stop startup.

Overall read: the change does what it says and is about as small as it can be while keeping reads concurrent. The new 65-receipt test was run against the base code and fails there with AggregateConflict after the retries run out; on this head it passes repeatedly. At startup the backfill is the only writer to the aggregate, because the redemption and mint workers start after run_startup_recovery, so the startup failure cannot recur from inside the pass. Sequential writes also make the ITN duplicate-deposit check deterministic when two deposits for the same request land in one pass. Checkpoint behavior is unchanged: a failed read or write still leaves the checkpoint where it was. The two earlier try_collect threads were already settled and nothing in the code changes that. No findings.

claude-opus-5-5 · high · 24 min

[rouzwelt]

@CodeRabbit review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[graphite-app]

Merge activity

• Oct 6, 4:11 PM UTC: rouzwelt added this pull request to the Graphite merge queue.
• Oct 6, 4:12 PM UTC: The Graphite merge queue removed this pull request due to downstack failures on PR #419.
• Oct 6, 4:12 PM UTC: The Graphite merge queue removed this pull request due to downstack failures on PR #419.
• Oct 6, 11:43 PM UTC: rouzwelt added this pull request to the Graphite merge queue.
• Oct 6, 11:44 PM UTC: CI is running for this pull request on a draft pull request (#455) due to your merge queue CI optimization settings.
• Oct 6, 11:44 PM UTC: Merged by the Graphite merge queue via draft PR: #455.