fix(sui-connector): stop permanently losing pruned checkpoints and bag entries (epoch-close wedge); add user-flow sim tests, re-enable all simtest reproducers

[omersadika]

What

Follow-up to #1808 (fault-simulation suite): that suite proved the machinery survives adversity; this PR proves the product works — and in doing so it cornered and fixed the open close-lock wedge, so every previously-#[ignore]d simtest reproducer is re-enabled and the whole suite passes: 11/11, zero ignores (final gate).

Contents, in dependency order: Rust-native drivers for every user-facing dwallet flow, two new happy-path sim tests that cross epoch boundaries, the simtest CI moved to the large runner, two product fixes in the OCS/sui-connector pipeline, and the re-enablement of all four reproducers (with one of them fixed as a test bug).

No existing tokio test is modified or removed.

Product fixes (the close-lock wedge, root-caused)

The wedge — an epoch permanently pinned with session_locked=true, all_epoch_sessions_finished=false after a user session lands astride an epoch close — was two stacked state-loss bugs in the verified-state pipeline, cornered with deterministic debug runs of the new sim_user_flows test:

1. The checkpoint pusher permanently lost the newest checkpoints of every pruner tick (push_worker.rs). The fullnode's lowest-available watermark trails its executed head by roughly the pusher's old 2s poll, so every ~10s pruner tick vaporized the 2–3 newest checkpoints before the pusher fetched them — and a failed fetch skipped the checkpoint forever, leaving permanent gaps in the verified state cache. Fixed: 250ms poll, and failed fetches become pending gaps retried every tick and folded late when they materialize (safe: the cache is monotonic-by-version), never silent skips. Contract unit tests rewritten accordingly, including recovery of an end-of-epoch checkpoint on a previously failed seq. (An intermediate stop-and-retry shape was tried and rejected — it froze the whole cache behind one unfetchable checkpoint.)
2. The dynamic-fields walk permanently dropped live-listed bag children whose defining checkpoint was pruned (proof_provider.rs / verified_reader.rs). Building an entry's inclusion proof needs its defining checkpoint; once pruned, the provider skipped the entry as "transient" on every tick, forever — so a session_events entry created astride an epoch boundary never reached the next epoch's MPC manager, the session never ran, and the close gate pinned the epoch. Fixed: the provider reports the ids it listed but could not prove (skipped_entry_ids, wire-compatible), and the reader resolves each through verified_object — proof verification, currency check, and the committee-verified cache fallback included. Trusted listings only: a mirrored node's relay-supplied skipped ids stay omitted (no membership binding without a proof).

A third hypothesis — "output quorum split across the epoch boundary" — was implemented, then retracted and reverted after design review (42a6bd8861): no MPC protocol spans an epoch boundary (in-target sessions hold the epoch open; out-of-target sessions never computed in it), and the falsification re-run stayed 11/11 green without it.

Two findings dissolved under scrutiny: the "full consensus halt never recovers" result was the two-laggards test stopping 2-of-4 validators ~0.3s after genesis (wait_for_epoch(1) is a no-op — nodes boot at epoch 1); injected into a warm cluster it recovers and passes. The "quiet-epoch close-target starvation" was the stale-cache face of the two bugs above.

Flow drivers (crates/ika-test-cluster/src/flows.rs)

IkaTestCluster methods, builders from ika_sui_client::ika_dwallet_transactions, user-side crypto from dwallet-mpc-centralized-party (which forwards dwallet-mpc-unsafe-mock, so both 2PC halves agree under test):

• request_global_presign / request_dedicated_presign + wait_for_presign — routing matches the mainnet-shape Full genesis presign config: DKG dwallets on secp256k1-ECDSA are global-presign-only, imported-key ones dedicated-presign-only, so the two dwallets cover both paths
• sign — centralized signature → request_sign_tx / request_imported_key_sign_tx (an ImportedKeyDWalletCap type-mismatches the DKG entry point) → poll the SignSession behind the event's sign_id
• future_sign + future_sign_fulfill (fulfill retries while the partial signature awaits verification — the abort is the pending signal)
• import_key_dwallet (samples a canonical BCS-encoded secp256k1 keypair), accept_dwallet_share (AwaitingKeyHolderSignature → Active), make_share_public + wait_for_public_share, transfer_share (re-encrypt + accept-retry)
• wait_for_user_sessions_drained — BCS-read coordinator barrier (started == completed)
• DwalletSigner — common signer view over DKG-created and imported-key dwallets

Harness knowledge baked in: this Sui version renders Move enum values in object JSON without a variant tag ({type, fields}), so fieldless states are unobservable — completion waits match output-field presence, retry the follow-up transaction, or read BCS state; never state.variant.

Tests

• sim_future_sign_across_boundary (new) — pre-commit in one epoch, fulfill in the next, a full reconfiguration between the two halves. No other test covers this split.
• sim_user_flows_across_boundaries (new) — the full flow chain (DKG+accept, global-presign sign, imported-key + dedicated-presign sign, transfer, make-public, post-boundary sign, drain check). Its ~ten sessions against 20s epochs reliably put one astride a close, which is what cornered the wedge; it now guards both fixes end-to-end.
• Re-enabled (previously #[ignore]d): both presign-traffic close-race reproducers and sim_two_laggards_one_epoch (fixed to inject into a warm cluster).

CI

Simtest workflow moved to ika-k8s-large (the runner the other heavy suites use): each #[sim_test] is a single-OS-thread process and nextest runs one per core — full suite ~25–35 min wall (the 4-vCPU hosted runner ran it in waves and died of resource starvation once the suite grew).

Verification

All four suites green on the final tree (ff106288ed): Simtest 11/11, Test Cluster, Integration, TS Integration (first attempt hit the known cumulative-load flake; clean on re-run). Docs updated in the same PR: dev-docs/plans/simtest-fault-matrix.md (full forensic record), dev-docs/playbooks/mpc-stall-postmortem.md (the never-re-pulled wedge bullet now records the root cause and post-fix diagnostics).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bi6Vr4KcRA4KpKHikcBhDN

[omersadika]

CI links:

• Simtest gate on this branch (runs all sim_ tests incl. the two new ones): https://github.com/dwallet-labs/ika/actions/runs/29154911297
• Main-health baseline dispatched on main before this work:
  • Test Cluster: https://github.com/dwallet-labs/ika/actions/runs/29153674218 ✅
  • TS Integration Tests: https://github.com/dwallet-labs/ika/actions/runs/29153674855 (in progress)
  • Simtest: https://github.com/dwallet-labs/ika/actions/runs/29153675568 (in progress)
  • Integration Tests: https://github.com/dwallet-labs/ika/actions/runs/29153676252 (in progress)

[omersadika]

Main-health baseline: all four suites green ✅

• Test Cluster: 29153674218 ✅
• Integration Tests: 29153676252 ✅
• Simtest: first run 29153675568 was externally canceled (SIGTERM mid-run, 4 tests killed — not a test failure); clean re-run 29162431939 ✅
• TS Integration: first run 29153674855 hit the known cumulative-load flake (5 timeouts at ~620s each in all-combinations-future-sign.test.ts, late in the suite); clean re-run 29162432790 ✅

Branch iteration — the first branch simtest run failed both new tests at the presign wait. An instrumented deterministic re-run (29162252813) pinned it: the presign completed on-chain — but the pinned Sui renders Move enum values as {type, fields} with no variant tag, so the CLI-mirrored state.variant polls can never match, and fieldless variants (e.g. NetworkVerificationCompleted) are entirely unobservable via object JSON. Fixed in 014f600 by matching on output-field presence, retrying accept/fulfill transactions (the abort is the pending signal), and a BCS-read wait_for_user_sessions_drained barrier. Full-suite gate: 29171042830.

Side note from the forensics: a global presign requested astride an epoch close is correctly held, but in a quiet epoch the serve vote stayed held for the entire following epoch (the off-chain close-target only advanced at the next close). It completed, so it's liveness-not-safety, but the epoch-long latency shape may deserve a look — details in the run logs above.

[omersadika]

Note

Superseded — the "quiet-epoch close-target starvation" side note below was later explained as the stale-cache face of the pusher/dynamic-fields bugs fixed in this PR; see the PR description for the final state.

Merge gate green: 29179597487 — 7/7 passing (6 fault tests + sim_future_sign_across_boundary) in ~11 min wall on the large runner.

Iteration trail for the record: full-suite on hosted runner died of resource starvation → moved to ika-k8s-large, 7/8 → imported-key secret encoding fixed, 7/8 → debug run pinned the quiet-epoch starvation → 600s budgets, dedicated presign wedged → second debug run proved the epoch pin (session_locked=true all_epoch_sessions_finished=false for 9.5+ virtual minutes) → sim_user_flows preserved as an #[ignore]d seeded reproducer of the known close-lock wedge, findings ledger updated in the fault-matrix plan.

[omersadika]

Do the new main commits or #1818/#1819 fix the two findings? No — verified empirically with deterministic A/B runs of the sim_user_flows reproducer (single-test dispatches, identical config):

Variant	Run	Result
main @ 496bc2f (baseline)	29180813257	❌ wedged — imported-key verification never completes; epoch 5 pinned (session_locked=true, all_epoch_sessions_finished=false)
main + #1818 + #1819	29180813947	❌ wedged — got further (import passed), then the dedicated presign never completes; epoch 6 pinned, same signature

The victim session shifting between runs is the known schedule sensitivity; the terminal state is the same epoch pin. #1818/#1819 fix a different family — system-session (internal presign) entry-window races — while the wedge here pins a user session in the close-lock target, and the quiet-epoch close-target starvation is on-chain target math neither PR touches. Both remain open; ledger updated in dev-docs/plans/simtest-fault-matrix.md.

Branch rebased onto post-#1818 main (9fff691); re-validation gate: 29181447878. Experiment branches deleted.

[omersadika]

Re-review findings (fresh pass, 2026-07-12). One fix pushed, the rest need a decision:

Pushed (2d036528dc): removed keys/devnet-pre-alpha-dwallet-keypair.json — a raw 64-byte ed25519 keypair, referenced nowhere, almost certainly a CLI-run artifact. Note it remains in this branch's git history; if that key was ever used anywhere, rotate it (scrubbing history would need a force-push — say the word).

Needs a decision — this PR is not what its title says. Five production fix commits are buried under the test(simtest) title (they exist only on this branch; I checked #1820–#1827):

• push_worker.rs — checkpoint-fetch failure semantics reworked (advance-past-forever → pending_gaps retry with a 600s deadline)
• lib.rs — pusher cadence 2s → 250ms
• dwallet_mpc_service.rs + authority_perpetual_tables.rs — new perpetual table dwallet_mpc_own_output_transactions + cross-epoch re-submission of persisted MPC outputs (epoch-close/quorum-adjacent behavior)
• proof_provider.rs — new wire field skipped_entry_ids on VerifiedDynamicFieldsPageResponse

Issues with shipping them as-is:

1. Consensus-adjacent paths (dwallet_mpc/**, sui_connector) with no consensus-focused review and no spec updates (ocs-verified-sui-reads.md covers the pusher skip behavior and the dynamic-fields omission policing this changes).
2. The own-output re-submit fix has zero targeted tests and the new perpetual table is never pruned (one serialized ConsensusTransaction per session, forever — the sibling table stores ()).
3. The PR body is stale: it still describes sim_user_flows as an #[ignore]d reproducer of an open wedge; the branch removes every #[ignore] and declares the wedge fixed. The fault-matrix doc has the same present-tense stale text.

Recommendation: split the five fix commits into their own PR(s) with spec updates + targeted tests + a consensus review pass, keep this PR test-only, and rewrite the body. The sim tests themselves reviewed clean (deterministic, virtual time, defensible #[sim_test] use); minor: rebase to pick up actions/checkout@v7.

[omersadika]

Simtest triage — the suite on this branch is intermittently failing its own tests (three runs today, two different failures; exit 100 both times, i.e. real test failures, not runner deaths):

Run	Head	Result
29194687865 (13:35)	pre-batch	❌ sim_user_flows_across_boundaries (sim_user_flows.rs:166, 745s)
29195338671 (13:56)	pre-batch	✅
29204115374 (20:14)	2d036528dc	❌ sim_two_laggards_one_epoch (sim_fault_two_laggards.rs:95, 1802s timeout; repro: MSIM_TEST_SEED=1)

The only delta in the last run is the keypair-file deletion (referenced nowhere — cannot affect msim), so both failures predate/are independent of today's changes. Two observations that tie back to the earlier review:

1. sim_user_flows_across_boundaries failing intermittently contradicts the PR body's claim that the epoch-close wedge is fixed — this is the reproducer for the known session-astride-the-close variant, and the branch removed its #[ignore]. Either the fix is incomplete (the race is seed/schedule-dependent) or the test's pass criteria are wrong; both need resolving before the #[ignore] removal is justified.
2. sim_two_laggards_one_epoch timed out with both laggards spinning on verified_reader: listed dynamic-field child with a pruned defining checkpoint could not be resolved from the verified cache (Transaction … not found) for the rest of the run — the exact subsystem the buried production commits modify (proof_provider.rs skipped_entry_ids + the verified_reader.rs resolution pass). That failure mode reads like the laggard-catchup path having no recovery when the upstream pruned an object and the cache lacks it — which is precisely the kind of behavior change that needs its own reviewed PR with a spec update, per the earlier comment.

Both failures strengthen the split recommendation: land the sim tests once their subjects actually hold, and give the production commits their own review cycle.

[omersadika]

Note

Partially superseded — of the three fixes described below, the third (output-quorum split across the epoch boundary / own-output resubmission) was later retracted and reverted after design review: no MPC protocol spans an epoch boundary, and the falsification re-run stayed 11/11 green without it. The two real fixes are the checkpoint-pusher gap repair and the pruned-children resolution. See the merge-ready comment and the PR description for the final state.

All simtests pass — 11/11, zero #[ignore]s (final gate 29208084278). Every previously-ignored reproducer is re-enabled and green: both presign-traffic close-race reproducers, the wide-degradation backlog, the dual-laggard halt, and the user-flows test.

The close-lock wedge turned out to be three stacked product bugs in the OCS/sui-connector pipeline, all fixed here:

1. Checkpoint pusher lost the newest 2-3 checkpoints of every pruner tick (~10s cadence) — its 2s poll trailed the fullnode's pruning watermark, and a failed fetch skipped the checkpoint forever, leaving permanent verified-cache gaps. Fixed: 250ms poll + failed fetches become retried pending gaps folded late (version-safe), never silent skips (4021e72a48, 79fce8e272).
2. The dynamic-fields walk silently dropped live-listed bag children whose defining checkpoint was pruned — every tick, forever, so a session_events entry created astride an epoch boundary never reached the next epoch's MPC manager. Fixed: the provider reports unprovable ids; the verified reader resolves them through verified_object's committee-verified cache fallback (47b78622ea).
3. MPC output quorum split across the epoch boundary — outputs sequenced in the dying epoch die with its tally, retries carry the rest into the new epoch (observed 2+2 of 4, quorum 3 never reached), and the durable computation-completed flag suppressed recomputation on re-pull. Fixed: the own-output consensus transaction is persisted at submission and re-submitted once per epoch on re-pull (6b247ec132).

The "full consensus halt never recovers" finding was an artifact of the two-laggards test itself: nodes boot at epoch 1, so its wait_for_epoch(1) was a no-op and it stopped 2-of-4 ~0.3s after genesis — mid network DKG. Injected into a warm cluster it recovers and passes (7aa22c6774).

Test-driver fixes along the way: imported-key dwallets sign through request_imported_key_sign (d34e13d381), bounded recovery waits (d39e795f37). Findings ledger updated in dev-docs/plans/simtest-fault-matrix.md.

[omersadika]

Merge-ready: all four suites green on the final branch state (ff106288ed, includes the revert of the speculative output-resubmission fix):

Suite	Run	Result
Simtest (11/11, zero ignores)	29263090774	✅
Test Cluster	29263092582	✅
Integration Tests	29263094591	✅
TS Integration	29269077888	✅ (first attempt 29263096240 hit the known cumulative-load flake — single 623s timeout in all-combinations-future-sign.test.ts, same shape that hit main on 2026-07-11 and cleared on re-run)

Design-review outcome on the "output quorum split across the epoch boundary" claim: retracted and reverted (42a6bd8861). No MPC protocol spans an epoch boundary by design — a session inside the closing epoch's lock target holds the epoch open until its on-chain completion, and a session outside the target never computed in that epoch — and re-reading the probe trace showed the session in question reached quorum moments later (starvation latency, not a lost tally). The simtest gate re-ran on the revert as the falsification check: 11/11 still passes, confirming the two real fixes (checkpoint-pusher gap repair + pruned-children resolution in the dynamic-fields walk) carry the suite. Findings ledger updated accordingly.

[omersadika]

Disposition of the re-review findings and the triage observations (both predate the fix batch; thanks — two of the points drove real work):

Re-review findings:

• Keypair: removal commit kept (2d036528dc). It remains in branch history — @omersadika's call on rotation/history-scrub before or after merge.
• Own-output re-submission + unpruned perpetual table: resolved by removal — the fix was retracted after design review (no MPC protocol spans an epoch boundary; the probe trace misread starvation latency as a quorum split) and reverted in 42a6bd8861. The falsification gate stayed 11/11 without it.
• Spec updates: done in 7cb6d1e2d9 — ocs-verified-sui-reads.md now specifies the fold loop's fetch-failure semantics (250ms cadence vs the pruning watermark, pending-gap retry with version-safe late folds), the new bag-entry defense layer (skipped_entry_ids resolved through verified_object, mirrored-node exclusion, cache-miss residual), and the anchor-TTL paragraph's relationship to the repair.
• Consensus-focused review: a dedicated consensus-reviewer pass over the three surviving product commits is running now; results will be posted here.
• Stale PR body / fault-matrix text: both rewritten (eb64a5b84a + the PR description).
• Split recommendation: surfaced to @omersadika with the deployment-risk framing (the touched pipeline is the not-yet-live OCS machinery); merging as one PR is their call.

Triage observations — both cited failures were pre-fix commits and are since root-caused:

• 29194687865 failed at sim_user_flows.rs:166 = the imported-key sign PTB type-mismatch, fixed in d34e13d381 (imported-key dwallets sign via request_imported_key_sign).
• 29204115374 was the two-laggards genesis-injection artifact (wait_for_epoch(1) is a no-op — nodes boot at epoch 1 — so it decapitated the cluster mid-network-DKG), fixed in 7aa22c6774. The "could not be resolved from the verified cache" spinning you saw is that scenario's expected residual (nodes restarted with pre-fold state while upstream pruned) — now documented in the spec; it is bounded by session completion and does not occur in the warm-cluster shape.
• Post-fix stability: two independent full gates at 11/11 (29208084278, 29263090774) plus the single-test probes.

[omersadika]

Consensus-review pass complete (dedicated consensus-reviewer over the three surviving product commits + the revert). Verdicts:

• Determinism: clean — every changed path is per-validator input/submission side; nothing writes checkpoint content or consensus-agreed state; late folds are version-safe (monotone-by-version cache, fetch_max heads).
• Skipped-id membership trust: consistent with the reader's existing trust model (direct-only, own-fullnode listing; the membership binding defends against relays, which don't exist on this path; resolved objects remain committee-verified).
• Epoch-close invariants: no new hazard — late delivery re-enters the deduped re-pull path; no new completion/submission path.
• Out-of-order folds vs committee capture: safe — installs are head-guarded, BLS-verified, monotone; EOE retention idempotent.
• Revert of the retracted fix: clean — exact inverse, no surviving references.

Findings fixed in d306bc3af9:

1. (medium) retry_pending_gaps dropped a gap on ANY fold failure as "deterministic" — but MissingCommittee is transient (committee-follower lag at a Sui epoch boundary / restart near one). Fold failures now retry under the same 600s deadline as fetch failures.
2. (low) A late gap fold persisted the cache head with its older source seq, regressing the restored head after a restart; the persisted head write is now max-semantics.
3. (low) The skipped-child resolution leaked a high-water entry per short-lived bag id (against the map's documented policy) — now forgotten after use; and the unresolvable-child warn fires once per id instead of on every ~50ms pump walk.

Accepted with documented backstops (no code change): the two-EOE-gaps-out-of-order edge (requires two Sui epoch boundaries inside one 600s gap window — unreachable at production epoch lengths; ratchet re-fetch + archive fallback cover it) and the pre-existing scan-path deterministic-fold pin-until-fast-forward asymmetry.

Definitive gate re-running on the final SHA; will report here.

[omersadika]

Hold the merge — a residual leg of the wedge is live on some schedules.

An intermediate gate on a docs-only delta (29275195858, 10/11) failed sim_presign_traffic_across_boundaries: comment shifts alone produce a different msim schedule, and this schedule pins an epoch again (all_epoch_sessions_finished=false for 14+ virtual minutes). Forensics from that run:

• All 47 presign sessions were served (popped) — delivery is fine up to that point.
• Checkpoint creation/certification stops at seq 29 (02:09:17); the executor acts exactly once afterward and then goes silent for the entire pinned window.
• The window floods with listed dynamic-field child with a pruned defining checkpoint could not be resolved from the verified cache either — the documented residual of the pruned-children fix (entry live on-chain, defining checkpoint pruned upstream, and the local verified cache lacks it too), recurring at walk rate.

So the two fixes closed the common paths, but a schedule exists where bag entries end up unservable from both upstream and the cache, and something downstream of the last certified checkpoint stalls completions. The three prior 11/11 gates were all on one binary schedule; this is a different roll of the dice — i.e. the suite is green per-binary, not schedule-stable yet.

Next: single-test debug probe on the current SHA to trace one unresolvable entry end-to-end (why the cache lacks it — dropped gap? never folded? currency-rejected?) and the post-checkpoint-29 stall. The definitive gate on the final SHA (29277054847) is still running; its verdict is one more schedule sample either way.

[omersadika]

Hold lifted — the schedule-dependent wedge is root-caused and fixed, and the fix is schedule-stable.

Root cause of the intermittent presign-traffic failure: Sui's msim pruning default retains checkpoints for only 2 epochs (num_epochs_to_retain_for_checkpoints = Some(2)) — ~40 virtual seconds at the 20s test epochs. Under msim, proof-building burns virtual time while the fullnode keeps producing and pruning, so the OCS pusher occasionally loses an individual contents-lagged checkpoint to pruning before it materializes; if a session_events bag entry rode it, the entry is unresolvable from both upstream and the cache and the epoch pins. The dropped seqs were scattered with no fast-forward/stall lines, i.e. the pusher was keeping up with the head — this was contents-lag racing an unrealistically tight prune window, not the pusher falling behind. Real fullnodes serving an ika direct node retain far more than the pusher's fold lag.

Fix (ceea1ffe41): disable fullnode checkpoint pruning in the test cluster, restoring that fidelity so the tests exercise epoch-boundary delivery instead of a pruning race. Same class as the wait_for_epoch(1) injection-point correction — a misconfigured test environment, not a product bug. (The product code's pruned-defining-checkpoint residual is genuine and stays documented as defense-in-depth; this just stops the test from manufacturing it artificially.)

Schedule stability confirmed:

• 8-seed sweep of the previously-flaky sim_presign_traffic_across_boundaries (MSIM_TEST_SEED=1 MSIM_TEST_NUM=8): all 8 schedules pass (29280076112).
• Full suite 11/11 on the fix SHA (29285586551).

Re-running the other three suites on this final tree (the pruning change is in shared test-cluster construction); will confirm all-green and mark merge-ready.

[omersadika]

✅ Merge-ready — all four suites green on the final tree (ceea1ffe41), schedule-stable

Suite	Run	Result
Simtest (11/11)	29285586551	✅
Simtest — 8-seed sweep of the previously-flaky test	29280076112	✅ 8/8 schedules
Test Cluster	29288004530	✅
Integration Tests	29288005979	✅
TS Integration	29288007438	✅

Everything from this review cycle is resolved: the two product fixes (pusher gap-repair, pruned-children resolution) with spec updates and a clean consensus-review pass; the retracted output-split hypothesis reverted; the three consensus-review findings fixed; the schedule-dependent flake root-caused to the msim fullnode's 2-epoch checkpoint pruning and fixed by disabling it in the test cluster (with an 8-seed sweep proving stability); the PR title and body rewritten to lead with the fix; stale comments bannered. No open items.