Repository navigation
feat: persist and recover committor intent execution - #1368
Merged
Merged
Conversation
# Conflicts: # magicblock-accounts/src/scheduled_commits_processor.rs # magicblock-api/src/magic_validator.rs
# Conflicts: # magicblock-accounts/src/scheduled_commits_processor.rs # magicblock-api/src/magic_validator.rs # magicblock-committor-service/src/committor_processor.rs # magicblock-committor-service/src/service.rs # test-integration/test-committor-service/tests/test_ix_commit_local.rs
# Conflicts: # magicblock-api/src/magic_validator.rs # magicblock-committor-service/src/committor_processor.rs
# Conflicts: # .agents/context/crates/magicblock-committor-service.md # .agents/specs/validator-specification.md # Cargo.lock # Cargo.toml # magicblock-committor-service/Cargo.toml # magicblock-magic-program-api/src/instruction.rs # test-integration/Cargo.lock
# Conflicts: # Cargo.toml # magicblock-magic-program-api/src/instruction.rs # test-integration/Cargo.lock
snawaz
force-pushed
the
feat/mimd-0025/main
branch
from
September 19, 2026 19:05
fcfe18f to
2539b16
Compare
Restore ScheduledCommitSent to its original enum position and tuple payload so precomputed validator instructions keep their serialized shape. Append the new outbox instructions after existing variants to keep CloseMagicAta's discriminant stable, while continuing to ignore the ScheduledCommitSent uniqueness bump during processing.
Duplicate submission can surface as AlreadyProcessed when the exact signed transaction was accepted before confirmation or recovery observes it. Do not treat that response as success by itself; resolve the original signature status first. - Check RPC signature status when accept, set-stage, or close resubmission returns AlreadyProcessed. - Apply the same signature-status check for engine-submitted ScheduledCommitSent notifications. - Return failed or missing signature status instead of assuming the duplicate submission succeeded. - Mirror the behavior in the schedule-intent test outbox client. - Add focused coverage for the RPC response shape seen in CI. Validation: - make fmt - cargo check -p magicblock-committor-service --tests - cargo check --manifest-path test-integration/Cargo.toml -p test-schedule-intent --tests - cargo test -p magicblock-committor-service detects_already_processed_rpc_response_message -- --nocapture - git diff --check
snawaz
force-pushed
the
feat/mimd-0025/main
branch
from
September 22, 2026 09:10
f300e81 to
4015274
Compare
The schedule-intent recovery tests need to put intents into specific durable outbox states and then manually drive executor recovery. The previous test helper did this by reading the next intent id, scheduling in one transaction, and then racing the validator's accept loop with a second accept transaction. That approach was flaky because AcceptScheduledCommits pops the FIFO front of the scheduled queue, not a specific requested id. If the validator accept loop consumed the front intent first, the test could fail to fetch the intended outbox PDA or observe a state that the real validator had already started executing. This commit makes recovery and the tests deterministic: - Make ScheduledCommitSent terminal by logging the execution outcome and closing the outbox PDA atomically. - Report only terminal execution outcomes to outbox; retryable lifecycle failures leave the outbox record open for recovery. - Persist selected commit nonces in outbox stage updates so restart recovery can reject stale intents after delegation-session or nonce changes. - Reconcile current outbox state before executor retries and before resuming stage execution, so timeouts after durable stage persistence do not rebuild from stale Accepted state. - Preserve the uniqueness nonce for finalize-only recovery paths. - Replace the test-only steal helper with an atomic schedule+accept transaction, keeping the real schedule and accept program instructions while removing the race window. A cfg/test-harness switch to disable the validator accept loop for these white-box recovery tests was considered but not implemented. The atomic transaction approach is smaller, keeps the validator service wiring unchanged, and gives these tests the deterministic accepted outbox bundle they need without adding a separate test-only validator mode.
Clarify that ScheduledIntentBundle carries the durable MagicContext intent identifier, not a per-account DLP commit nonce. - Rename ScheduledIntentBundle.id to intent_id and document the distinction from DLP commit nonces. - Update outbox acceptance, recovery, scheduling, result routing, and stage execution paths to use the renamed field. - Keep serialized field order and instruction ABI unchanged; this is a naming cleanup, not a behavior change. - Update affected program and integration tests to use intent_id explicitly.
Bring the bottom MIMD stack branch up to the current dev tip while preserving the existing merge-based stack history described in edwin-stack-merging.md. - Pull in dev's Engine 0.11.1 and Hydra scheduler changes. - Preserve this branch's outbox intent work, including deterministic outbox PDA acceptance and the renamed intent_id field. - Resolve Cargo.toml by keeping the outbox reader/client accountsdb dependency and moving it to the same Engine 0.11.1 tag as the rest of dev. - Resolve instruction_utils.rs by keeping the outbox-specific helpers from this branch while accepting dev's removal of the obsolete ExecuteCrank builder; crank execution is now handled by Hydra. - Leave the merge ready for the rest of the stack to be updated bottom-up: first merge this branch into outbox-intent-program, then merge that into block-failed-pubkeys.
Point the RedSuite CI workflow at the temporary fix-commits branch so this PR can validate the updated commit_exactly_once scenario before a new RedSuite release/tag is cut. The branch contains the RedSuite fix that stops expecting commit recovery to advance the DLP nonce twice after an already-landed base transaction. Validation: - git diff --check -- .github/workflows/ci-redsuite.yml
# Conflicts: # Cargo.toml
- Allow base-chain undelegation completion to replace a local Transient image without regressing the lifecycle slot. - Refresh undelegation-tracked accounts from RPC so missed pubsub updates during reconnect can still complete undelegation. - Pre-track CommitFinalizeAndUndelegate accounts before scheduling so the refresh path observes them. - Add focused coverage for transient replacement, targeted refresh, and commit-finalize undelegation tracking.
snawaz
force-pushed
the
feat/mimd-0025/main
branch
from
September 26, 2026 20:08
625cdcc to
fe0ed1c
Compare
The RedSuite undelegation_reconnect_gap scenario exposed a validator-side slot-ordering bug. In that path, ER can hold an undelegating account at a high local transient slot while base completes undelegation at a lower base-chain slot. The existing completion path intentionally raised the completed base image to the local transient slot so Engine would accept the lifecycle transition. That solved completion, but it lost the source/base slot boundary. After that handoff, a later base update can have a newer base slot while still being lower than the synthetic local slot. The normal subscription update ordering check then treats the update as stale and drops it. In CI this left ER stuck at the completed value 700 and it never accepted the follow-up base value 900, so RedSuite waited 100s and failed with observed [Some(700), Some(700)]. Fix this by tracking completed undelegation slot handoffs per account: - record the real base slot and synthetic local slot only after the completion clone is accepted - reject replayed or non-advancing base-source slots for that account - assign monotonic local slots to newer base updates while base slots are still behind the ER-local slot - drop the handoff once base slots catch up, and clear it if the account becomes delegated again - keep the handoff map bounded with an LRU Add a focused regression test that seeds a transient account at local slot 100, completes undelegation from base slot 10, then verifies a follow-up base slot 11 update is accepted by materializing it at local slot 101. Checks run: - make fmt - cargo check -p magicblock-chainlink --tests - cargo test -p magicblock-chainlink completed_undelegation_accepts_follow_up_lower_slot_base_update -- --nocapture
The RedSuite undelegation_reconnect_gap failure was exposing a broader Chainlink freshness bug, not another undelegation-only edge case. When a WebSocket reconnect dropped account subscriptions, the bulk replay path could make partial progress and still return success, leaving the rest to the 60s periodic reconciler. That allowed ordinary readonly accounts to keep stale local state across the reconnect gap, so a later base update could be missed and local reads could continue returning the old value. This commit tightens that behavior: - Make WebSocket bulk resubscription return AccountSubscriptionsOutOfSync when any requested account remains missing after replay. - Have SubMux capture the exact tracked accounts that lose all live pubsub coverage when a client disconnects. - Feed those reconnect-gap candidates into RemoteAccountProvider immediately, before the periodic reconciler would run. - Invalidate ordinary tracked accounts that had no live coverage during the gap so they are evicted and recloned instead of trusted as fresh. - Preserve undelegation-tracked accounts, because they already have the dedicated RPC catch-up path needed during undelegation. - Add focused coverage for direct-account invalidation and undelegation-tracking retention. Validation: - make fmt - cargo check -p magicblock-chainlink --tests - cargo test -p magicblock-chainlink test_reconnect_gap_invalidates_uncovered_direct_subscription -- --nocapture - cargo build -p magicblock-validator -p magicblock-verifier - redsuite run redshift/undelegation_reconnect_gap --profile lite --serial
RedSuite issue magicblock-labs/redsuite#119 and RedSuite PR magicblock-labs/redsuite#153 added the new redhat/ephemeral_accounts scenario. That scenario intentionally exercises magic ephemeral create, resize, and close instructions in transactions where a later instruction fails. The important thing it checks is that the validator reports both the final account state and the transaction metadata as if the earlier successful instruction was rolled back. The validator already rolled the account state back correctly. The failing CI evidence was narrower: getTransaction.meta.postBalances still contained the transient balances produced by the earlier ephemeral instruction in the failed transaction. RedSuite saw the live account state match the expected rolled-back state, then failed only when comparing self.info.post_balances for the create: rollback, resize: rollback, and close: rollback cases. This happens because the Engine-provided transaction details can carry post balances collected from the executed loaded accounts before the failed transaction has been represented as fully rolled back in RPC metadata. Aperture already had a compatibility normalizer for failed transaction balances, but it only repaired missing or length-mismatched postBalances. The RedSuite failures had arrays with the correct length, just stale contents, so the old normalizer returned early and left the bad metadata visible. Fix the validator side at the Aperture metadata boundary. When building TransactionStatusMeta for a failed transaction with non-empty preBalances, derive postBalances from preBalances and subtract the transaction fee from the payer with saturating subtraction. Successful transactions keep their Engine postBalances unchanged, and failed transactions without preBalances keep the original postBalances because there is no reliable base to repair from. Apply the same rule to both retained Engine transactions and live processed transactions, so getTransaction, getBlock transaction details, and Geyser delivery agree on failed transaction balance semantics. Also strengthen the JSON fallback normalizer used by getTransaction so ledger-backed responses overwrite same-length stale postBalances instead of only filling missing arrays. No RedSuite-side change is needed for this step. The test was useful here precisely because it separately verified live state before asserting the RPC metadata, which showed this was a receipt/reporting bug rather than an execution rollback bug. Validation performed locally: - make fmt - cargo check -p magicblock-aperture --tests - cargo test -p magicblock-aperture post_balances -- --nocapture - cargo build -p magicblock-validator -p magicblock-verifier - redsuite run redhat/ephemeral_accounts --profile lite --serial
RedSuite's undelegation_reconnect_gap scenario can drive the validator through a short window where the pubsub mux still has configured clients, but none of them are currently marked connected. During that window companion-account discovery asks Chainlink to subscribe to the delegation record account. Before this change, SubMux took an immediate connected-client snapshot and passed it straight into AccountSubscriptionTask. If the snapshot was empty, the task failed with "No clients provided for Subscribe". That made the fetch fail opportunistically even though the reconnector could make the same client usable milliseconds later. In CI this showed up as ER never discovering the completed undelegation within the scenario timeout. The fix keeps the existing safety behavior but adds a small reconnect-aware wait at the SubMux boundary. Account and program subscriptions now poll briefly when all configured clients are temporarily disconnected, then fan out through the existing AccountSubscriptionTask once a connected client appears. If no client comes back, the code still falls through to the same failure path instead of pretending coverage exists. This belongs in SubMux rather than the higher-level fetch path because SubMux owns the difference between "there are no clients" and "all configured clients are between disconnect and reconnect". Callers can continue requesting subscriptions normally, while the mux preserves the existing confirmation and rollback behavior. Add a regression test that forces the connected-client snapshot to zero, starts a subscription while reconnect is disabled, reenables reconnect shortly afterward, and verifies that the subscription succeeds after the client becomes visible again.
snawaz
approved these changes
Sep 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Record accepted intents and their execution stages in outbox accounts so a restart doesn't mean starting over. The committor can resume pending work using the transaction signatures and blockhashes retained by stage-specific executors.
This replaces the committor persister with outbox acceptance, stage updates, recovery scans, and cleanup of completed intents, following MIMD-0025 and MIMD-0027.
Closes #1702
Impact
The persistence format changes. Before rollout, we still need to decide how to recover work stored by the old persister. Shutdown coordination is tracked separately in #1262.
Reviewer notes
A transaction may have succeeded even if confirmation timed out. Check how stage persistence is ordered relative to submission, and how recovery checks the previous signature before replacing an attempt.
If commit succeeded but finalization didn't finish, the retry must keep the confirmed commit signature. Stage transitions must remain valid, and completed outboxes must be cleaned up so recovery doesn't pick them up again.