feat: simplify transaction scheduler, make it best effort

[bmuddha]

Summary

This PR removes the complexity associated with fairness enforcement, which led to subtle bugs (deadlocks) under synthetic high loads. This is achieved by making the locks best effort attempts, not dealing with any future contention. I.e. if the lock on account fails, the transaction is simply requeued and retried when the executor becomes available. This simplification makes deadlocks impossible by design and in practice doesn't lead to writer/reader starvation even under unrealistic loads.

Compatibility

• No breaking changes, but the scheduling logic is now different

Testing

• tests have been rewritten to accomodate the new scheduling logic

Checklist

• docs updated README
• related to Feat: multi threaded scheduler #589

Summary by CodeRabbit

• Documentation

  • README reorganized into Architecture/Core Components with a diagram and clearer ExecutionCoordinator and scheduling workflow.

• Changes

  • Scheduler redesigned with coordinator-driven lock/queue protocol, improved rescheduling and executor readiness prioritization; strict FIFO ordering no longer guaranteed but transaction correctness preserved and scheduler robustness improved.

• Tests

  • Tests updated to expect unordered completion and cover new lock/queue and rescheduling semantics.

[github-actions]

Manual Deploy Available

You can trigger a manual deploy of this PR branch to testnet:

Deploy to Testnet 🚀

Alternative: Comment /deploy on this PR to trigger deployment directly.

⚠️ Note: Manual deploy requires authorization. Only authorized users can trigger deployments.

Comment updated automatically when the PR is synchronized.

[coderabbitai]

📝 Walkthrough

Walkthrough

Reworks the scheduler and README toward an ExecutionCoordinator-driven architecture. Replaces per-executor FIFO VecDeque queues with per-executor BinaryHeaps of TransactionWithId (min-heap by ID). Consolidates contention tracking into per-executor acquired_locks and a centralized LocksCache; removes Rc indirection by storing AccountLock directly and introducing a WRITE_BIT mask. Adds atomic transaction ID generation via next_transaction_id(). Lock acquisition is all-or-nothing and returns an ExecutorId blocker on conflict; blocked transactions are enqueued in heaps. Tests and scheduling verification updated to reflect relaxed FIFO guarantees and the new APIs.

Suggested reviewers

• GabrielePicco
• thlorenz
• Dodecahedr0x

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch bmuddha/fix/simple-scheduler

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Fix all issues with AI agents

In `@magicblock-processor/README.md`:
- Around line 13-32: The fenced ASCII diagram block is missing a language
specifier; update the opening fence from ``` to ```text (i.e., add the "text"
specifier for the ASCII diagram block) so the diagram renders and lints
correctly—locate the ASCII diagram block in the README (the block containing
"TransactionScheduler" / "ExecutionCoordinator" / "Executor 0") and change its
opening fence accordingly.

In `@magicblock-processor/src/scheduler/mod.rs`:
- Around line 190-193: The SAFETY comment above the unsafe impl Send for
TransactionScheduler incorrectly mentions Rc<RefCell>; replace it with a brief,
accurate invariant: state that TransactionScheduler is safe to mark Send because
it is designed to run on a single dedicated thread and none of its non-Send
internal state escapes that thread or is accessed from other threads; reference
the unsafe impl Send for TransactionScheduler and ensure the comment documents
the single-threaded architectural guarantee and the requirement that no non-Send
handles are moved across threads.
- Around line 160-165: The call program_cache.write().unwrap() in
transition_to_new_slot risks panics on poisoned locks; replace the unwrap with a
poison-aware recovery (e.g., use .write().unwrap_or_else(|poison|
poison.into_inner()) or map_err to recover the guard) and optionally log the
poisoning event before continuing, then proceed to call cache.prune(...) and set
cache.latest_root_slot as before; ensure the change touches the
transition_to_new_slot function and the program_cache write acquisition to
remove the direct unwrap and handle PoisonError appropriately.

[coderabbitai]

Actionable comments posted: 3

🤖 Fix all issues with AI agents

In `@magicblock-processor/src/scheduler/coordinator.rs`:
- Around line 99-122: Reduce visibility of scheduler lock-management methods
from pub(crate) to pub(super): change the signature of try_acquire_locks and the
unlock_accounts function to use pub(super) instead of pub(crate) so they remain
accessible to the parent module but not the entire crate; update both function
declarations (try_acquire_locks and unlock_accounts) accordingly and run tests
to ensure no external references rely on the wider visibility.

In `@magicblock-processor/src/scheduler/mod.rs`:
- Around line 83-91: The spawn function calls
Builder::new_current_thread().build().expect(...), which must not use .expect()
in production; change it to handle the error explicitly: call build() and match
or use .unwrap_or_else to log the build error via the project's logger (e.g.,
tracing::error! or processLogger equivalent) including the error details, then
panic with a clear message if you still consider it unrecoverable. Update the
code around the spawn function / runtime variable so the runtime build error is
logged before aborting (or propagate/return an error from spawn if you prefer),
and ensure the callsite behavior remains consistent with run()'s expectations.

In `@magicblock-processor/src/scheduler/tests.rs`:
- Around line 250-261: The existing comment in the test
transaction_with_duplicate_accounts is misleading because
Transaction::new_signed_with_payer compiles the message and deduplicates
account_keys, so there is only a single lock attempt; update the comment above
the test (referencing transaction_with_duplicate_accounts, mock_txn, and
try_schedule on ExecutionCoordinator) to state that Solana message compilation
deduplicates account keys and that the test simply verifies the scheduler does
not panic when given duplicate inputs rather than asserting a second lock is a
no-op.

[taco-paco]

LGTM! The only thing - I would keep some of the comments as they're useful,

[coderabbitai]

Actionable comments posted: 4

🤖 Fix all issues with AI agents

In `@magicblock-processor/src/scheduler/coordinator.rs`:
- Around line 36-57: The LocksCache currently grows unbounded because entries
created via or_default() in try_acquire_locks are never removed; add a pruning
step in ExecutionCoordinator that removes LocksCache entries whose AccountLock
state is zero (no active locks) — e.g., implement a new private method on
ExecutionCoordinator (prune_unused_locks or clear_account_contention) that
iterates LocksCache and removes entries with zero state, and invoke this method
at appropriate transition points (slot transition / after releasing locks in the
code paths that use acquired_locks) so entries are cleaned up periodically;
update try_acquire_locks/release paths to call the new method and reference
ExecutionCoordinator, LocksCache, try_acquire_locks, acquired_locks, and the new
prune method name when locating changes.

In `@magicblock-processor/src/scheduler/locks.rs`:
- Around line 52-63: The comment about trailing_zeros in write() is imprecise;
update the write() implementation to both (a) document the invariant that when
write() runs the low set bit must be an executor bit (i.e. state always includes
WRITE_BIT | (1 << executor)), and (b) enforce it with a debug assertion before
calling trailing_zeros (e.g. assert that state & !WRITE_BIT != 0) so
trailing_zeros() cannot return the sentinel index for WRITE_BIT-only state;
refer to the write() function, the state field, WRITE_BIT, and trailing_zeros()
when making the change.

In `@magicblock-processor/src/scheduler/mod.rs`:
- Around line 145-150: The use of expect() in handle_new_transaction is
disallowed; replace the panic-prone call to
self.coordinator.get_ready_executor().expect(...) with a defensive match or if
let that handles the None case: call get_ready_executor(), if Some(executor)
then proceed to call self.schedule_transaction(executor,
TransactionWithId::new(txn)); if None then log an error/warning (or return
early) including context (e.g., mention handle_new_transaction and txn id) to
avoid panic; keep the is_ready() logic as a comment if desired but do not rely
on it to justify expect().
- Around line 183-187: The code logs try_send failures but drops the txn while
leaving account locks held; modify the closure handling self.executors[executor
as usize].try_send(txn) to explicitly release/unlock the accounts that were
locked by try_schedule (the same locks associated with txn and executor) when
try_send returns Err (channel full/closed). Locate the scheduling code that
calls try_schedule/try_send and, in the inspect_err (or match on try_send), call
the existing unlock/release function (or implement one) with the executor
identifier and the txn/account ids so locks are freed, and ensure the function
returns None after unlocking to preserve current control flow.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

magicblock-processor/src/scheduler/mod.rs (1)

193-221: ⚠️ Potential issue | 🟠 Major

drop(cache) is positioned too late to prevent the deadlock described in the comment.

The comment on line 219 says "Release lock before syscall lookup (prevents deadlock if sysvar is accessed)", yet the program_cache write lock is held during all accountsdb operations (lines 202–217). If those operations can trigger program cache lookups, the deadlock can still occur. Move the drop immediately after the cache operations are complete:

🐛 Suggested fix — drop cache before sysvar updates

         cache.prune(block.slot, 0);
         cache.latest_root_slot = block.slot;
 
+        // Release lock before sysvar updates (prevents deadlock if sysvar
+        // access triggers a program cache lookup)
+        drop(cache);
+
         // Update Clock sysvar
         if let Some(mut account) = self.accountsdb.get_account(&clock::ID) {
             let _ = account.serialize_data(&block.clock);
@@ ..
             let _ = self.accountsdb.insert_account(&slot_hashes::ID, &acc);
         }
-        // Release lock before syscall lookup (prevents deadlock if sysvar is accessed)
-        drop(cache);
     }

🤖 Fix all issues with AI agents

In `@magicblock-processor/src/scheduler/mod.rs`:
- Around line 201-205: The Clock sysvar update currently swallows errors from
account.serialize_data and self.accountsdb.insert_account; update the block that
gets the clock account (accountsdb.get_account & clock::ID) to check each Result
and emit a warning on failure (mirroring the SlotHashes update's behavior) so
failures in account.serialize_data(...) and self.accountsdb.insert_account(...)
are logged with context (e.g., "failed to serialize Clock sysvar" and "failed to
insert Clock account") including the error details.