[ReubenBond]

Top of the durable messaging stack: #11282 (contracts) → #11284 (inbox) → #11285 (outbox) → #10693 (integration, documentation, samples, NativeAOT smoke, benchmarks, and playground). Runtime prerequisites #11505 and #11506 are merged into main.

Adds Microsoft.Orleans.Messaging, built on Orleans Journaling and Durable Jobs. Fixes #11475 and #11462.

Problem

Command-driven grains need atomic business state and outgoing intent, recoverable accepted work, reclaimable identity history, and durable application action when delivery stops with rejection or unknown outcome.

Solution

• One journaled send time: staging records SentAt once; every attempt and recovery reuses it. Retries end at SentAt + OutboxRetryTimeout, with that budget shorter than DeduplicationWindow.
• Bound newly stored times, preserve known answers: following the durable-stack coordinator's decision, known equal/earlier sends get their recorded answer even after rollback puts them beyond the current future-time bound. New admissions and later extensions beyond receiver time plus the window throw without staging. Untracked times at or below the durable floor receive Expired first. Equality at the bound is admitted.
• Durable retention and replay: a non-decreasing journaled admission floor preserves expiration after rollback. Acknowledged snapshots and recovery reclaim eligible completed records. Processed max-date extensions are self-contained upserts, so replay succeeds when the preceding snapshot omitted the older record. Pending extension deltas require their entry explicitly; sequence barriers protect interleaved changes. Scalar acknowledgements remain constant-time.
• Journal-owned processing: acceptance, business changes, outgoing intent, and completion use their owning atomic boundaries. The pump, Durable Jobs wake-up, and activation keep-alive resume ready work. Scheduled messages occupy capacity until due and completed.
• Crash-safe application action: IOutboxMessageSource.OnRetired stages terminal state and executable follow-up atomically with removal. A self-addressed follow-up resumes after recovery. Rejected means durable receiver rejection; DeadLettered includes retry exhaustion, source failure, and Expired with unknown earlier acceptance.

Integration layer

Messaging guides, operational/idempotency guidance, compiled recipes, a stock-reservation sample, NativeAOT smoke coverage, and a sequential benchmark exercise the API. playground/OrderProcessing demonstrates checkout, a durable grace period, stock reservation, payment authorization, and a buyer projection.

Operations guidance distinguishes expiration from future-time exceptions, preserves known outcomes after lost replies and clock rollback, and explains clock diagnosis, window sizing, and application reconciliation. Warnings include receiver grain identity. Automatic retries retain the original timestamp; a newly authorized submission gets a new one.

Rationale

The journal supplies atomicity and acknowledgement. Bounded new send times and the durable floor make history reclaimable while preserving recorded decisions. Existing source callbacks and inbox work supply recoverable application action across failures and lifecycle transitions.

Current-head CI, including NativeAOT publishing/trimming coverage, remains required before merge.

[Copilot]

Pull request overview

Adds a new durable, grain-scoped inbox/outbox messaging primitive (Microsoft.Orleans.DurableMessaging) built on Orleans Journaling + Durable Jobs, alongside supporting journaling participant/observer capabilities and Durable Jobs execution semantics needed for reliability, routing, and recovery.

Changes:

• Introduces Microsoft.Orleans.DurableMessaging (envelopes, routing handlers, inbox/outbox contracts, DI/hosting integration, diagnostics, and instrumentation).
• Extends Journaling with composable grain participants and write/recovery observers, plus rollback fencing and codec fixes for snapshot reference-scope replay.
• Extends Durable Jobs with feature handler registry, explicit RetryAt disposition/reset-rescheduling support, and improved execution deduplication.

Show a summary per file

File	Description
test/Orleans.Journaling.Tests/OrleansBinaryCommandCodecTests.cs	Adds regression tests ensuring pre-change snapshot payloads replay with independent reference scopes.
test/Orleans.Journaling.Tests/JournaledGrainParticipantTests.cs	Adds integration coverage for composed journaling participants initialization and failure propagation.
test/Orleans.Journaling.Tests/DurableListDirectWriteTests.cs	Updates test double to implement new revert/rollback surface.
test/Orleans.Journaling.Tests/DurableCollectionDirectWriteTests.cs	Updates test double to implement new revert/rollback surface.
test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs	Adds snapshot-based test synchronization utility.
test/Orleans.DurableMessaging.Tests/Support/HandlerProbe.cs	Adds barrier utility to block/release handlers deterministically in tests.
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingTestGrains.cs	Adds durable-messaging test grain, message/effect models, and handler behavior for scenarios.
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingClusterFixture.cs	Adds reusable in-process cluster fixture with Durable Jobs + Journaling + Durable Messaging wiring.
test/Orleans.DurableMessaging.Tests/Support/ControlledJournalStorageProvider.cs	Adds controllable journal storage provider for injecting write blocking/failures in tests.
test/Orleans.DurableMessaging.Tests/Orleans.DurableMessaging.Tests.csproj	Introduces new Durable Messaging test project.
test/Orleans.DurableMessaging.Tests/Hosting/PublicDurableMessagingRegistrationTests.cs	Validates DI registrations, options behavior, and absence of friend-access requirements.
test/Orleans.DurableMessaging.Tests/Functional/MultiSiloDurableMessagingFailoverTests.cs	Adds multi-silo failover recovery test for stable inbox job ownership and exactly-once effects.
test/Orleans.DurableMessaging.Tests/Functional/InboxCapacityBehaviorTests.cs	Adds backpressure + recovery behavior tests under capacity constraints.
test/Orleans.DurableMessaging.Tests/Functional/DedupeExpiryBehaviorTests.cs	Adds dedupe expiry/compaction behavior test allowing later reprocessing.
test/Orleans.DurableMessaging.Tests/Contracts/HandlerRoutingContractTests.cs	Adds routing contract tests for exact, prefix, correlation, and typed handler behaviors.
test/Orleans.DurableMessaging.Tests/Contracts/DurableEnvelopeContractTests.cs	Adds envelope/builder/serializer contract tests including reply-to and context.
test/Orleans.DurableMessaging.Tests/Contracts/DeliveryAndOptionsContractTests.cs	Adds contracts for delivery result/status and options validation defaults/boundaries.
test/Orleans.DurableJobs.Tests/DurableJobs/JobShardTests.cs	Adds shard-level test distinguishing reschedule reset vs failure retry attempt semantics.
test/Orleans.DurableJobs.Tests/DurableJobs/JobShardManagerTestsRunner.cs	Adds reassignment test ensuring reschedule reset semantics persist through shard reassignment.
test/Orleans.DurableJobs.Tests/DurableJobs/DurableJobFeatureHandlerTests.cs	Adds coverage for registry behavior and durable job run-result compatibility semantics.
test/Orleans.Core.Tests/Orleans.Core.Tests.csproj	Links HierarchicalKey tests into core test project.
test/Orleans.Core.Tests/DurableJobs/ShardExecutorTests.cs	Adds executor tests for RetryAt, unknown disposition handling, and legacy shard behavior.
test/Orleans.Core.Tests/DurableJobs/DurableJobsExtensionsTests.cs	Adds DI scope/registry behavior tests and explicit replacement/decoration rejection.
test/Orleans.Core.Tests/DurableJobs/DurableJobReceiverExtensionTests.cs	Updates execution identity to (JobId, RunId), adds retention/dedup + feature handler precedence tests.
test/Benchmarks/Journaling/DurableListJournalBenchmarks.cs	Updates benchmark test double to implement new revert/rollback surface.
src/Orleans.Journaling/JournaledStateManager.cs	Adds rollback support flag, observer notifications, recovery fencing, and revert-pending-changes support.
src/Orleans.Journaling/IJournaledStateObserver.cs	Introduces observer interface for write/recovery boundaries.
src/Orleans.Journaling/IJournaledStateManager.cs	Adds SupportsRollback, observer registration, and RevertPendingChangesAsync surface.
src/Orleans.Journaling/IJournaledGrainParticipant.cs	Introduces participant initialization hook for composed journaling features.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableSetCommandCodec.cs	Fixes snapshot replay to reset reference scope per element.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableQueueCommandCodec.cs	Fixes snapshot replay to reset reference scope per element.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableListCommandCodec.cs	Fixes snapshot replay to reset reference scope per element.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableDictionaryCommandCodec.cs	Fixes snapshot replay to reset reference scope per key/value entry.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryCommandCodecHelpers.cs	Adds helper to read values using independent serializer sessions (reference-scope reset).
src/Orleans.Journaling/DurableGrain.cs	Initializes composed participants before recovery and documents WriteStateAsync.
src/Orleans.DurableMessaging/RoutePrefixHandler.cs	Adds prefix-based routing handler base class and documentation.
src/Orleans.DurableMessaging/RouteKeyHandler.cs	Adds exact-route routing handler base class and documentation.
src/Orleans.DurableMessaging/README.md	Adds package readme with configuration and semantics overview.
src/Orleans.DurableMessaging/Orleans.DurableMessaging.csproj	Adds new packable Durable Messaging project.
src/Orleans.DurableMessaging/InboxHandlerContext.cs	Adds handler context implementation for sending outbox messages and building envelopes.
src/Orleans.DurableMessaging/IInboxHandlerContext.cs	Adds public handler context contract.
src/Orleans.DurableMessaging/IInboxHandler.cs	Adds handler interfaces including typed handler adapter and documentation.
src/Orleans.DurableMessaging/IDurableOutbox.cs	Adds outbox public contract.
src/Orleans.DurableMessaging/IDurableMessagingDiagnostics.cs	Adds diagnostics contract + internal implementation for dead letters.
src/Orleans.DurableMessaging/IDurableInboxExtension.cs	Adds grain extension contract for durable inbox delivery.
src/Orleans.DurableMessaging/IDurableInbox.cs	Adds inbox public contract including handler registration and lookup.
src/Orleans.DurableMessaging/Hosting/DurableMessagingExtensions.cs	Adds ISiloBuilder/IServiceCollection registration, option validation, and rollback requirement checks.
src/Orleans.DurableMessaging/DurableMessagingPumpResults.cs	Adds pump execution result tracking and one-shot timer handle helper.
src/Orleans.DurableMessaging/DurableMessagingInstruments.cs	Adds meters/counters/histograms for inbox/outbox behavior and depth tracking.
src/Orleans.DurableMessaging/DurableMessagingGrainParticipant.cs	Ensures messaging services materialize via journaling participant initialization.
src/Orleans.DurableMessaging/DurableMessageState.cs	Adds durable state models for inbox/outbox attempts and dead letters.
src/Orleans.DurableMessaging/DurableInbox.cs	Adds inbox implementation with handler registration/selection and storage-backed message access.
src/Orleans.DurableMessaging/DurableEnvelopeData.cs	Adds deferred (slice-based) envelope body/context storage and accessors.
src/Orleans.DurableMessaging/DurableEnvelope.cs	Adds durable envelope type (routing, correlation, reply-to, metadata, and payload).
src/Orleans.DurableMessaging/DeliveryStatus.cs	Adds delivery status enum contract.
src/Orleans.DurableMessaging/DeliveryResult.cs	Adds delivery result struct contract and factories.
src/Orleans.DurableMessaging/CorrelationHandler.cs	Adds correlation-hierarchy-based routing handler base class.
src/Orleans.DurableMessaging/Configuration/DurableInboxOptions.cs	Adds durable messaging options and validation contract.
src/Orleans.DurableJobs/ShardExecutor.cs	Adds RetryAt handling with reset-rescheduling path and explicit legacy-shard failure.
src/Orleans.DurableJobs/JournaledJobShard.cs	Implements reset-rescheduling in journaled shard via IResettableJobShard.
src/Orleans.DurableJobs/JobShard.cs	Adds reset-rescheduling support to base shard and introduces IResettableJobShard.
src/Orleans.DurableJobs/IDurableJobReceiverExtension.cs	Updates receiver extension for feature-handler lookup, new execution identity, and retention semantics.
src/Orleans.DurableJobs/IDurableJobHandlerRegistry.cs	Adds activation-scoped feature handler registry + lookup implementation.
src/Orleans.DurableJobs/Hosting/DurableJobsOptions.cs	Adds completed-attempt retention option and validation.
src/Orleans.DurableJobs/Hosting/DurableJobsExtensions.cs	Registers registry in DI and explicitly rejects replacement/decoration.
src/Orleans.DurableJobs/DurableJobRunResult.cs	Adds RetryAt disposition and related fields/compatibility semantics.
src/api/Orleans.Journaling/Orleans.Journaling.cs	Updates generated API surface for journaling observer/participant/rollback additions.
src/api/Orleans.DurableJobs/Orleans.DurableJobs.cs	Updates generated API surface for feature handlers/registry and RetryAt semantics.
src/api/Orleans.Core.Abstractions/Orleans.Core.Abstractions.cs	Updates generated API surface to include HierarchicalKey type and codec.
Orleans.slnx	Adds Durable Messaging projects (src + tests) to the solution.
docs/site/src/data/unpublished-api-packages.json	Marks Durable Messaging package as unpublished.
docs/site/src/data/external-link-allowlist.json	Allow-lists NuGet link for unpublished Durable Messaging package.
docs/site/src/content/docs/toc.yml	Adds Durable Messaging doc page to Journaling section.
docs/site/src/content/docs/resources/nuget-packages.md	Adds Durable Messaging to NuGet package catalog and updates guidance blurb.
docs/site/src/content/docs/grains/durable-messaging.md	Adds end-user documentation for durable messaging semantics and requirements.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 83/83 changed files
• Comments generated: 4
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (4)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.DurableMessaging/DurableInbox.cs:43

• Exact-route handler storage uses the default string comparer, which is culture-sensitive. Route keys are protocol identifiers and should use ordinal comparisons (matching the rest of this PR’s routing semantics). Use StringComparer.Ordinal for _exactRouteHandlers to avoid surprising behavior under non-invariant cultures (e.g., Turkish-I issues).

        _inbox = inbox;
        _processed = processed;
        _handlers = new List<IInboxHandler>();
        _exactRouteHandlers = new Dictionary<string, IInboxHandler>();
        _capacity = capacity;

src/Orleans.Journaling/IJournaledStateObserver.cs:16

• The remarks contradict the interface contract: they say the manager does not notify observers before a write, but this interface defines OnWriteStarted and JournaledStateManager invokes it before capturing state. This is confusing for implementers; update the remarks to reflect the actual callback sequence.
  src/Orleans.DurableMessaging/RouteKeyHandler.cs:21
• The remarks suggest implementing IInboxHandler directly for prefix-based routing, but this PR also introduces RoutePrefixHandler for that exact scenario. Updating the docs helps steer users to the supported helper type and keeps the guidance consistent.
  src/Orleans.DurableMessaging/InboxHandlerContext.cs:125
• The XML docs reference IStateMachineManager.WriteStateAsync(), which doesn't exist in this package and appears to be a leftover name. This should point to IJournaledStateManager.WriteStateAsync() to avoid misleading API consumers.

• Files reviewed: 83/83 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/Orleans.DurableJobs/IDurableJobReceiverExtension.cs:185

• LongPollGetJobStatusAsync creates a CancellationTokenSource but never cancels it. That means the Task.Delay will always run to completion even when the job finishes quickly, causing avoidable timer allocations/work per call.

                using var cts = new CancellationTokenSource();
                var longPollDuration = TimeSpan.FromTicks(Math.Min(_shared.MessagingOptions.ResponseTimeout.Divide(2).Ticks, _shared.Options.JobStatusPollInterval.Ticks));
                await Task.WhenAny(Task.Delay(longPollDuration, cts.Token), state.Task);

                if (!state.Task.IsCompleted)
                {
                    return DurableJobRunResult.PollAfter(_shared.Options.JobStatusPollInterval);
                }

src/Orleans.DurableMessaging/Hosting/DurableMessagingExtensions.cs:58

• The DurableInboxOptions validation predicate swallows all exceptions and turns them into a generic validation failure. That can hide unexpected exceptions (eg, NullReferenceException) and makes misconfiguration harder to diagnose. Consider only converting known validation exceptions into a failed predicate and letting other exceptions bubble.

        optionsBuilder.Validate(
            options =>
            {
                try
                {
                    options.Validate();
                    return true;
                }
                catch
                {
                    return false;
                }
            },
            "DurableInboxOptions validation failed.");

• Files reviewed: 83/83 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs:33

• WaitAsync adds a waiter to the per-grain list and then returns waiter.Completion.Task.WaitAsync(...), but if that wait times out/throws, the waiter remains in _waiters indefinitely. This can leak memory and can also keep evaluating stale predicates on subsequent Publish calls. Consider awaiting with a cleanup finally that removes the waiter (and optionally removes the empty list from _waiters).

• Files reviewed: 85/85 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (4)

Previously missed (3) — in code that hasn't changed since the last review.

src/Orleans.Journaling/JournaledStateManager.cs:309

• WorkLoop captures a stable observer snapshot into the local observers array, but OnWriteStarted() is invoked by iterating _observers directly. If an observer is registered during OnWritePreparingAsync (or from within another observer callback), it can change _observers mid-iteration, causing inconsistent callback sequencing and potential InvalidOperationException due to collection mutation during enumeration. Use the captured observers snapshot for OnWriteStarted to keep the write boundary consistent and mutation-safe.
  src/Orleans.Journaling/JournaledStateManager.cs:493
• OnWriteCompleted() is invoked by iterating _observers directly, even though a stable observers snapshot was captured earlier for this write. If observers are registered during the write pipeline, enumerating the live HashSet can throw (collection modified) or cause some observers to see OnWriteCompleted without the corresponding OnWritePreparingAsync/OnWriteStarted. Use the per-write snapshot for completion callbacks.

This issue also appears on line 516 of the same file.
src/Orleans.Journaling/JournaledStateManager.cs:826

• Recovery completion notifies observers by enumerating _observers directly inside a lock. If any observer callback registers another observer (re-entrantly) this can mutate the HashSet during enumeration and throw, potentially breaking recovery. Take a snapshot of _observers before iterating so callbacks are mutation-safe and sequencing is stable.

src/Orleans.Journaling/JournaledStateManager.cs:516

• In the no-op write path (!hasCommittedBuffer), OnWriteCompleted() is called by iterating _observers directly. This has the same collection-mutation risk as the committed write path and can also notify observers which were registered after OnWritePreparingAsync ran for this write. Prefer using the stable observers snapshot captured at the start of the work item.

• Files reviewed: 85/85 changed files
• Comments generated: 0 new
• Review effort level: Lite

[ReubenBond]

Addressed the latest observer-snapshot review findings in \3d4829b8d. Each write now uses one stable observer snapshot for preparation, start, and completion (including no-op writes), and recovery snapshots observers before callbacks so re-entrant registration cannot mutate an active enumeration. Focused regression coverage passes on net8.0 and net10.0.

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs:33

• WaitAsync adds a waiter to the per-grain list but never removes it if the 30s timeout elapses (WaitAsync throws). That can leak waiters across a test run and slow down Publish() due to ever-growing lists.

• Files reviewed: 85/85 changed files
• Comments generated: 1
• Review effort level: Lite

[ReubenBond]

Addressed the remaining suppressed review findings in two focused commits:\n\n- \�222b4b77\ removes timed-out SnapshotProbe waiters so stale predicates are not retained or evaluated; regression coverage passes on net8.0 and net10.0.\n- \�d214b07b\ narrows durable inbox options validation to the documented \ArgumentOutOfRangeException, allowing unexpected faults to surface; the options-contract test passes on net8.0 and net10.0.

[Copilot]

Review details

Suppressed comments (1)

src/Orleans.Journaling/IJournaledStateManager.cs:58

• IJournaledStateManager is a public interface and RevertPendingChangesAsync is added as a required member (no default implementation). This is a breaking change for any external implementations of IJournaledStateManager. Consider providing a default interface implementation (similar to RegisterObserver) which throws NotSupportedException, and use SupportsRollback to indicate capability.

• Files reviewed: 86/86 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 86/86 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.DurableMessaging/DurableOutbox.cs:168

• The XML docs reference IStateMachineManager.WriteStateAsync(), but durable messaging commits are driven by journaling. This should reference IJournaledStateManager.WriteStateAsync() to match the rest of the package docs and avoid misleading API consumers.

    /// The message is persisted atomically with grain state when IStateMachineManager.WriteStateAsync()
    /// is called. The background pump will deliver the message to the target grain ONLY AFTER
    /// the message has been durably persisted.

• Files reviewed: 86/86 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The main blocker is a moderate issue in DurableTaskCompletionSource.cs: the replay handlers ApplyCompleted and ApplyFaulted call MarkModified during journal recovery, which inflates StagedChangeCount, while the remaining findings are documentation, test naming, and message nits.

5 open issues

• Replay handlers incorrectly mark changes as staged · New
• Update package docs to describe current behavior · New
• Let validation exceptions reach the caller · New
• Rename the test or write application state · New
• Fix subject-verb agreement in the sentence · New

_{Review effort: Lite}