fix(journaling)!: recover failed operations in new activations

docs/site/src/content/docs/grains/journaling/azure-storage.md

@@ -60,7 +60,7 @@ Metadata-enabled catalog queries select the header's format and caller-owned met
 
 ## Optimistic concurrency
 
-Both providers condition append, replace, and delete operations on the last observed ETag. Metadata-only conflicts receive a bounded in-place refresh and retry. A journal-content conflict raises <xref:Orleans.Storage.InconsistentStateException>, which causes the Journaling state manager to recover before later work.
+Both providers condition append, replace, and delete operations on the last observed ETag. Metadata-only conflicts receive a bounded in-place refresh and retry. A journal-content conflict raises <xref:Orleans.Storage.InconsistentStateException>, which permanently fences the Journaling state manager and requests grain deactivation. A fresh activation replays the durable journal.
 
 Configure the metadata-only retry cap and backoff with the corresponding `MaxMetadataOnlyConflictRetries`, `MetadataOnlyConflictInitialBackoff`, and `MetadataOnlyConflictMaxBackoff` options.
 

docs/site/src/content/docs/grains/journaling/index.md

@@ -26,6 +26,10 @@ A journaling grain derives from <xref:Orleans.Journaling.DurableGrain> and recei
 
 Mutations update the activation's in-memory state and add encoded operations to its pending journal buffer. Await <xref:Orleans.Journaling.DurableGrain.WriteStateAsync*> at the application durability point. The returned task completes after the storage provider acknowledges the append or snapshot replacement.
 
+Prepare fallible work in operation-local data and stage mutations once they are safe to commit. Interleaved
+calls share the pending journal. A failed journal operation fences the manager and requests grain deactivation;
+a fresh activation reconstructs the durable outcome before processing resumes.
+
 Each named state has a stable stream identity within the grain journal. Keep those names stable across deployments so recovery can bind stored operations to the intended state.
 
 ## Journal lifecycle

docs/site/src/content/docs/grains/journaling/operations.md

@@ -103,8 +103,8 @@ Exercise restore procedures regularly:
 | --- | --- | --- |
 | Activation fails with a journal format key | Recovery leaves storage unchanged | Deploy the required format and command codecs, or restore metadata which names the actual stored format |
 | Activation reports malformed or truncated journal data | Recovery stops before activation completes | Quarantine writes, restore a consistent backup, and preserve the failed data for diagnosis |
-| <xref:Orleans.Storage.InconsistentStateException> during a write | The write faults and the manager recovers before later work | Find stale or competing writers, confirm grain identity and cluster configuration, then retry the application command according to its idempotency contract |
-| Snapshot replacement fails after its pending append succeeds | The write faults while the mutation remains durable in the append history | Reconcile using the operation identifier and retry the command only through its idempotency protocol |
+| <xref:Orleans.Storage.InconsistentStateException> during a write | The manager fences operations and requests deactivation; a fresh activation recovers durable state | Find stale or competing writers, confirm grain identity and cluster configuration, then retry the application command according to its idempotency contract |
+| Storage commits a write but its acknowledgement is lost | The manager fences operations and requests deactivation; the new activation observes the committed mutation | Reconcile using the operation identifier and retry the command only through its idempotency protocol |
 | Write latency rises with storage queue duration | Writes for the journal are waiting behind earlier operations | Inspect provider latency and throttling, hot-grain traffic, write frequency, and snapshot size |
 | Recovery time and journal bytes grow | The current append history is large | Confirm compaction thresholds are enabled and that snapshot replacements succeed |
 | Format migration reports an unregistered state | A retired stream still requires its previous codec | Re-register the state for migration or retain the previous write format until retirement and compaction remove it |

docs/site/src/content/docs/grains/journaling/redis-journal-storage.md

@@ -23,7 +23,7 @@ The [runnable Redis Journaling sample](samples.md#run-the-redis-sample) uses the
 
 The provider stores journal data in Redis strings and journal metadata in Redis hashes. Per-journal reads and mutations use atomic Lua scripts. Catalog operations discover journals by scanning metadata keys on each connected primary Redis server.
 
-Optimistic concurrency protects append, replace, and delete operations. A stale writer receives <xref:Orleans.Storage.InconsistentStateException>, and the Journaling state manager recovers the stored journal before processing later work. Concurrent reads observe either the journal before a replacement or the complete replacement.
+Optimistic concurrency protects append, replace, and delete operations. A stale writer receives <xref:Orleans.Storage.InconsistentStateException>, and the Journaling state manager permanently fences operations and requests grain deactivation. A fresh activation recovers the stored journal. Concurrent reads observe either the journal before a replacement or the complete replacement.
 
 <xref:Orleans.Journaling.RedisJournalStorageOptions> supports:
 

docs/site/src/content/docs/grains/journaling/runtime-behavior.md

@@ -35,29 +35,52 @@ Durable collections encode their operation before applying it to the in-memory c
 Concurrent calls made while the same kind of write is queued can share that queued operation. Each caller observes its completion or failure. Calls made after a storage operation starts are processed by a later operation.
 
 > [!IMPORTANT]
-> In-memory mutation is visible before storage acknowledgement. Return success to a caller only after the required <xref:Orleans.Journaling.DurableGrain.WriteStateAsync*> completes. Recovery can rewind changes that were never acknowledged by storage.
+> In-memory mutation is visible before storage acknowledgement. Return success to a caller only after the required <xref:Orleans.Journaling.DurableGrain.WriteStateAsync*> completes. Recovery reconstructs durable state in a new activation.
+
+## Safe-to-commit staging
+
+All interleaved callers share the manager's pending journal. Prepare fallible work, external acknowledgements,
+and proposed output in operation-local data. After establishing that an outcome is safe to commit, apply its
+mutations to durable state and initiate a write. Coordinate that transition with other interleaved operations
+which can affect the same decision. Any caller's write can include staged mutations from other calls.
+
+If an application error occurs after staging and makes those mutations unsafe to commit, end the activation's
+use of the manager and request deactivation. In-flight methods can retain local decisions and references
+across awaits; a fresh activation reconstructs both application and durable state together.
 
 ## Consistency and competing writers
 
 Orleans grain placement normally supplies a single active writer for a grain identity. Journal storage providers also use optimistic concurrency to protect the journal when a stale or competing writer reaches storage.
 
-An append, replacement, or delete with stale storage metadata throws <xref:Orleans.Storage.InconsistentStateException>. The state manager marks the journal for recovery, resets its in-memory durable states, and replays the current stored journal before processing later work. The failed write task remains faulted so the grain call reports an uncertain outcome.
+An append, replacement, or delete with stale storage metadata throws <xref:Orleans.Storage.InconsistentStateException>. The state manager permanently fences further operations, faults queued work, and requests grain deactivation. The failed operation reports its original exception. A new activation replays the stored journal to determine the durable outcome.
 
 Design commands to tolerate retries at the application boundary. Use operation identifiers when a caller can repeat a command after an uncertain network outcome.
 
 ## Storage failures
 
-A normal append failure leaves encoded pending entries available for a later write attempt.
+A failed append, snapshot replacement, delete, or initialization permanently fences that manager instance.
+Queued operations fault, and later write, delete, registration, and initialization requests fail explicitly.
+Existing in-memory state remains available to in-flight calls until deactivation completes. The grain runtime
+starts deactivation as part of handling the failure.
+
+A storage operation can commit before its acknowledgement is lost. Treat a failed write as an uncertain
+application outcome and reconcile in a new activation using an operation identifier or another idempotency
+mechanism.
+
+Owners of standalone managers created through <xref:Orleans.Journaling.IJournaledStateManagerFactory>
+dispose the failed manager and create a new one for the same <xref:Orleans.Journaling.JournalId>. Register
+new durable state instances and initialize them before resuming processing.
 
-A compaction write has two storage stages: it first appends committed pending entries, then publishes a snapshot replacement. The append can succeed before the replacement fails. In that outcome, <xref:Orleans.Journaling.DurableGrain.WriteStateAsync*> faults even though the state mutation is durable in the append history. Treat every failed write as an uncertain application outcome and retry commands using an operation identifier or another idempotency mechanism.
+Cancelling a write's cancellation token stops the caller's wait. An already queued write continues to its
+storage outcome, so the caller reconciles that outcome before retrying the command.
 
-Recovery exceptions fault activation or queued work rather than replacing or truncating stored data. Restore the required format/codec registration or repair the backing data before retrying activation.
+An initialization failure preserves stored data for diagnosis. Restore the required format/codec registration
+or repair the backing data before creating a fresh manager or retrying activation.
 
 ## Compaction
 
 Each provider reports when its journal crosses a configured storage threshold. The next <xref:Orleans.Journaling.DurableGrain.WriteStateAsync*>:
 
-1. Persists any already-buffered append data.
 1. Builds a snapshot containing the state directory and every active durable state.
 1. Atomically replaces the published journal with the snapshot.
 1. Clears the compaction request after storage acknowledges the replacement.

src/Orleans.Journaling/CompatibilitySuppressions.xml

@@ -1,6 +1,13 @@
 <?xml version="1.0" encoding="utf-8"?>
 <!-- https://learn.microsoft.com/dotnet/fundamentals/package-validation/diagnostic-ids -->
 <Suppressions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
+  <Suppression>
+    <DiagnosticId>CP0002</DiagnosticId>
+    <Target>M:Orleans.Journaling.IJournaledStateManager.RevertPendingChangesAsync(System.Threading.CancellationToken)</Target>
+    <Left>lib/net10.0/Orleans.Journaling.dll</Left>
+    <Right>lib/net10.0/Orleans.Journaling.dll</Right>
+    <IsBaselineSuppression>true</IsBaselineSuppression>
+  </Suppression>
   <Suppression>
     <DiagnosticId>CP0002</DiagnosticId>
     <Target>M:Orleans.Journaling.IJournalStorageCatalog.ListAsync(Orleans.Journaling.JournalId,System.Threading.CancellationToken)</Target>
@@ -15,6 +22,13 @@
     <Right>lib/net10.0/Orleans.Journaling.dll</Right>
     <IsBaselineSuppression>true</IsBaselineSuppression>
   </Suppression>
+  <Suppression>
+    <DiagnosticId>CP0002</DiagnosticId>
+    <Target>M:Orleans.Journaling.IJournaledStateManager.RevertPendingChangesAsync(System.Threading.CancellationToken)</Target>
+    <Left>lib/net8.0/Orleans.Journaling.dll</Left>
+    <Right>lib/net8.0/Orleans.Journaling.dll</Right>
+    <IsBaselineSuppression>true</IsBaselineSuppression>
+  </Suppression>
   <Suppression>
     <DiagnosticId>CP0002</DiagnosticId>
     <Target>M:Orleans.Journaling.IJournalStorageCatalog.ListAsync(Orleans.Journaling.JournalId,System.Threading.CancellationToken)</Target>

src/Orleans.Journaling/IJournaledState.cs

@@ -14,7 +14,7 @@ namespace Orleans.Journaling;
 /// both apply the mutation locally and emit the corresponding command to the journal).
 /// </item>
 /// <item>
-/// At the end of the grain turn, the journaled state manager calls
+/// When the application requests a write, the journaled state manager calls
 /// <see cref="AppendEntries"/> (and occasionally <see cref="AppendSnapshot"/>) to materialize
 /// the pending changes, then flushes the journal to durable storage.
 /// </item>
@@ -25,16 +25,14 @@ namespace Orleans.Journaling;
 /// rather than treating in-memory mutations as durable.
 /// </item>
 /// <item>
-/// On failure, the journaled state manager triggers recovery by calling <see cref="Reset"/>
-/// followed by replaying snapshots and entries from durable storage. Implementations must
-/// therefore make any volatile in-memory bookkeeping fully recoverable from the journal.
+/// A failed journal operation permanently fences the manager and requests grain deactivation.
+/// A new manager initializes new state instances by calling <see cref="Reset"/> and replaying durable entries.
 /// </item>
 /// </list>
 /// <para>
-/// In other words, <em>in-memory mutations are journaled within the same grain turn and become
-/// durable when the journal flushes</em>. Implementations are free to apply mutations eagerly
-/// (before the durable write completes); recovery rebuilds in-memory state from the journal,
-/// so a turn-failure-and-recovery cycle observably rewinds any unflushed changes.
+/// Application code prepares fallible work in operation-local data and stages only mutations which are
+/// safe to commit. Staged mutations are shared by all interleaved callers using the same manager.
+/// Storage acknowledgement establishes durability; recovery takes place in a fresh manager and state instances.
 /// </para>
 /// </remarks>
 public interface IJournaledState
@@ -55,7 +53,7 @@ public interface IJournaledState
     /// </summary>
     /// <remarks>
     /// If the state has any volatile state, it must be cleared by this method.
-    /// This method can be called at any point in the state's lifetime, including during recovery.
+    /// The manager calls this method when binding a stream, deleting the journal, or retiring a state.
     /// </remarks>
     void Reset(JournalStreamWriter writer);
 

src/Orleans.Journaling/IJournaledStateManager.cs

@@ -3,16 +3,19 @@
 namespace Orleans.Journaling;
 
 /// <summary>
-/// Manages the states for a given grain.
+/// Manages the durable states associated with a journal.
 /// </summary>
 public interface IJournaledStateManager : IAsyncDisposable
 {
     /// <inheritdoc/>
     ValueTask IAsyncDisposable.DisposeAsync() => default;
 
     /// <summary>
-    /// Initializes the state manager.
+    /// Initializes the state manager by replaying its journal.
     /// </summary>
+    /// <remarks>
+    /// A failed initialization permanently fences this instance. Recover by creating a new manager and new state instances.
+    /// </remarks>
     /// <param name="cancellationToken">The cancellation token.</param>
     /// <returns>A <see cref="ValueTask"/> which represents the operation.</returns>
     ValueTask InitializeAsync(CancellationToken cancellationToken);
@@ -34,20 +37,24 @@ public interface IJournaledStateManager : IAsyncDisposable
     /// <summary>
     /// Prepares and persists an update to the journal.
     /// </summary>
+    /// <remarks>
+    /// Stage mutations only after the operation has established that they are safe to commit. Pending changes
+    /// are shared by all callers using this manager. A write failure permanently fences the manager and requests
+    /// deactivation of its owning grain. Owners of standalone managers must dispose the failed instance and
+    /// create a new manager with new state instances to recover durable state.
+    /// Cancellation stops the caller's wait; an already queued write continues to completion.
+    /// </remarks>
     /// <param name="cancellationToken">The cancellation token.</param>
     /// <returns>A <see cref="ValueTask"/> which represents the operation.</returns>
     ValueTask WriteStateAsync(CancellationToken cancellationToken);
 
-    /// <summary>
-    /// Discards uncommitted mutations and reloads the last durable state.
-    /// </summary>
-    /// <param name="cancellationToken">The cancellation token.</param>
-    /// <returns>A <see cref="ValueTask"/> which represents the operation.</returns>
-    ValueTask RevertPendingChangesAsync(CancellationToken cancellationToken);
-
     /// <summary>
     /// Resets this instance, removing any persistent state.
     /// </summary>
+    /// <remarks>
+    /// Quiesce other operations before deleting state: deletion resets every registered durable state.
+    /// A failed deletion permanently fences the manager and requests deactivation of its owning grain.
+    /// </remarks>
     /// <param name="cancellationToken">The cancellation token.</param>
     /// <returns>A <see cref="ValueTask"/> which represents the operation.</returns>
     ValueTask DeleteStateAsync(CancellationToken cancellationToken);