refactor(runtime): make activation states symmetric by ReubenBond · Pull Request #11652 · dotnet/orleans

ReubenBond · 2026-10-30T21:10:46Z

Activation state transitions were asymmetric. A deactivation request during setup set the activation's state to Deactivating while setup code was still running. Teardown then inferred which steps to undo from the state at request time and from special cases such as the DirectoryFailure unregister skip. This refactor makes the activation state machine a linear chain, symmetric around Active, so each teardown step pairs with the setup step it undoes. It lands before #11650 (part of #8812) so that fix reduces to the activation-outcome rule.

State machine

Setup		Teardown
`None`	↔	`Destroyed`
`Constructing` (instance, previous registration)	↔	`Disposing` (instance, services, leave catalog, forward)
`Registering`	↔	`Unregistering`
`Rehydrating`	↔	`Dehydrating` (migration)
`Starting` (lifecycle start)	↔	`Stopping` (lifecycle stop)
`Activating` (`OnActivateAsync`)	↔	`Deactivating` (`OnDeactivateAsync`)
`Active`

ActivationState is a byte with spaced values: None = 0, Active = 127, Destroyed = 254, and each teardown phase at Destroyed - s of its setup phase. Group checks are single comparisons against Active, and a new phase pair can be inserted at mirrored values without renumbering.
Each transition advances one state (GetNext). SetState asserts the transition (ActivationStateExtensions.IsValidTransition).
Teardown starts at the mirror of the last completed setup phase. A failed phase leaves the phase before it as the last completed one. Lifecycle stop tolerates a partial start, so a failed Starting phase enters at Stopping. Disposing tolerates partial construction.
Registering precedes Rehydrating so that only the activation which holds the directory registration installs migrated state. Dehydrating precedes Unregistering because a migrating activation unregisters only if its handoff fails.
A deactivation request during setup is recorded intent (IsDeactivationRequested), not a state. It cancels the setup token, and setup stops at its next phase boundary, leaving the activation in its last completed phase. The deactivation operation, which runs after setup, is the only place which maps that phase to its teardown entry. The request, any transition into teardown, and the scheduled deactivation operation are recorded before observers and cancellation callbacks run, so a reentrant request finds the first one and a recorded request always completes.
Forced disposal (DisposeAsync, used when the activation directory is disposed) is an explicit forced transition to Disposing. It skips the phases which have not run.

Setup is a sequence of phase methods (RegisterAsync, RunRehydration, StartLifecycleAsync, RunOnActivateAsync) separated by TryAdvanceSetup boundaries. Teardown is one loop over states, running one method per phase (RunOnDeactivateAsync, StopLifecycleAsync, DehydrateAsync, UnregisterAsync): Deactivating → Stopping → Dehydrating → Unregistering → Disposing → Destroyed. ActivateAsync and FinishDeactivating are each about half their size on main. This replaces the PreviousState captured when deactivation was requested, so OnDeactivateAsync and retirement migration run exactly when activation completed. Unregistering always removes the registration unless it was handed to a migration target; the DirectoryFailure reason no longer decides it.

Behavior changes

The aim is to preserve main's behavior, so the following list covers every change. An OnActivateAsync which returns normally after its token was canceled still fails activation. That rule changes in #11650.

Rehydration follows registration. Migration participants' OnRehydrate runs after directory registration, during Rehydrating. It used to run before registration. The previous registration is still read while the activation is constructed, so the registration CAS and the cache updates on queued requests are unchanged. When registration finds a competing activation, the target no longer rehydrates state it would discard.
Deactivation during registration.
- A registration which commits after a deactivation request now diverts at the registration boundary: the activation unregisters and never starts its lifecycle. Previously the activation re-entered Activating after the request.
- A registration which throws is treated as possibly committed, so teardown starts at Unregistering and unregisters it. On main, the DirectoryFailure reason skipped unregistration unless a deactivation request preceded the failure, which could leave a committed registration pointing at a destroyed activation. Unregistration is conditional on the activation identity, so it leaves other registrations in place. The same applies to activations which LegacyGrainDirectory deactivates when their directory partition owner fails: they now issue an unregistration which finds no matching entry.
- A competing registration (a duplicate) skips unregistering, because nothing of this activation's was registered.
- A deactivation before registration no longer issues a directory unregister call for an activation which never registered.
Migration. Only an activation which completed activation migrates (invariant 4 of the design note). A migration requested during setup stops setup, and the activation deactivates without migrating. On main, it migrated after setup stopped, which could send state the activation never loaded, such as grain state whose storage read was canceled, to the target as initialized state. TryStartMigration accepts requests from lifecycle start until teardown reaches Dehydrating, where the migration decision is made. It used to accept requests from registration until disposal, so it could return true for an activation which would never migrate.
Completion racing a request. A deactivation request which arrives after OnActivateAsync returns and passes its cancellation check, but before the activation becomes active, now runs OnDeactivateAsync. Activation completed, so teardown starts at Deactivating. That activation is reported as canceled: it emits no Activated event and no success metric.
Forced disposal mid-teardown. Forced disposal during teardown skips the remaining lifecycle stop, dehydration, and unregistration. The in-flight deactivation still leaves the catalog, forwards queued requests, and completes Deactivated. Forced disposal happens when the silo's services are disposed. Forced disposal during rehydration also skips migration participants which have not run.
Deactivation before start. A deactivation requested before the activation starts (for example, from IConfigureGrainContext) skips construction. Previously, constructing the grain instance threw.
Activation span. The activation span now ends when activation ends on every path, including registration failure, where it previously never ended.
State names. State names change in the orleans.grain.state activity tag, in logs, and in activation ToString/detailed reports, which IManagementGrain reports include. The mapping:
- Creating → None/Constructing/Registering/Rehydrating
- Activating → Starting/Activating
- Valid → Active
- Deactivating → Deactivating through Unregistering
- Invalid → Disposing/Destroyed
A deactivation requested during setup no longer shows as Deactivating until setup stops. The rehydrate span now follows the directory registration span.

ActivationState is internal, so there is no public API change.

Tests

ActivationStateMachineTests:
- An Accordant model-based test generates sequences of phase completions, phase failures, deactivation requests, migration requests, and forced disposal over a grain whose directory registration, lifecycle start, OnActivateAsync, OnDeactivateAsync, lifecycle stop, and directory unregistration wait at gates. It compares each phase, state, and migration-request outcome with the model.
- A mutation which entered Deactivating after a failed OnActivateAsync is caught.
- Also covered: the transition table, the teardown-entry rules, forced disposal during setup and teardown, and the completion race in change 4. The race is made deterministic with an ActivityListener hook.
ActivationDataMigrationTests: rehydration only after registration, discarding a migration requested during setup, deactivation during registration (registration commits or throws), deactivation during rehydration, deactivation before start, and the directory events for failed registrations.
ActivationAssertionRecorder fails the lifecycle tests on ActivationData assertion failures, which the activation's operation loop otherwise swallows.
Updated state names in the existing lifecycle tests.

Follow-up, unchanged from main: the 5-second back-off scheduled after a failed activation is canceled immediately by the deactivation it precedes.

Replace ActivationState with a linear chain which is symmetric around Active. Each setup phase pairs with the teardown phase which undoes it, teardown starts at the mirror of the last completed setup phase, a deactivation request during setup is recorded intent which diverts setup at its next phase boundary, and forced disposal is an explicit transition to Disposing. SetState asserts each transition. Part of dotnet#8812.

Copilot

🟡 Changes recommended

Forced disposal can trigger rehydration callbacks on disposed participants, and the regression test leaves an activation registered with deactivation incomplete, causing fixture shutdown to wait until timeout.

2 open issues

What changed in this PR

Refactors Orleans activation handling into a state chain symmetric around Active, pairing setup phases with teardown phases. This prepares for #11650’s activation-cancellation fix without introducing that behavior change here.

Changes:

Separates deactivation requests from lifecycle state and consolidates teardown into a phase-based pipeline.
Moves rehydration after registration and updates runtime state checks.
Adds model-based and regression tests, and documents the lifecycle changes.

File	Description
`test/Orleans.Runtime.Internal.Tests/Orleans.Runtime.Internal.Tests.csproj`	Adds Accordant for model-based tests.
`test/Orleans.Runtime.Internal.Tests/ActivationsLifeCycleTests/RetirementHandoffTests.cs`	Updates state and deactivation-request assertions.
`test/Orleans.Runtime.Internal.Tests/ActivationsLifeCycleTests/ActivationStateMachineTests.cs`	Adds transition, lifecycle-model, disposal, and completion-race tests.
`test/Orleans.Runtime.Internal.Tests/ActivationsLifeCycleTests/ActivationDataMigrationTests.cs`	Covers registration boundaries, rehydration ordering, and early deactivation.
`test/Orleans.Runtime.Internal.Tests/ActivationsLifeCycleTests/ActivationDataLifecycleTests.cs`	Updates the terminal-state assertion.
`src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs`	Uses termination intent when filtering activations.
`src/Orleans.Runtime/Core/GrainRuntime.cs`	Rejects runtime access during disposal and after destruction.
`src/Orleans.Runtime/Catalog/ActivationState.cs`	Defines symmetric states and transition rules.
`src/Orleans.Runtime/Catalog/ActivationData.cs`	Implements phased setup, teardown, and rehydration ownership.
`docs/site/src/content/docs/implementation/activation-lifecycle.md`	Documents the state chain and revised migration ordering.

_{Review effort: Balanced}

Setup boundaries stop setup without entering teardown; the deactivation operation alone maps the last completed phase to its teardown entry. ActivateAsync and FinishDeactivating become sequences of phase methods, and teardown is one loop over its phases. Rehydration-context ownership moves to a single TakeRehydrationContext helper.

Copilot

🟡 Changes recommended

A critical forced-disposal race can invoke OnRehydrate on an already-disposed grain and its services because the state is not rechecked under the lock.

4 open issues

_{Review effort: Balanced}

…ed disposal Recheck Rehydrating under the lock before invoking migration participants, so forced disposal which preempts rehydration skips them. Allow an activation operation queued before a pre-start deactivation request to find the activation in None. Remove the forcibly disposed activation from the catalog in the disposal test, as ActivationDirectory disposal does.

Copilot

🟡 Changes recommended

The unscoped rehydration listener can dispose the target activation during registration in response to a concurrent migration, allowing the test to pass without exercising the intended lifecycle gap.

1 open issue

Restrict rehydration hook to the target activation · New

4 resolved since last review

_{Review effort: Balanced}

…ctivation The process-wide activity listener now disposes only when the rehydrate span starts on the target activation's scheduler, and the test asserts the activation was Rehydrating at that point.

Copilot

🔵 Needs a closer look

Human assessment is needed because critical lifecycle reentrancy and disposal-exception paths can trigger invalid assertions and leave activation resources undisposed.

No open issues 🎉

1 resolved since last review

Restrict rehydration hook to the target activation

_{Review effort: Balanced}

…ntrant observers Dispose an unconsumed rehydration context alongside the service scope, so a context whose disposal throws cannot prevent the grain instance and services from being disposed. Deactivate no longer asserts that the activation is not disposing, since lifecycle observers and diagnostic listeners run synchronously inside it and may force disposal.

ReubenBond · 2026-10-31T04:15:24Z

Follow-up on the latest Copilot review summary ("lifecycle reentrancy and disposal-exception paths"), which had no inline threads. Both paths were real; fixed in dcce06e:

Disposal exception: an activation that tore down before rehydrating disposed its unconsumed migration context before the instance and scope disposal block. A context whose Dispose threw skipped disposing the grain instance and services. The context is now disposed alongside the service scope, through the same exception-tolerant helper. Covered by Runtime_UnconsumedRehydrationContextDisposalFailure_StillDisposesActivation, which fails without the fix.
Reentrancy: lifecycle observers and diagnostic listeners run synchronously inside Deactivate, and one that forced disposal broke Deactivate's postcondition assertion (!State.IsDisposingOrDestroyed()). The assertion now checks only that deactivation was requested. Covered by ForcedDisposal_FromDeactivatingObserver_CompletesDeactivation, which fails without the fix.

Copilot

🔵 Needs a closer look

Human assessment is needed because rehydration cleanup exceptions can abort activation, while reentrant disposal can allow callbacks to run after disposal begins.

No open issues 🎉

_{Review effort: Balanced}

…ion setup Deactivate and FailSetup record the request, the transition into teardown, and the scheduled deactivation before observers and cancellation callbacks run. RunOnActivateAsync reports observed cancellation, so forced disposal during OnActivateAsync is a cancellation, not a failure. The success tail runs after setup, the activation span ends on every path, a throwing rehydration context no longer fails activation, and teardown after a registration failure no longer receives the canceled activation token. Tests now fail on ActivationData assertion failures which the operation loop swallows, record lifecycle start, stop, and deactivation so teardown entries are exact, and observe directory calls for pre-start deactivation.

Copilot

🟡 Changes recommended

Reentrant disposal during rehydration can expose later callbacks to disposing or disposed services, and pre-start deactivation leaves the activation activity unfinished.

2 open issues

_{Review effort: Balanced}

…r in teardown Dehydrating discards a migration requested during setup, so an activation whose setup stopped never sends partially initialized state, such as grain state whose storage read was canceled. Unregistering removes the activation's registration regardless of the deactivation reason: a registration which threw may have committed, and unregistration is conditional on the activation's identity. The Accordant model now gates directory registration and unregistration and covers migration requests and forced disposal.

Copilot

🔵 Needs a closer look

Pre-start deactivation leaves the activation span unstopped, while reentrant disposal can allow subsequent migration callbacks to run on an activation whose services have already been released.

2 open issues

_{Review effort: Balanced}

ActivationState is byte-backed with None = 0, Active = 127, and Destroyed = 254, so the mirror remains Destroyed - s and new phase pairs can be inserted at mirrored values without renumbering. Transitions use an explicit GetNext rather than arithmetic. Rehydration rechecks Rehydrating before each migration participant, so a participant which forces disposal stops the rest. An activation operation which finds setup stopped before registration ends the activation span. Fixes the missing final newline flagged by the whitespace check.

Copilot

🔵 Needs a closer look

The lifecycle refactor requires maintainer validation of reentrancy and concurrent shutdown invariants across net8.0/net10.0, as runtime execution and affected-code coverage evidence were unavailable.

No open issues 🎉

2 resolved since last review

_{Review effort: Balanced}

…e cases Accordant's transition-coverage generation at depth 10 produces about 9,400 overlapping sequences covering 51 model transitions, which took minutes to execute. The test now greedily selects a deterministic subset which covers every transition the generated cases cover (19 cases), running in about a second.

Copilot

🔵 Needs a closer look

The coupled lifecycle changes require maintainer review and runtime validation on net8.0 and net10.0 because static review cannot establish cross-silo recovery, shutdown, or forced-disposal race guarantees.

No open issues 🎉

_{Review effort: Balanced}

github-actions · 2026-10-31T17:43:10Z

Code coverage

Metric	Pull request	Current main	Variance
Lines	84.12% (121,388 / 144,309)	84.10% (121,188 / 144,095)	+0.0139 pp
Branches	74.37% (36,773 / 49,445)	74.34% (36,648 / 49,295)	+0.0273 pp

Report-only conclusion: improved.

The current-main baseline is commit ebfde71720 and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

Pull request commit: 797425b318
View the pull request test run and coverage artifacts
View the trusted coverage reporting run

Copilot AI balanced review requested due to automatic review settings October 30, 2026 21:10

Copilot started reviewing on behalf of ReubenBond October 30, 2026 21:11

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Comment thread src/Orleans.Runtime/Catalog/ActivationData.cs Resolved

Comment thread test/Orleans.Runtime.Internal.Tests/ActivationsLifeCycleTests/ActivationStateMachineTests.cs Resolved

ReubenBond mentioned this pull request Oct 30, 2026

fix(runtime): honor successful activation after cancellation #11650

Open

Copilot AI balanced review requested due to automatic review settings October 30, 2026 22:45

Copilot started reviewing on behalf of ReubenBond October 30, 2026 22:46

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Comment thread src/Orleans.Runtime/Catalog/ActivationData.cs Resolved

Comment thread src/Orleans.Runtime/Catalog/ActivationData.cs Resolved

Copilot AI balanced review requested due to automatic review settings October 31, 2026 00:49

Copilot started reviewing on behalf of ReubenBond October 31, 2026 00:50

View session