Skip to content

fix(runtime): release stateless worker activation timeout timers - #11663

Open
KSemenenko wants to merge 4 commits into
dotnet:mainfrom
KSemenenko:codex/stateless-worker-activation-timeout
Open

KSemenenko wants to merge 4 commits into
dotnet:mainfrom
KSemenenko:codex/stateless-worker-activation-timeout

Conversation

@KSemenenko

@KSemenenko KSemenenko commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

New stateless-worker activations leave a duplicate activation-timeout CTS and timer pending after activation has completed. CreateWorker creates a timed source but never disposes it, while ActivationData.Activate independently applies the same configured timeout and already owns operation cleanup.

Cause and allocation impact

The wrapper's source is linked into the activation command's CTS. Disposing that linked command CTS removes the registration but does not dispose the wrapper's source. The extra source and timer remain queued until their deadline, even if the worker finished activation or deactivated earlier.

This is an avoidable CTS, timer and link registration per newly created stateless worker. Retention is bounded by ActivationTimeout (30 seconds by default), and repeated messages to a stable worker pool do not reproduce the timer growth. Exact allocated/retained bytes, CPU savings and throughput gains have not been measured.

Correction and timeout ownership

Pass CancellationToken.None from the wrapper and keep ActivationData as the sole owner of the configured activation timeout. It creates the CTS, applies GrainCollectionOptions.ActivationTimeout, and disposes the source through the activation command's finally cleanup.

The removed token was locally created from that same option, rather than supplied by a client request. Genuine caller tokens remain linked at the ActivationData boundary: external cancellation can shorten an activation attempt, and the configured runtime deadline still applies if a caller token is absent or has a longer lifetime. Runtime deactivation and wrapper disposal continue to cancel pending activation work. Cancellation remains cooperative, with the callback joined before activation services are released.

Evidence and regressions

The real-cluster timer regression fails before the fix with 9 timers expected versus 109 observed after 100 additional completed activations. After the fix, those activations leave the warmed timer baseline unchanged. A separate 10.4.2 native control has no timer growth for 10,000 calls to a stable pool, while 10,000 newly created/deactivated stateless workers leave 10,000 additional timers pending through forced GC before their deadlines expire.

The committed regressions exercise the real one-second activation timeout, both caller-token/configured-timeout orderings, wrapper Deactivate and Dispose with a one-hour activation timeout, joined callback lifetime, scoped-service cleanup, and healthy replacement activation. The direct caller-token cases exercise the shared ActivationData boundary; stateless-worker cases exercise the wrapper and real workers. Focused lifecycle, management, race and activation-cancellation regressions run on both net8.0 and net10.0. The exact timer inventory assertion runs as the only selected test in a fresh process, since timers left by previous collections can expire in a shared test host even when collection parallelism is disabled. The parent reuses its current apphost/dotnet host, checks that one test ran successfully, and joins child-process and output-stream cleanup.

All 18 focused cases pass on each runtime, net8.0 and net10.0, and whitespace verification passes. The refreshed branch includes current main 625b2baf761fa3cbfc35d1125c6b0d3244fdd847. Its canonical full-solution build passes with warnings treated as errors: zero warnings and zero errors.

This change leaves public APIs, serialization, persistence, activation-outcome policy and startup scheduling intact. Callback joining is already provided by the target main branch.

Build validation marks the message-only benchmark protobuf input with GrpcServices="None", matching the serialization test project. Protobuf C# model generation still runs, while the benchmark no longer invokes a service plugin for an input without services. Native ARM64 validation uses official protoc 33.5 through the supported PROTOBUF_PROTOC setting.

Closes #11662

Regression sources: timer inventory and timeout, caller cancellation and wrapper shutdown.

Microsoft Reviewers: Open in CodeFlow

Use the configured activation deadline already owned and disposed by
ActivationData instead of creating a duplicate timed CTS in the stateless
wrapper with no owner to release it after successful activation.

Cover real timer inventory, activation timeout, caller cancellation in both
deadline orderings, and wrapper deactivation/disposal. Assert that callback
completion precedes scope disposal and that subsequent calls remain healthy.
ProtoIntClass declares messages without services. Keep C# protobuf model generation while avoiding the unused gRPC C# plugin, which the current Grpc.Tools package supplies only as macOS x64.
Timer.ActiveCount can decrease when timers left by earlier collections expire, even with test parallelism disabled. Run the exact inventory assertion in a fresh test process using the established self-hosted test pattern. Keep the equality checks, join process and stream cleanup, and remove the unused parent cluster fixture.
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request
Lines 84.15% (121,370 / 144,229)
Branches 74.47% (36,792 / 49,405)

Report-only conclusion: current-main baseline stale.

The newest successful coverage run tested 9a0c764, not current main 625b2ba.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stateless worker activations retain a duplicate timeout CTS and timer after activation completes

1 participant