Repository navigation
fix(runtime): release stateless worker activation timeout timers - #11663
Open
KSemenenko wants to merge 4 commits into
Open
KSemenenko wants to merge 4 commits into
KSemenenko wants to merge 4 commits into
Conversation
Use the configured activation deadline already owned and disposed by ActivationData instead of creating a duplicate timed CTS in the stateless wrapper with no owner to release it after successful activation. Cover real timer inventory, activation timeout, caller cancellation in both deadline orderings, and wrapper deactivation/disposal. Assert that callback completion precedes scope disposal and that subsequent calls remain healthy.
ProtoIntClass declares messages without services. Keep C# protobuf model generation while avoiding the unused gRPC C# plugin, which the current Grpc.Tools package supplies only as macOS x64.
Timer.ActiveCount can decrease when timers left by earlier collections expire, even with test parallelism disabled. Run the exact inventory assertion in a fresh test process using the established self-hosted test pattern. Keep the equality checks, join process and stream cleanup, and remove the unused parent cluster fixture.
Contributor
Code coverage
Report-only conclusion: current-main baseline stale. The newest successful coverage run tested 9a0c764, not current main 625b2ba. Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New stateless-worker activations leave a duplicate activation-timeout CTS and timer pending after activation has completed.
CreateWorkercreates a timed source but never disposes it, whileActivationData.Activateindependently applies the same configured timeout and already owns operation cleanup.Cause and allocation impact
The wrapper's source is linked into the activation command's CTS. Disposing that linked command CTS removes the registration but does not dispose the wrapper's source. The extra source and timer remain queued until their deadline, even if the worker finished activation or deactivated earlier.
This is an avoidable CTS, timer and link registration per newly created stateless worker. Retention is bounded by
ActivationTimeout(30 seconds by default), and repeated messages to a stable worker pool do not reproduce the timer growth. Exact allocated/retained bytes, CPU savings and throughput gains have not been measured.Correction and timeout ownership
Pass
CancellationToken.Nonefrom the wrapper and keepActivationDataas the sole owner of the configured activation timeout. It creates the CTS, appliesGrainCollectionOptions.ActivationTimeout, and disposes the source through the activation command'sfinallycleanup.The removed token was locally created from that same option, rather than supplied by a client request. Genuine caller tokens remain linked at the
ActivationDataboundary: external cancellation can shorten an activation attempt, and the configured runtime deadline still applies if a caller token is absent or has a longer lifetime. Runtime deactivation and wrapper disposal continue to cancel pending activation work. Cancellation remains cooperative, with the callback joined before activation services are released.Evidence and regressions
The real-cluster timer regression fails before the fix with 9 timers expected versus 109 observed after 100 additional completed activations. After the fix, those activations leave the warmed timer baseline unchanged. A separate 10.4.2 native control has no timer growth for 10,000 calls to a stable pool, while 10,000 newly created/deactivated stateless workers leave 10,000 additional timers pending through forced GC before their deadlines expire.
The committed regressions exercise the real one-second activation timeout, both caller-token/configured-timeout orderings, wrapper
DeactivateandDisposewith a one-hour activation timeout, joined callback lifetime, scoped-service cleanup, and healthy replacement activation. The direct caller-token cases exercise the sharedActivationDataboundary; stateless-worker cases exercise the wrapper and real workers. Focused lifecycle, management, race and activation-cancellation regressions run on both net8.0 and net10.0. The exact timer inventory assertion runs as the only selected test in a fresh process, since timers left by previous collections can expire in a shared test host even when collection parallelism is disabled. The parent reuses its current apphost/dotnet host, checks that one test ran successfully, and joins child-process and output-stream cleanup.All 18 focused cases pass on each runtime, net8.0 and net10.0, and whitespace verification passes. The refreshed branch includes current main
625b2baf761fa3cbfc35d1125c6b0d3244fdd847. Its canonical full-solution build passes with warnings treated as errors: zero warnings and zero errors.This change leaves public APIs, serialization, persistence, activation-outcome policy and startup scheduling intact. Callback joining is already provided by the target main branch.
Build validation marks the message-only benchmark protobuf input with
GrpcServices="None", matching the serialization test project. Protobuf C# model generation still runs, while the benchmark no longer invokes a service plugin for an input without services. Native ARM64 validation uses official protoc 33.5 through the supportedPROTOBUF_PROTOCsetting.Closes #11662
Regression sources: timer inventory and timeout, caller cancellation and wrapper shutdown.
Microsoft Reviewers: Open in CodeFlow