fix: support cancellation during sandbox execution by baywet · Pull Request #275 · hyperlight-dev/hyperlight-sandbox

baywet · 2026-11-04T00:21:55Z

Summary

Related to #9.

Expose independently owned native interruption handles through the Rust backend abstraction and .NET FFI. Add Sandbox.Run(string, CancellationToken) and make RunAsync forward cancellation into running guest execution, while preserving Run(string).

Cancellation uses a dedicated interruption worker, retries the transition into guest execution, and waits for native execution and cancellation work to finish before returning. The callback never accesses mutable sandbox state or acquires the execution gate. Existing snapshot/restore behavior permits reuse after interruption.

There is no implicit SDK deadline: hosts can supply a timeout-backed CancellationTokenSource. Runtime initialization and host callbacks cannot be interrupted; host tools need their own cancellation policy.

Validation

Executed in Ubuntu x64 containers on a KVM-capable host using .NET SDK 10.0.403 and Rust 1.94. The net8.0 tests ran with DOTNET_ROLL_FORWARD=Major.

Before the implementation: dotnet vstest src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/bin/Debug/net8.0/HyperlightSandbox.Tests.dll --TestCaseFilter:FullyQualifiedName~JavaScript_RunAsync_CancellationDuringExecution_StopsGuestAsync --Logger:"console;verbosity=normal" failed as expected after a genuine warm-up and five seconds of guest execution: expected OperationCanceledException, but none was thrown.
just wasm guest-build — passed, building the actual Python/Wasm guest.
just fmt and just fmt-check — passed.
just lint-rust and just dotnet lint-rust — passed.
dotnet build src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/HyperlightSandbox.Tests.csproj --nologo -p:TreatWarningsAsErrors=true — passed, zero warnings/errors.
just test-rust — passed, 144 unit tests and 5 HTTP integration tests; one existing documentation example remains ignored.
just dotnet test-rust — passed, 78 FFI tests.
dotnet vstest src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/bin/Debug/net8.0/HyperlightSandbox.Tests.dll --Logger:"console;verbosity=normal" — passed, all 147 tests, including real JavaScript immediate/timer cancellation and recovery, pre-canceled execution, isolation from completed cancellation registrations, and Python/Wasm interruption followed by snapshot restoration.

The first Rust HTTP run failed because a network-disabled container could not resolve the existing HTTPS test's destination. The full Rust suite passed when run with its required networking. FFI and .NET guest tests remained resource-limited and network-disabled.

Windows execution, the Python SDK suite, and package publishing were not run locally; upstream CI remains pending. This PR must remain a draft.

AI assistance

Copilot assisted with implementation, tests, documentation, and this description.

Pin Node 26 in CI and publishing so builds use a supported LTS release rather than the Node 27 alpha selected by `latest`. Node 26 supports the existing manylinux_2_28 libraries, which preserves Linux wheel compatibility while allowing Node 26 patch updates. This addresses the build environment failure seen in #275 without changing its cancellation implementation. Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Copilot

🟡 Changes recommended

The interruption test can fail despite successful cancellation because kill() may return False while cancellation is pending, so the assertion should check that an attempt occurred rather than require a True result.

1 open issue

Avoid requiring kill() to return True for interruption · New

What changed in this PR

Adds interruption of running guest code across the Rust backend abstraction and SDK bindings, while preserving the existing .NET Run(string) API.

Changes:

Exposes independently owned interrupt handles for JavaScript and Wasm backends.
Connects .NET cancellation tokens and Python cross-thread interruption to native execution, preserving cancellation error types.
Adds interruption and snapshot-recovery tests, plus documentation of cancellation limits.

File	Description
`src/wasm_sandbox/src/lib.rs`	Exposes interruption and preserves typed execution errors.
`src/sdk/python/wasm_backend/src/lib.rs`	Adds interrupt handles and releases the GIL during execution.
`src/sdk/python/pyo3_common/src/lib.rs`	Shares Python interrupt bindings and error translation.
`src/sdk/python/hyperlight_js_backend/src/lib.rs`	Adds interrupt handles and releases the GIL during execution.
`src/sdk/python/core/tests/test_interruption.py`	Tests cross-thread interruption, handle lifetime, and recovery.
`src/sdk/python/core/hyperlight_sandbox/__init__.py`	Exposes and documents the public Python interruption API.
`src/sdk/dotnet/README.md`	Documents cancellation APIs, limits, and recovery.
`src/sdk/dotnet/ffi/src/lib.rs`	Exports interrupt-handle operations and classifies cancellation errors.
`src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/PInvokeLayerTests.cs`	Tests cancellation error mapping and late-cancellation behavior.
`src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/JavaScriptBackendTests.cs`	Tests JavaScript cancellation, recovery, and registration isolation.
`src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/IntegrationTests.cs`	Tests Python/Wasm cancellation and snapshot recovery.
`src/sdk/dotnet/core/PInvoke/SafeNativeMethods.cs`	Declares native interruption functions.
`src/sdk/dotnet/core/PInvoke/SafeHandles.cs`	Manages native interrupt-handle ownership.
`src/sdk/dotnet/core/PInvoke/FFIResult.cs`	Maps native cancellation to token-aware exceptions.
`src/sdk/dotnet/core/PInvoke/FFIErrorCode.cs`	Adds the native cancellation error code.
`src/sdk/dotnet/core/Api/Sandbox.cs`	Implements cancellable execution with interruption-worker cleanup.
`src/javascript_sandbox/src/lib.rs`	Exposes interruption and propagates cancellation and poison errors.
`src/hyperlight_sandbox/src/lib.rs`	Adds the shared interrupt handle and backend interface.
`src/hyperlight_sandbox/Cargo.toml`	Adds the native host dependency for interruption support.
`Cargo.lock`	Records the shared crate’s new dependency.

_{Review effort: Balanced}

Download the shared guest bundle after Rust cache restoration in every consumer job. On partial cache hits, rust-cache cleans target/ and deletes the previously downloaded bundle manifest, causing valid guests to fail verification. Keeping downloads after that cleanup preserves strict verification without rebuilding guests in each job. The regression failed in CI before the fix. This addresses the cache failure exposed by #275; it does not change cancellation behavior. --------- Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d6807e7-70a6-4c43-8dc1-7bab20376cd4 Signed-off-by: Vincent Biret <vibiret@microsoft.com> Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Copilot

🟡 Changes recommended

Error wrapping hides native failure details and can cause misclassification, while an integration test fails without a built Python guest and Rust cancellation documentation omits pending-cancellation behavior.

3 open issues

1 resolved since last review

Avoid requiring kill() to return True for interruption

_{Review effort: Max}

jsturtevant · 2026-11-06T17:22:06Z

Fixed the CI lint failure in a285a01 with a fixup for a6042a2. Removed the redundant anyhow error conversion; the source message and typed cause are unchanged. Both failing lint commands now pass locally. Self-review and adversarial review found no issues.

Refactor cancellation handling and error messaging in guest execution. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

baywet mentioned this pull request Nov 4, 2026

[BREAKING] .NET: fix(hyperlight): propagate cancellation and limit execution microsoft/agent-framework#10066

Draft

1 of 7 tasks

baywet force-pushed the fix/sandbox-interruption branch from 25eecb7 to 280a15c Compare November 4, 2026 00:37

baywet marked this pull request as ready for review November 4, 2026 12:05

jsturtevant mentioned this pull request Nov 5, 2026

fix: pin Node 26 LTS in build workflows #276

Merged

jsturtevant force-pushed the fix/sandbox-interruption branch from dfdc4bd to 664f74c Compare November 6, 2026 00:23

jsturtevant requested a balanced review from Copilot November 6, 2026 02:25

Copilot started reviewing on behalf of jsturtevant November 6, 2026 02:26

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Comment thread src/sdk/python/core/tests/test_interruption.py Resolved

jsturtevant force-pushed the fix/sandbox-interruption branch from 220d7bc to 310e46c Compare November 6, 2026 06:54

jsturtevant mentioned this pull request Nov 6, 2026

fix: download guest artifacts after Rust cache restore #278

Merged

baywet and others added 7 commits November 6, 2026 06:54

fix: support cancellation during sandbox execution

a6fb0a2

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d6807e7-70a6-4c43-8dc1-7bab20376cd4 Signed-off-by: Vincent Biret <vibiret@microsoft.com> Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Preserve typed native interruption and cancellation errors

7388b35

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Preserve execution outcomes when C# cancellation arrives late

9316fbe

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Expose cross-thread guest interruption in Python SDK

c06d2bb

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

fixup! Preserve typed native interruption and cancellation errors

1881ec1

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

fixup! Expose cross-thread guest interruption in Python SDK

daee729

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

fixup! Expose cross-thread guest interruption in Python SDK

b443b73

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

jsturtevant force-pushed the fix/sandbox-interruption branch from 5306728 to b443b73 Compare November 6, 2026 14:54

jsturtevant closed this Nov 6, 2026

jsturtevant reopened this Nov 6, 2026

jsturtevant requested a max review from Copilot November 6, 2026 15:03

Copilot started reviewing on behalf of jsturtevant November 6, 2026 15:03

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Comment thread src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/IntegrationTests.cs Resolved

Comment thread src/wasm_sandbox/src/lib.rs Resolved

Comment thread src/hyperlight_sandbox/src/lib.rs Resolved

baywet mentioned this pull request Nov 6, 2026

[BREAKING] Python: fix(hyperlight): interrupt guest execution and enforce time limits microsoft/agent-framework#10152

Draft

2 of 7 tasks

jsturtevant approved these changes Nov 6, 2026

View reviewed changes

jsturtevant and others added 2 commits November 6, 2026 09:44

Improve cancellation and error handling in integration tests

8d64395

Refactor cancellation handling and error messaging in guest execution. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

fixup! Improve cancellation and error handling in integration tests

7dca6ab

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

jsturtevant force-pushed the fix/sandbox-interruption branch from a285a01 to 7dca6ab Compare November 6, 2026 17:44

jsturtevant merged commit 93ff8ff into hyperlight-dev:main Nov 6, 2026
30 checks passed

baywet deleted the fix/sandbox-interruption branch November 6, 2026 18:34

Navigation Menu

Repository navigation

fix: support cancellation during sandbox execution - #275

fix: support cancellation during sandbox execution#275
jsturtevant merged 9 commits into
hyperlight-dev:mainfrom
baywet:fix/sandbox-interruption

baywet commented Nov 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsturtevant commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

baywet commented Nov 4, 2026

Summary

Validation

AI assistance

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsturtevant commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants