Skip to content

Commit 93ff8ff

Browse files
baywetCopilotjsturtevantCopilot
authored
fix: support cancellation during sandbox execution (#275)
## Summary Related to #9. Expose independently owned native interruption handles through the Rust backend abstraction and .NET FFI. Add `Sandbox.Run(string, CancellationToken)` and make `RunAsync` forward cancellation into running guest execution, while preserving `Run(string)`. Cancellation uses a dedicated interruption worker, retries the transition into guest execution, and waits for native execution and cancellation work to finish before returning. The callback never accesses mutable sandbox state or acquires the execution gate. Existing snapshot/restore behavior permits reuse after interruption. There is no implicit SDK deadline: hosts can supply a timeout-backed `CancellationTokenSource`. Runtime initialization and host callbacks cannot be interrupted; host tools need their own cancellation policy. ## Validation Executed in Ubuntu x64 containers on a KVM-capable host using .NET SDK 10.0.403 and Rust 1.94. The net8.0 tests ran with `DOTNET_ROLL_FORWARD=Major`. - Before the implementation: `dotnet vstest src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/bin/Debug/net8.0/HyperlightSandbox.Tests.dll --TestCaseFilter:FullyQualifiedName~JavaScript_RunAsync_CancellationDuringExecution_StopsGuestAsync --Logger:"console;verbosity=normal"` **failed as expected** after a genuine warm-up and five seconds of guest execution: expected `OperationCanceledException`, but none was thrown. - `just wasm guest-build` — **passed**, building the actual Python/Wasm guest. - `just fmt` and `just fmt-check` — **passed**. - `just lint-rust` and `just dotnet lint-rust` — **passed**. - `dotnet build src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/HyperlightSandbox.Tests.csproj --nologo -p:TreatWarningsAsErrors=true` — **passed**, zero warnings/errors. - `just test-rust` — **passed**, 144 unit tests and 5 HTTP integration tests; one existing documentation example remains ignored. - `just dotnet test-rust` — **passed**, 78 FFI tests. - `dotnet vstest src/sdk/dotnet/core/Tests/HyperlightSandbox.Tests/bin/Debug/net8.0/HyperlightSandbox.Tests.dll --Logger:"console;verbosity=normal"` — **passed**, all 147 tests, including real JavaScript immediate/timer cancellation and recovery, pre-canceled execution, isolation from completed cancellation registrations, and Python/Wasm interruption followed by snapshot restoration. The first Rust HTTP run failed because a network-disabled container could not resolve the existing HTTPS test's destination. The full Rust suite passed when run with its required networking. FFI and .NET guest tests remained resource-limited and network-disabled. Windows execution, the Python SDK suite, and package publishing were not run locally; upstream CI remains pending. This PR must remain a draft. ## AI assistance Copilot assisted with implementation, tests, documentation, and this description. --------- Signed-off-by: Vincent Biret <vibiret@microsoft.com> Signed-off-by: James Sturtevant <jsturtevant@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: James Sturtevant <jsturtevant@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Copilot-Session: 3d6807e7-70a6-4c43-8dc1-7bab20376cd4
1 parent fa6ebec commit 93ff8ff

20 files changed

Lines changed: 736 additions & 105 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎src/hyperlight_sandbox/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ bytes = "1"
1616
cap-std = "4.0.3"
1717
http = "1"
1818
http-body-util = "0.1"
19+
hyperlight-host = { workspace = true, features = ["kvm", "mshv3", "hvf"] }
1920
hyper = { version = "1", features = ["server", "http1"] }
2021
hyper-util = { version = "0.1", features = ["tokio"] }
2122
log = "0.4"

‎src/hyperlight_sandbox/src/lib.rs‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ pub use cap_fs::{
1717
CapFs, DescriptorFlags, DescriptorStat, DescriptorType, Dir, DirPerms, FilePerms,
1818
FilesystemLimits, FsError, OpenFlags,
1919
};
20+
pub use hyperlight_host::HyperlightError;
2021
pub use network::{HttpMethod, MethodFilter, NetworkPermission, NetworkPermissions};
2122
use serde::{Deserialize, Serialize};
2223
pub use tools::{ArgType, ToolRegistry, ToolSchema};
@@ -106,6 +107,30 @@ impl<T> Snapshot<T> {
106107
// Guest traits
107108
// ---------------------------------------------------------------------------
108109

110+
/// An independently owned handle for interrupting guest execution.
111+
///
112+
/// Host callbacks are not interrupted. After an interruption, restore a
113+
/// snapshot before reusing the sandbox.
114+
#[derive(Clone)]
115+
pub struct InterruptHandle {
116+
inner: std::sync::Arc<dyn hyperlight_host::hypervisor::InterruptHandle>,
117+
}
118+
119+
impl InterruptHandle {
120+
pub fn new(inner: std::sync::Arc<dyn hyperlight_host::hypervisor::InterruptHandle>) -> Self {
121+
Self { inner }
122+
}
123+
124+
/// Request interruption, returning whether the vCPU was running.
125+
///
126+
/// This may block until the running vCPU stops. A false return during a
127+
/// host callback can still leave cancellation pending until it returns.
128+
/// A request just before guest entry can be missed; callers can retry.
129+
pub fn kill(&self) -> bool {
130+
self.inner.kill()
131+
}
132+
}
133+
109134
pub trait Guest: Sized {
110135
type Sandbox: GuestSandbox;
111136
fn build(
@@ -129,6 +154,10 @@ pub trait GuestSandbox: Send {
129154
fn snapshot(&mut self) -> Result<Snapshot<Self::SnapshotData>>;
130155
/// Restore a previously captured guest runtime state.
131156
fn restore(&mut self, snapshot: &Snapshot<Self::SnapshotData>) -> Result<()>;
157+
/// Obtain an independently owned interruption handle.
158+
fn interrupt_handle(&self) -> Result<InterruptHandle> {
159+
anyhow::bail!("guest backend does not support interruption")
160+
}
132161
}
133162

134163
// ---------------------------------------------------------------------------
@@ -173,6 +202,10 @@ impl<G: Guest> Sandbox<G> {
173202
self.inner.run(code)
174203
}
175204

205+
pub fn interrupt_handle(&self) -> Result<InterruptHandle> {
206+
self.inner.interrupt_handle()
207+
}
208+
176209
pub fn snapshot(&mut self) -> Result<Snapshot<<G::Sandbox as GuestSandbox>::SnapshotData>> {
177210
self.inner.snapshot()
178211
}
@@ -373,6 +406,12 @@ where
373406
mod tests {
374407
use super::*;
375408

409+
#[test]
410+
fn interrupt_handle_is_send_and_sync() {
411+
fn assert_send_sync<T: Send + Sync>() {}
412+
assert_send_sync::<InterruptHandle>();
413+
}
414+
376415
struct TestGuest;
377416

378417
struct TestGuestSandbox;

‎src/javascript_sandbox/src/lib.rs‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ use hyperlight_js::{
77
};
88
use hyperlight_sandbox::runtime::BlockOn;
99
use hyperlight_sandbox::{
10-
CapFs, ExecutionResult, Guest, GuestSandbox, NetworkPermissions, SandboxConfig, Snapshot,
11-
ToolRegistry, http as sandbox_http,
10+
CapFs, ExecutionResult, Guest, GuestSandbox, InterruptHandle, NetworkPermissions,
11+
SandboxConfig, Snapshot, ToolRegistry, http as sandbox_http,
1212
};
1313
use serde::Deserialize;
1414

@@ -393,6 +393,15 @@ impl JsGuestSandbox {
393393
exit_code: response.exit_code,
394394
})
395395
}
396+
Err(error)
397+
if matches!(
398+
error,
399+
hyperlight_sandbox::HyperlightError::ExecutionCanceledByHost()
400+
| hyperlight_sandbox::HyperlightError::PoisonedSandbox
401+
) =>
402+
{
403+
Err(error.into())
404+
}
396405
Err(error) => {
397406
let mut stderr = error.to_string();
398407
let stdout = match self.stdout.lock() {
@@ -415,6 +424,11 @@ impl JsGuestSandbox {
415424
impl GuestSandbox for JsGuestSandbox {
416425
type SnapshotData = JsSnapshot;
417426

427+
fn interrupt_handle(&self) -> Result<InterruptHandle> {
428+
let handle = self.sandbox.interrupt_handle();
429+
Ok(InterruptHandle::new(handle))
430+
}
431+
418432
fn run(&mut self, code: &str) -> Result<ExecutionResult> {
419433
self.run_impl(code)
420434
}

‎src/sdk/dotnet/README.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -266,7 +266,8 @@ just dotnet copilot-sdk-example # Run Copilot SDK example
266266
| Method | Description |
267267
|--------|-------------|
268268
| `Run(string code)` | Execute guest code, returns `ExecutionResult` |
269-
| `RunAsync(string, CancellationToken)` | Async version on thread pool |
269+
| `Run(string, CancellationToken)` | Execute guest code with interruption on cancellation |
270+
| `RunAsync(string, CancellationToken)` | Execute on the thread pool with interruption on cancellation |
270271
| `RegisterTool<TArgs,TResult>(name, handler)` | Register typed tool |
271272
| `RegisterTool(name, Func<string,string>)` | Register raw JSON tool |
272273
| `AllowDomain(target, methods?)` | Add domain to network allowlist |
@@ -275,6 +276,13 @@ just dotnet copilot-sdk-example # Run Copilot SDK example
275276
| `Snapshot()` | Capture sandbox state |
276277
| `Restore(snapshot)` | Restore to captured state |
277278

279+
Cancellation stops running guest execution on both backends and throws
280+
`OperationCanceledException` only after native execution and interruption callbacks
281+
have finished. Runtime initialization and host tool callbacks cannot be interrupted.
282+
Capture a snapshot before execution and restore it after cancellation before reusing
283+
the sandbox. Use a `CancellationTokenSource` with a finite timeout to bound guest
284+
execution; host tools must enforce their own cancellation policy.
285+
278286
### `ExecutionResult`
279287

280288
| Property | Type | Description |

‎src/sdk/dotnet/core/Api/Sandbox.cs‎

Lines changed: 68 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -319,6 +319,21 @@ public void RegisterToolAsync(string name, Func<string, Task<string>> handler)
319319
/// permissions). Subsequent calls reuse the initialized runtime.
320320
/// </remarks>
321321
public ExecutionResult Run(string code)
322+
=> Run(code, CancellationToken.None);
323+
324+
/// <summary>Executes guest code with cancellation of running guest execution.</summary>
325+
/// <param name="code">The code to execute.</param>
326+
/// <param name="cancellationToken">Token used to interrupt guest execution.</param>
327+
/// <returns>The execution result.</returns>
328+
/// <remarks>
329+
/// Runtime initialization and host callbacks cannot be interrupted.
330+
/// Cancellation waits for native execution to stop before returning.
331+
/// Completed results and unrelated failures are preserved if cancellation arrives late.
332+
/// Restore a snapshot before reusing an interrupted sandbox.
333+
/// </remarks>
334+
/// <exception cref="OperationCanceledException">Thrown before execution if cancellation is
335+
/// requested, or when native execution reports cancellation for this token.</exception>
336+
public ExecutionResult Run(string code, CancellationToken cancellationToken)
322337
{
323338
lock (_gate)
324339
{
@@ -333,9 +348,43 @@ public ExecutionResult Run(string code)
333348
nameof(code));
334349
}
335350

336-
var result = SafeNativeMethods.hyperlight_sandbox_run(_handle, code);
337-
GC.KeepAlive(this);
338-
result.ThrowIfError();
351+
cancellationToken.ThrowIfCancellationRequested();
352+
using var completed = new ManualResetEventSlim();
353+
using var interruptHandle = cancellationToken.CanBeCanceled ? GetInterruptHandle() : null;
354+
Task? interruption = null;
355+
using var registration = cancellationToken.Register(() =>
356+
{
357+
// kill() can miss an idle guest just before native execution starts.
358+
// Retry off the callback thread, which may itself be a host tool.
359+
interruption = Task.Factory.StartNew(() =>
360+
{
361+
while (!completed.IsSet)
362+
{
363+
if (SafeNativeMethods.hyperlight_sandbox_interrupt(interruptHandle!))
364+
{
365+
break;
366+
}
367+
368+
completed.Wait(10);
369+
}
370+
}, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
371+
});
372+
373+
FFIResult result;
374+
try
375+
{
376+
cancellationToken.ThrowIfCancellationRequested();
377+
result = SafeNativeMethods.hyperlight_sandbox_run(_handle, code);
378+
}
379+
finally
380+
{
381+
completed.Set();
382+
registration.Dispose();
383+
interruption?.GetAwaiter().GetResult();
384+
GC.KeepAlive(this);
385+
}
386+
387+
result.ThrowIfError(cancellationToken);
339388

340389
var json = FFIResult.StringFromPtr(result.value);
341390
if (json == null)
@@ -359,14 +408,25 @@ public ExecutionResult Run(string code)
359408
/// For ASP.NET Core, prefer calling <see cref="Run"/> directly.
360409
/// </summary>
361410
/// <remarks>
362-
/// The cancellation token prevents scheduling of the task but cannot
363-
/// cancel an in-progress FFI call. Once <see cref="Run"/> starts
364-
/// executing in the native layer, it will run to completion.
411+
/// Cancellation interrupts running guest execution, but not runtime
412+
/// initialization or host callbacks. Restore a snapshot after interruption.
365413
/// </remarks>
366414
/// <param name="code">The code to execute.</param>
367-
/// <param name="cancellationToken">Token to prevent scheduling (does not cancel in-progress execution).</param>
415+
/// <param name="cancellationToken">Token to prevent scheduling or interrupt running guest execution.</param>
368416
public Task<ExecutionResult> RunAsync(string code, CancellationToken cancellationToken = default)
369-
=> Task.Run(() => Run(code), cancellationToken);
417+
=> Task.Run(() => Run(code, cancellationToken), cancellationToken);
418+
419+
private InterruptSafeHandle GetInterruptHandle()
420+
{
421+
var result = SafeNativeMethods.hyperlight_sandbox_get_interrupt_handle(_handle);
422+
result.ThrowIfError();
423+
if (result.value == IntPtr.Zero)
424+
{
425+
throw new SandboxException("Native interruption handle was null.");
426+
}
427+
428+
return new InterruptSafeHandle(result.value);
429+
}
370430

371431
// -----------------------------------------------------------------------
372432
// Network

‎src/sdk/dotnet/core/PInvoke/FFIErrorCode.cs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,4 +33,7 @@ internal enum FFIErrorCode : uint
3333

3434
/// <summary>Filesystem I/O error.</summary>
3535
IoError = 7,
36+
37+
/// <summary>Guest execution was interrupted by the host.</summary>
38+
Cancelled = 8,
3639
}

‎src/sdk/dotnet/core/PInvoke/FFIResult.cs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ internal struct FFIResult
5959
/// Maps <see cref="FFIErrorCode"/> values to specific exception types
6060
/// for structured error handling in the API layer.
6161
/// </remarks>
62-
public void ThrowIfError()
62+
public void ThrowIfError(CancellationToken cancellationToken = default)
6363
{
6464
if (is_success)
6565
{
@@ -71,6 +71,8 @@ public void ThrowIfError()
7171

7272
throw code switch
7373
{
74+
FFIErrorCode.Cancelled when cancellationToken.IsCancellationRequested =>
75+
new OperationCanceledException(errorMessage, cancellationToken),
7476
FFIErrorCode.Timeout =>
7577
new SandboxTimeoutException(errorMessage),
7678
FFIErrorCode.Poisoned =>

‎src/sdk/dotnet/core/PInvoke/SafeHandles.cs‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,3 +103,25 @@ protected override bool ReleaseHandle()
103103
return true;
104104
}
105105
}
106+
107+
/// <summary>Owns a native interruption handle independently of sandbox state.</summary>
108+
internal sealed class InterruptSafeHandle : SafeHandle
109+
{
110+
public InterruptSafeHandle(IntPtr handle) : base(IntPtr.Zero, ownsHandle: true)
111+
{
112+
SetHandle(handle);
113+
}
114+
115+
public override bool IsInvalid => handle == IntPtr.Zero;
116+
117+
protected override bool ReleaseHandle()
118+
{
119+
IntPtr oldHandle = Interlocked.Exchange(ref handle, IntPtr.Zero);
120+
if (oldHandle != IntPtr.Zero)
121+
{
122+
SafeNativeMethods.hyperlight_sandbox_free_interrupt_handle(oldHandle);
123+
}
124+
125+
return true;
126+
}
127+
}

‎src/sdk/dotnet/core/PInvoke/SafeNativeMethods.cs‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,19 @@ internal static partial FFIResult hyperlight_sandbox_run(
214214
SandboxSafeHandle handle,
215215
[MarshalAs(UnmanagedType.LPUTF8Str)] string code);
216216

217+
[LibraryImport(LibName)]
218+
[UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
219+
internal static partial FFIResult hyperlight_sandbox_get_interrupt_handle(SandboxSafeHandle handle);
220+
221+
[LibraryImport(LibName)]
222+
[UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
223+
[return: MarshalAs(UnmanagedType.I1)]
224+
internal static partial bool hyperlight_sandbox_interrupt(InterruptSafeHandle handle);
225+
226+
[LibraryImport(LibName)]
227+
[UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
228+
internal static partial void hyperlight_sandbox_free_interrupt_handle(IntPtr handle);
229+
217230
// -----------------------------------------------------------------------
218231
// Filesystem
219232
// -----------------------------------------------------------------------

0 commit comments

Comments
 (0)