fix(sandbox): keep UnixLocal workspace-root removal off the event loop - #4941
fix(sandbox): keep UnixLocal workspace-root removal off the event loop#4941ranjan-del wants to merge 1 commit into
Conversation
UnixLocalSandboxClient.delete() removed the workspace root with an inline shutil.rmtree call. That removal walks the whole workspace tree, so it held the event loop for its full duration and no other task could advance. Route it through run_blocking_workspace_io, as rm(recursive=True), persist_workspace, and hydrate_workspace already do. The helper forwards positional arguments only, so the call now relies on shutil.rmtree's stdlib default ignore_errors=False, which is how the sibling rm(recursive=True) call site invokes it. One released behavior changes. For a valid UnixLocalSandboxSession whose manifest has no ephemeral mount targets, delete() previously had no suspension point and so returned the session; it can now raise CancelledError, because delivering cancellation requires one. The removal still runs to completion and the caller still waits for it, since the helper keeps the worker owned. A caller bounding the call with asyncio.wait_for now receives TimeoutError after that same wait. Sessions with ephemeral mounts already awaited in the unmount loop and could already raise, and the pre-existing TypeError guard for a foreign session type is unchanged. Add a regression test that pins loop responsiveness across the removal itself via an event handshake, so an await elsewhere in delete() cannot satisfy it.
seratch
left a comment
There was a problem hiding this comment.
Thanks for tracing this and documenting the cancellation behavior. The direct rmtree call does block the event loop, and reusing the existing helper is a focused approach.
Before proceeding, please provide a realistic workspace example showing meaningful delays to another coroutine during deletion, comparing v0.22.2 with this change without artificially delaying rmtree. The handshake test establishes scheduling behavior, while #4675's evidence covers archive creation and extraction. Deletion-specific impact would help justify this change, including the newly reachable cancellation path that skips resume-state serialization.
|
Fair ask. The handshake test only establishes scheduling, and #4675's numbers are about archive and extract, so here is a deletion-specific measurement. What variesOnly the line under test. I toggle it on disk between runs and restore it from git afterwards: I used this branch's parent as the baseline rather than checking out the
Workspace
MeasurementSession built through the public Five trials per cell, arms interleaved so neither systematically runs on a warmer machine. Medians. Apple M1, macOS 26.6.2, APFS, Python 3.13.14.
The With no delete in flight, worst lateness is under 1 ms on both arms, so what the table shows is the deletion and not the harness. CostDeletion itself gets slower: +1.2% at 8.7k files, +5.5% at 26k, +4.2% at 52k. Same worker doing the same unlinks either way, so that is the thread hop and the wait loop. Stating it rather than burying it. The fixture understates thisEach trial workspace is built with CancellationFired from a plain thread with
That is the delta I described in the PR body, measured rather than reasoned about. On the baseline the cancel cannot be delivered at all, because there is no suspension point to deliver it at. The removal completes on both arms, so nothing leaks either way. Resume stateDriving the real
So the skip is not new, any cleanup error already reaches it. What this PR adds is a way for cancellation to be that error. If you would rather it stayed unreachable, absorbing The harness is four short scripts. Happy to paste them or push them somewhere if you want to rerun any of this. |
Summary
This pull request moves the UnixLocal workspace-root removal off the event loop, completing the set of operations #4700 converted.
UnixLocalSandboxClient.delete()removed the workspace root with a directshutil.rmtreecall inside anasync def:Removing the root walks the whole workspace tree, so the loop was held for the full removal and no other task could advance.
rm(recursive=True),persist_workspace, andhydrate_workspacealready hand that class of work torun_blocking_workspace_io, whose docstring states the contract: it runs the function in a worker thread and keeps ownership until that worker finishes. Its module docstring explains why that matters over plainasyncio.to_thread, which does not stop its worker when the awaiting task is cancelled. #4700 introduced the helper and converted those three sites.git blameputs this line at2d665c9a6(2026-04-15) andgit log -Lreturns only that commit for it, so #4700 did not touch it. The rationale for preferring the helper over bareto_threadis in the #4700 description, which also records that it is why #4678 was closed.Scope note, since this is a
rmtree/archive-class fix rather than a sweep: other inline synchronous filesystem calls remain in this module, notably theshutil.copyfileobjinwriteand theos.scandirinls. Those are bounded by a caller-supplied payload and a single directory rather than by a recursive tree walk, and I have left them alone here.shutil.rmtreerelies on its stdlib defaultignore_errors=False, because the helper forwards positional arguments only. Errors are therefore still raised and still swallowed by the unchangedexceptclauses. When no cancellation is observed,task.result()re-raises the worker's original exception object, so the types reaching those clauses are the same; when a cancellation is observed the helper raises thatCancelledErrorinstead. This matches how the siblingrm(recursive=True)call site invokes it.Present in
v0.22.1, not only onmain.Released-behavior delta, disclosed
delete()can now raiseCancelledError. Scoped precisely, because it depends on the manifest:delete(), so it returned the sessionCancelledError; underasyncio.wait_forthe caller still waits for the full removal and then receivesTimeoutErrormount_entry.unmount(...), so cancellation could already be delivered before the removal was reachedThe removal always runs to completion, because the helper keeps the worker owned, so nothing leaks and the workspace is still removed. Delivering cancellation is impossible without a suspension point, so this is inherent to the fix. It is the same trade-off #4700 accepted for the three sibling sites. Note that
CancelledErrorderives fromBaseException, so neither the method's ownexcept Exceptionnor a caller'scontextlib.suppress(Exception)absorbs it.Worth flagging for the in-SDK path: the sole caller,
_SandboxSessionResources.cleanup, wraps the call inexcept BaseException, records it ascleanup_errorif no earlier error was recorded, and still runs_aclose_dependencies()in afinally, so dependency teardown stays complete. It then re-raises, and atSandboxRuntimeSessionManager.cleanupa non-Nonecleanup_errorskipsserialize_resume_state(). That skip is pre-existing for any cleanup error; what is new is that cancellation can now reach it from here, so a cancelled cleanup loses the serialized sandbox resume state. Happy to guard that separately if you would rather it were absorbed.Otherwise unchanged: the
workspace_root_ownedandunmount_failedearly returns and the unmount ordering are byte-identical tov0.22.1, and the same session is returned on every path that returns at all. The pre-existingTypeErrorguard for a foreign session type is untouched.Test plan
Added
test_client_delete_keeps_workspace_removal_off_the_event_loopintests/sandbox/test_unix_local.py. It builds the session through the publicclient.resume(state), replacesshutil.rmtreewith a wrapper that still performs the real removal, and uses an event handshake rather than wall-clock timing: the worker signals that the removal has begun and then waits for a loop-side observer to answer, so the assertion covers the removal itself rather than the wholedelete()call.That scoping matters. An earlier draft counted loop progress across the whole call, which an
awaitelsewhere indelete(), such as the ephemeral unmount loop, could satisfy on its own. Verified against both regressions:shutil.rmtreeassert [False] == [True]shutil.rmtreeplus an unrelatedawait asyncio.sleep(0.01)indelete()assert [False] == [True]The test also asserts the workspace root is really gone and that the same session is returned, so it pins the removal target, not just the dispatch mechanism.
Ownership-on-cancellation is not separately pinned here; it is a property of
run_blocking_workspace_ioand is already covered forhydrate_workspaceby the test #4700 added..agents/skills/code-change-verification/scripts/run.shcompletes with exit 0:make format,make lint(All checks passed!),make typecheck(mypy0 errors, pyrightno issues found in 310 source files), andmake tests(9539 passed, 29 skipped). The pre-existingtests/sandbox/test_runtime.py -k deletetests also pass unchanged.Issue number
No existing issue. Found while auditing the module for remaining instances of the pattern #4700 addressed. Happy to open one first if you would prefer the discussion to start there.
Checks
.agents/skills/code-change-verification/scripts/run.sh/reviewbefore submitting this PRNot using Codex, so the last item does not apply.