fix: route delegated peers to their owning runtime - #482
Conversation
Resolve peer placement before creating an AgentBox, relay cross-runtime chat events through the management plane, and fail closed when the route or delegated session cannot be established.\n\nTested: npm test; npm run build\nConfidence: high\nScope-risk: moderate
jacoblee-io
left a comment
jacoblee-io
left a comment
There was a problem hiding this comment.
Reviewed the routing change. Direction looks right — making the placement decision before touching the local AgentBoxManager and failing closed is the correct shape, and reusing the reverse WS lane avoids exposing private Gateway addresses.
For the record on scope: this is not a regression. src/gateway/delegate-api.ts has exactly one prior commit (641b898a, "feat(delegation): siclaw-native agent-to-agent delegation"), and that first version already did deps.agentBoxManager.getOrCreate(peerAgentId) unconditionally. There has never been any Runtime-placement logic on the delegation path, and docs/design/coordinator-routing.md never states the single-Runtime assumption. So every multi-Runtime deployment has had this since delegation shipped.
Two things I think should be fixed before merge (details inline):
skipInitialPersistencehas no producer in this repo.server.tsconsumes it, but thedelegation.startpayload built here does not set it. The de-duplication the PR describes only happens if the management plane injects the flag while forwarding tochat.send.- The remote path can return a successful-but-empty result.
finalText/steps/artifactare rebuilt only from relayed events, so a dropped frame plus a deliveredprompt_doneyieldsok:true, status:"done", finalText: undefined, which the coordinator relays as a real answer.
Also: this adds a new cross-Runtime contract (delegation.resolveRoute / delegation.start / delegation.abort, the reverse delegation.event lane, and skipInitialPersistence) with no documentation change. Per CLAUDE.md's documentation rule these are contracts, not implementation details — worth a section in docs/design/coordinator-routing.md plus Change Impact Matrix rows for delegate-api.ts and frontend-ws-client.ts, including the deploy-ordering requirement.
Recover dropped relay output from durable history, fail empty completion, make timeout activity-based, prevent stopped turns from dispatching, and document the Runtime placement contract. Tests: npm test; npm run build; git diff --check
jacoblee-io
left a comment
jacoblee-io
left a comment
There was a problem hiding this comment.
Re-reviewed against 0c4570b1. All six points are addressed, several better than what I suggested — the idle watchdog instead of a total-duration cap, and the route.local !== (source === target) consistency assertion, are both improvements on the original comments.
Retracting my first P1. I claimed nothing sets skipInitialPersistence. I had only read this repo; the management plane injects it at its trust boundary. The source now sets it explicitly too, which is the right call — the flag is asserted on both sides rather than depending on one.
I also checked the two risks the new readback could have introduced, and both are clear:
recoverRemoteResult's boundary scan is safe:getMessagesreverses internally, solimit: 500is the most recent 500 rows, not the oldest.- No read-too-early race:
prompt_doneis emitted only afterconsumeAgentSseresolves, and every persistence flush inside it is awaited (sse-consumer.ts:1085). The assistant rows are committed before the terminal event can reach the source.
One finding left, inline.
Treat live relay frames as progress only and always build the final remote answer from persisted assistant rows, preventing partial frame loss from becoming a truncated success. Tests: npm test; npm run build
AgentBox starts a run before it acknowledges /api/prompt, so a lost or timed-out ack leaves a really-running turn behind a still-pending Gateway state, where prompt()'s rejection never reaches the post-accept compensation. The pending-only Stop branch skipped AgentBox abort entirely to avoid arming a pre-spawn latch the cancelled send would never consume, which left that turn running headless. Decide on the condition that actually governs the latch — whether the box holds the session — instead of on Gateway pending-vs-active state. A box that is running the turn has the session, so aborting it is both required and latch-free; a box that has nothing is left alone. The probe never falls back to getOrCreate: spawning a pod would manufacture the very box whose absence was the answer. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dary Stopping only on Stop left the larger leak open: AgentBox starts a run before acknowledging /api/prompt, so an ack lost during an ORDINARY send strands a turn with no consumer, and a Stop that probed just before the box created the session missed it too. Move the guarantee to the rejection path, where the dispatch outcome is genuinely unknown, and keep the Stop-time probe as the fast path that avoids waiting out the timeout. Both sites now share abortIfBoxHoldsSession: holding the session is what separates "a turn is running" from "aborting would arm a pre-spawn latch the next send would consume", and an unanswerable probe now leaves the box alone rather than forcing an abort that could plant that latch. Recovery also no longer caps a turn by its own length. Tool rows are messages, so a tool-heavy peer turn could bury its opening row past a single 500-row window and have its finished answer reported as unrecoverable; page backwards to the boundary instead, bounding total rows walked rather than one turn's size. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… window Three separate ways a delegation could still end wrong. A Stop the box never confirmed was answered ok:true, which tells the control plane to stop retrying and tear down its supervision. Probe and abort failures now surface, matching what the non-pending path already did; only "the box does not have this session" still counts as already stopped. A failed compensation on the prompt-rejection path is logged loudly instead, because there the original prompt failure is what the caller must see. Recovery paged by timestamp cursor, but created_at has one-second granularity, so a cursor set to a page's oldest row skipped every other row written in that same second — most of a busy turn. Read a growing window of the newest rows instead: one extra query per step, and nothing can be skipped. A terminal that was consumed and then re-sent (its ack lost) had no live consumer left and was rejected forever, which kept the sender's relay alive; that relay's idle expiry aborts by (agent, session) and would land on a new turn reusing the peer session. Recently settled delegations are now remembered and acknowledge a re-delivery. Also clamp the remote idle window to the control plane's relay lease — waiting past it cannot succeed, since the events have no route left. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Abandoning a turn's stream after a failed abort leaves the box running with nobody consuming it, and a single warning was the only trace. The box demonstrably holds the session at that point — it just accepted the prompt — so retrying is safe and cannot plant a pre-spawn latch. Retry briefly, then say so at error level rather than warn. The growing recovery window also doubled past its own ceiling on the last step (12800 → 25600 against a stated 20000); land exactly on it. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A session id names a CONVERSATION, and delegation deliberately reuses a peer session across turns, so "abort session S" is ambiguous the moment a turn ends. Every workaround in this area came from that: a probe that guessed whether a dispatch had landed, a pre-spawn latch that a later prompt could consume, a cold-start Stop skipped entirely to avoid arming it, and a control-plane abort arriving after its turn finished. Callers now name the turn they mean. AgentBox stores the turn a prompt is running and answers an abort for any other turn as already stopped; its pre-spawn latch records the turn that armed it and is consumed only by that turn's prompt. The Runtime mints an id per turn, forwards it, returns it in the chat.send ack, and tags every abort it sends. chat.abort accepts an optional turnId and ignores a stale one — the Stop button still sends none and still means "stop what is running". That removes rather than adds machinery: the listSessions probe is gone, and so is the special case that skipped a cold-start Stop, because a turn-scoped latch can only ever be consumed by the prompt being cancelled. Both fields are optional, so the deploy is additive in either order: an older box ignores turnId and keeps session-wide semantics, and a Runtime that sends none behaves exactly as before. The cold-spawn regression test is superseded by "lets a retry run after a cold-start Stop", which asserts the same no-dispatch and retry-runs properties plus the turn tagging. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reverse leg was made reliable, but the leg INTO the control plane stayed fire-and-forget. A human-facing turn survives losing its terminal because the frontend refetches; a delegated turn has a machine waiting on it and nobody to retry, so a terminal dropped there stranded the caller until its idle window elapsed — reported as a failure for a turn that had in fact finished. The target Runtime now also reports a delegated turn's terminal over an acknowledged RPC, retried a few times. The control plane treats it exactly as an observed terminal: supervision ends, and no abort is issued for a turn that is already over. It is idempotent by construction, because the event lane may deliver the same terminal and an unacknowledged attempt is retried. Only delegated turns pay the extra round-trip; ordinary chat is unchanged. A control plane without the method is not retried, and Portal-standalone answers it directly, so neither mode sees a delivery failure it cannot act on. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were decided over several review rounds and lived only in code comments. The doc now states what must hold: an abort names the turn it means because a peer session is reused across turns; a pre-spawn latch belongs to the turn that armed it; the Stop button still sends no turn and still means "stop what is running"; both legs of terminal delivery are acknowledged, and a re-delivered terminal must be acknowledged rather than rejected; durable readback widens a window instead of walking a second-granular timestamp cursor; and the idle window is bounded by the control plane's own lease. Also states the two rollout facts a future operator cannot infer: the control plane must precede the Runtime, while turn addressing spans the AgentBox image and degrades safely in either order — and that cross-Runtime delegation assumes a control plane whose Runtime connections all terminate in one process. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
All 14 threads are resolved. Summary of where this landed, since the shape of the fix changed materially after the first round. The root cause behind most of the findingsNearly every remaining issue traced to one thing: an abort was addressed by That single ambiguity produced the Aborts now name the turn they mean:
This removed machinery rather than adding it: the probe is gone, and so is the special case that skipped a cold-start Stop. Compensation on a failed prompt is now unconditional, because an abort naming a turn the box never started is a no-op it cannot confuse with a later one — which is what closes the lost-ack window in both the Stop and no-Stop cases. Both fields are optional, so this half of the rollout is additive in either order. Reliability of the result path
RolloutOne strict ordering and one that is not:
Cross-Runtime delegation also assumes a control plane whose Runtime connections all terminate in one process. Placement, command delivery and event subscription resolve against process-local connections, so a multi-replica control plane fails closed on a split pair rather than misrouting. Contracts are documented
One knowingly-deferred itemThe Verification
One test was replaced rather than dropped: |
…lost Three gaps in the first cut of turn addressing. A second send registers its turn before it can acquire the session lock, so two turns are live at once — one on the box, one queued. Remembering only the newest made Stop name the queued turn: the box rejected the mismatch, the consumer was torn down anyway, and the running turn continued headless. Track the live set and name all of it, snapshotted BEFORE breaking the consumer — reading it afterwards misses the very turn being stopped, because a settling turn removes itself. The turn id was learned from the chat.send acknowledgement, which left the one case that needs it most with nothing to name: a lost ack, whose compensation then fell back to stopping the session and could hit a later turn. chat.send now accepts a caller-supplied turnId and echoes it, so a supervisor fixes it before dispatch. A named pre-spawn latch is armed unconditionally. The "already has history on disk" guard existed only because a session-wide latch could cancel an unrelated later prompt; a turn-scoped one cannot, and skipping the guard is what lets a Stop on a reused session — which a delegated peer thread always is — arm anything. Also widen the terminal delivery retry budget past a WS reconnect, which cannot complete faster than its own backoff. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The turn-addressing section read as if one turn were live per session and as if the id could come from the acknowledgement. Both were the assumptions that had to be corrected in code: a second send makes two turns live, and an acknowledgement that never arrives is exactly the case a supervisor needs the id for. Also records why a named pre-spawn latch skips the on-disk-history guard, that a reported terminal is queued rather than acted on directly, and that the delivery retry budget has to outlast a WS reconnect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It read as if one turn were live per session and as if the id could come from the acknowledgement. Both were the assumptions that had to be corrected in code: a second send makes two turns live, and an acknowledgement that never arrives is exactly the case a supervisor needs the id for. Also records why a named pre-spawn latch skips the on-disk-history guard, that a reported terminal is queued rather than acted on directly, and that each delivery retry budget has to outlast a WS reconnect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shutdown and box removal end a turn on behalf of its own reporting path, which is then suppressed — so the one terminal a delegated caller cannot do without went out fire-and-forget on exactly the path where the transport is about to disappear. The caller was left to wait out its idle window and report a failure for a turn that had been interrupted, with the cause already known here. Delegated turns are now tracked so the supervisor can report their terminals the same acknowledged way, carrying the interruption cause it already names. Shutdown waits briefly for those deliveries before closing the connection they travel over — a live connection settles the first attempt in milliseconds, and a dead one must not hold the process open for the whole retry budget. A box removal does not wait: its connection stays up. The per-turn deliverer moved out of chat.send to be shared, unchanged otherwise. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Follow-up on the review of the turn-addressing work. Four P1s and three P2s, all addressed; head is P1 — two turns can be live at once
Turns are now tracked as a set per session, and a Stop that names none names all of them: the box stops the one it is running and answers the rest as already stopped. Falling back to a session-wide abort instead would re-arm a session-wide pre-spawn latch, which is what the turn scoping exists to prevent. Writing the test for this surfaced a second bug in the same code: P1 — a lost
|
Naming a turn on the wire is only half of it; the Runtime still cancelled by session, and three failures followed. An abort naming a QUEUED turn aborted the session's controllers, so the RUNNING turn's consumer was torn down while the box was told only about the queued one — leaving the running turn with nobody reading it. Controllers are now held per turn and only the named ones are cancelled. The supervisor reported every live turn as interrupted but cancelled only the streaming one, so a queued turn could still start after its caller had been told it was over. It now cancels every turn it reports, and the suppression of a turn's own terminal is per turn too — otherwise the other live turn still emitted a plain terminal, which reads as a turn that succeeded. AgentBox treated an abort for a not-currently-running turn as already stopped, but that turn's prompt may still be in flight or its session rebuilding, so the cancellation was simply lost. It now records a latch for the named turn, and latches are held per (session, turn): one slot per session let two turn-scoped latches overwrite each other, which is an ordinary sequence when one turn is cancelled while another runs. Also raises the terminal delivery budget past a full worst-case reconnect (the client's backoff caps at 30s plus up to 2s of jitter). Tests: npm test; npx tsc --noEmit. Both new Runtime tests were verified to fail against the previous behaviour rather than merely pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four, all in the seams rather than the idea. The choice between per-turn and session-wide cancellation was made from state read AFTER an awaited box lookup, so a turn settling in between dropped it into the session-wide branch — breaking the consumer of a successor that had started meanwhile. It is now taken from the snapshot and applied before the await. Shutdown only considered sessions with a registered consumer, so a turn still cold-starting was neither reported nor cancelled: its caller waited out an idle window while the box could still start it. It now covers every session with work of ours. Terminal delivery was awaited inside the turn, so a budget measured in reconnects held the session lock, the streaming registration and the supervisor's view of a live turn for up to a minute — blocking the next turn on that session and letting a SIGTERM re-report a turn that had already succeeded as interrupted. The turn now settles at once and deregisters itself; the delivery continues in a tracked set that shutdown flushes. A queued turn cancelled by the supervisor reaches the pre-stream catch, which emitted a second, PLAIN terminal — one without `aborted`, which reads as a turn that completed. That path now honours the same per-turn suppression. Tests: npm test; npx tsc --noEmit. The supervisor-cancellation test went through three signals before landing on one that actually fails without the fix: sampling after close(), and waiting on the session lock, both race the lock hand-off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cancelling on the Runtime side does not end a prompt: the consumer only notices its signal when the next event arrives, and a dropped SSE subscription merely unsubscribes. Boxes deliberately outlive a Runtime roll in K8s, so a turn already reported as `runtime_restart` kept executing there with nobody left to read it. The supervisor now asks the box to stop each DISPATCHED turn by name — `busyOn` is the record of that placement, so a turn that never reached a box is left alone. The terminals the supervisor starts were returned but never tracked, and the box-roll callback discards the return value. A shutdown right after a box removal therefore saw no live turn and no pending delivery, and closed the transport out from under a `box_rolled` terminal still being retried — losing the one report that tells the delegated caller its turn was interrupted rather than lost. Everything endTurns starts is now tracked centrally, and shutdown waits for it. Tests: npm test; npx tsc --noEmit. Both new tests were verified to fail against the previous behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A turn stays live until its consumer settles, and a real consumer settles only when its next event arrives — so a box removal followed by a shutdown reaches the same turn twice. Both passes reported it, putting two authoritative terminals with DIFFERENT causes in flight; whichever won the retry race named the cause, so a `box_rolled` interruption could be reported as `runtime_restart`. Each turn is now claimed before the first report. Later passes still cancel it and still stop its box, and the delivery they would otherwise have started is already tracked for shutdown to flush. Two things found while writing the test: The existing supervisor tests hid this window, because the mocked consumer settles the moment its signal aborts — which a real one does not. The mock can now model a consumer that ignores the abort, which is what makes the second pass reachable. A box removal was also asking the removed box to stop its turn: that endpoint is dead by definition, so the request only hung out its timeout and logged a misleading warning. Skipped for that reason. Tests: npm test; npx tsc --noEmit. The new test was verified to fail against the previous behaviour (two terminals instead of one). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit skipped it on the premise that the endpoint is already dead. That premise is wrong: the manager reports the interruption BEFORE asking the spawner to stop the box, and a failed stop keeps the drain mark and retries on a later tick. In that window the Runtime has already emitted an authoritative `box_rolled` terminal and dropped its consumer, while the prompt keeps running — and keeps producing tool side effects — on a box that is very much alive. The abort is issued for a box roll as well. A box that has in fact gone answers nothing, which is the outcome that was wanted, so a failure there is reported at log level rather than as a warning; a shutdown, where the box is expected to answer, still says so loudly. Tests: npm test; npx tsc --noEmit. The new test fails against the skip (its abort never arrives). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The same gap as the terminal deliveries, in the half I did not fix: the abort a supervisor issues was only pushed onto endTurns' return value, and the box-removal caller discards that. Once the local consumer settles the turn leaves liveTurnIds, so a SIGTERM an instant later saw nothing pending and the process exited with the abort in flight — and since K8s keeps the boxes, an already-reported turn would go on running there. Both kinds of outliving work now go through one central set that the bounded shutdown wait drains, so it no longer matters what a caller does with the return value. Two notes on the test: It asserts what the reviewer asked for — block abortSession, let the consumer settle, start close(), verify shutdown still waits — and fails against the untracked version, where close() completes immediately. It also needed its own session id. A test whose consumer never settles holds that session's turn lock for the remainder of the file, because sessionTurnLocks is process-wide; reusing the id made the new test hang in acquire() rather than test anything, and only in a full-file run. Tests: npm test; npx tsc --noEmit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shutdown looked once while every producer was still active: the command lane stays open so terminals can still be delivered, the HTTP servers still listen, and the manager's loops run until later. A turn admitted during that wait registered after the drain had looked, so it was neither reported nor cancelled — and the box, which K8s deliberately keeps, went on running it. chat.send now refuses once the fence is up, and a turn registers before the persistence awaits in its own handler, so a handler that got past the check is already visible to the drain instead of appearing while it was in the database. This REVERSES a documented decision (`server-shutdown-turns.test.ts`), which admitted such a turn precisely so it could report itself rather than hang. Admitting it is the worse half of that trade: a refusal is an explicit error the caller can act on by placing the turn on a Runtime that will still be there, which is more than an unsupervised turn offers. That test now asserts the refusal, and says why the trade flipped. If you would rather keep the old contract, the alternative is a repeated drain, and the fence should come back out. Also: the supervisor now asks a box to stop a given turn only once, so a box roll followed by a shutdown does not ask twice. A rescan loop was written first and then removed: under the fence nothing reachable turns up in a second pass — a turn finishing then finds itself already reported, and a box roll finds every live turn claimed and already asked — and I could not construct a test that distinguished it. Unexercised machinery on the shutdown path is its own hazard. Tests: npm test; npx tsc --noEmit. The fence half of the new test fails without it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two gaps in the previous commit. The fence gated `chat.send` only, while `POST /api/internal/delegate` starts AgentBox work of its own — a local peer prompt, or a remote relay — and kept being admitted during the drain. It now honours the same fence and answers 503, which is something a coordinator can act on; an unsupervised peer turn is not. The box abort marked a turn as asked before the request resolved and kept the mark on failure. The case that abort exists for is a box removal whose own `spawner.stop()` also failed, so treating a timed-out abort as done spent the only retry a shutdown could have made and left the prompt running on a box that was still there. A refused attempt now clears the mark. The invariant claim is narrowed to match the code rather than the other way round. One snapshot is sufficient for the turns this drain has ever known — the chat.send and delegation ingresses. The task coordinator's scheduled and fire-now jobs and capability runs keep their own AgentBox clients and are registered nowhere here, so they are neither fenced nor drained. That predates this drain, and closing it means giving those paths the same admission gate and registration — a change of its own, not a line in this one. Tests: npm test; npx tsc --noEmit. The delegation-ingress fence has a test through that ingress, and the abort-retry test fails when a refused attempt counts as asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Drawing the scope line explicitly, because this branch has been reviewed well past what it sets out to do. The PR's purpose is Runtime placement for delegated peers. Everything in the first review round was about that, and it has been settled for several rounds. What followed came from one decision — addressing aborts by turn rather than by session — which is not a delegation requirement but a pre-existing ambiguity that delegation exposed. Fixing it pulled in the shutdown path, and the shutdown path pulled in producers this branch does not otherwise touch. This branch stops here. Head is Two known gaps are deliberately left, both pre-existing and now documented in
Further findings adjacent to cancellation or shutdown are welcome as issues rather than as blockers here. A finding that this PR's own subject — placement, the two-leg terminal contract, or the durable result readback — is wrong is of course still a blocker. |
…self Both gaps were in the interleavings, and both of my previous tests avoided them. The delegation endpoint sampled the shutdown gate once, at entry, and then awaited the roster, the model binding, the route, session reuse, persistence, the session lock and — locally — a box spawn before dispatching. It could observe "not shutting down", pause in any of those, and dispatch after shutdown had taken its one look; and a delegation already under way was registered nowhere, so nothing wound it down. The gate is now re-read at the dispatch boundary, and the handler registers the same wind-down a client disconnect triggers, which shutdown runs before it takes stock so whatever that starts is waited on too. The box abort deduplicated on "asked", which a later caller cannot recover from: it finds the mark set while the first attempt is still outstanding, and by the time that attempt fails there is no later pass — and if the consumer settled meanwhile, the turn has left the bookkeeping entirely. Each attempt now retries itself inside the promise shutdown already awaits, and the mark is cleared only once every attempt has been refused. Tests: npm test; npx tsc --noEmit. Both new tests were verified against the previous behaviour: the dispatch-boundary one hangs waiting for a relay it should never have started, and the abort one never sees its second attempt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tions Three findings, one of which was making the whole suite exit non-zero. The registered wind-down scheduled its abort and returned void, so the promise shutdown tracked settled as soon as the request was dispatched — the transport could still close underneath it. It now returns the abort it issues. A client disconnect can fire and forget; a shutdown cannot. Cancellation while `delegation.start` was still pending rejected `remoteDone` before anything awaited it, an unhandled rejection that took the process exit code with it: `npm test` reported 245 files passed and exited 1. `remoteDone` is now marked handled at creation, which leaves the later await unchanged. The hook was registered at the top of the handler but released only by the inner `finally`, so every request rejected before dispatch — authorization, binding, route, parent, persistence — left a closure in the set for the lifetime of the Runtime, and shutdown would later invoke stale hooks. It is registered inside the guarded scope instead; nothing before that point has peer work to wind down, and the gate re-read covers that window. On my own process: I had been checking the suite's summary lines and not its exit code, which is why the unhandled rejection survived several rounds of me reporting the suite as clean. `npm test` now exits 0. Tests: npm test (exit 0); npx tsc --noEmit. The wind-down test fails against a fire-and-forget hook. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both halves of the finding hold. The wind-down caught a refused abort and resolved, so an immediately failing abort read as a completed wind-down. For a local peer there is no relay lease to fall back on and the box outlives the Runtime, so the prompt simply kept running. The attempt now retries a refusal, bounded, the same way the supervisor's own box abort does. And a disconnect started that attempt fire-and-forget, letting the handler reach its finally and unregister while the abort was still on the wire — a shutdown an instant later found neither a hook nor tracked work and was free to exit underneath it. The attempt is memoized, so a disconnect and a shutdown wait on the same one, and the hook stays registered until it settles. The test took three attempts to become discriminating, which is worth recording. Counting aborts on the remote path proved nothing, because the handler issues one of its own; asserting the hook before awaiting the handler proved nothing either, because it had not reached its finally yet. It now drives the LOCAL path, where the wind-down is the only source of abortSession, and asserts after the handler has settled. Both halves were verified to fail without their fix. Tests: npm test (exit 0); npx tsc --noEmit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2 participants
Root cause
Direct chat routes an agent to its configured Runtime, but coordinator delegation always called the source Runtime's local
AgentBoxManager. When the coordinator and peer belonged to different Runtimes, the peer was created in the coordinator's network environment and received the wrong MCP/tool surface.What changed
skipInitialPersistenceso the target Runtime cannot duplicate the delegated user rowSICLAW_REMOTE_DELEGATION_IDLE_TIMEOUTUser impact
A coordinator can delegate to a peer assigned to another Runtime without changing that peer's MCP bindings, cluster reachability, session history, live progress, or Stop behavior.
Validation
npm test— 245 test files passed; 5085 tests passed, 2 skippednpm run buildgit diff --checkgo test ./internal/siclaw/...go test -raceThe management-plane routing support should be deployed before rolling out this Runtime change; older management planes fail closed instead of creating a peer in the wrong Runtime.