Durable resume may replay an approved external side effect after a crash

[NeilTy123]

Summary

OpenWorker currently decides whether a persisted tool call has completed by looking for a role="tool" message with the same tool_call_id.

There is a crash window where:

1. a side-effecting tool is approved;
2. the external system successfully applies the side effect (for example, an email is sent);
3. OpenWorker crashes before the corresponding tool result is durably checkpointed;
4. durable resume sees the tool call as unanswered;
5. the already-resolved approval is reused; and
6. the tool executes again.

The current behavior is therefore at-least-once recovery for this window. That is risky because the user-facing approval model implies that one approved action should not be silently performed twice.

I reproduced this deterministically on main at 01b6f83b3927e02912dda84bb392942c13ca70d1.

Reproduction

I used a fault-injection test with a custom requires_approval=True tool that increments an external delivery counter:

1. The provider emits a tool call with a fixed tool_call_id.
2. The approval is parked and persisted.
3. The approval is resolved and the tool returns successfully (deliveries == 1).
4. The test injects a crash after registry.execute(...) returns but before _record_result() can append/checkpoint the role="tool" result.
5. The persisted conversation contains no tool result, while the external delivery already happened.
6. The engine is dropped and durable resume runs.
7. Resume reuses the resolved approval and executes the same tool call again (deliveries == 2).

The test failed with the same duplicate delivery on the initial run and three repeated runs:

durable resume repeated an external side effect whose local result was unknown
assert ['delivered', 'delivered'] == ['delivered']

The existing durable-resume, Inbox, and standing-approval tests still pass (27 passed), so this crash window is not currently covered.

Current behavior

• TurnEngine._unanswered_trailing_tool_calls() treats a call as completed only when a persisted role="tool" result exists.
• InboxStore.add() reuses an Inbox item by (session_id, tool_call_id), including an already-resolved approval.
• TurnEngine._execute_sync() and _record_result() leave an unavoidable gap between the external call returning and its result entering local history.
• _CHECKPOINTS does not include tool_finished, so the durable window extends until iteration_end.
• The audit table does not currently record tool_call_id, a stable execution ID, an attempt number, or a recovery decision.

There is also a persistence asymmetry: approval state is made durable before execution, while successful tool state is not made durable until later in the iteration.

Desired invariants

For approved tools with external side effects:

1. A live turn and durable resume must refer to the same internal execution identity.
2. A known successful execution must never call the external tool again merely because the conversation result is missing.
3. An execution whose external outcome is unknown must not be silently replayed unless the connector explicitly supports a safe recovery strategy.
4. The internal execution/idempotency identity must not be exposed as a model-controlled tool argument.
5. Recovery decisions and attempts must be observable.

This issue does not propose that OpenWorker can unilaterally guarantee distributed exactly-once semantics across third-party systems.

Proposed direction (request for alignment)

Introduce a small durable tool-execution module, separate from TurnEngine, keyed by an internal stable execution ID derived from (session_id, tool_call_id).

A minimal persisted state model could be:

no record -> dispatched -> succeeded(result snapshot)

For tools that require durable side-effect protection:

• persist dispatched immediately before invoking the external adapter;
• persist succeeded plus the recoverable result immediately after it returns;
• append the normal role="tool" conversation result afterward.

On recovery:

• succeeded + missing conversation result: reconstruct the tool result from the durable execution record and do not invoke the adapter;
• dispatched: the outcome is unknown:
  • retry only when the tool explicitly declares an idempotent recovery policy and the same internal execution ID can be propagated to the adapter;
  • reconcile first when a connector explicitly supports reconciliation;
  • otherwise halt and surface recovery_required for human review (the safe default);
• no record: execute normally for calls created under the new protocol.

Recovery policy should be explicit OpenWorker-owned metadata (for example on ToolSpec), defaulting to manual recovery for protected external side effects. It should not be inferred from category, and the first PR would not need to modify aisuite.

Legacy sessions need a conservative rule: for sessions predating the ledger, a resolved approval with no tool result and no execution record should default to recovery_required rather than execution. "No record means safe to execute" only holds once the session is known—through a persisted protocol/version marker—to run under the new execution protocol.

Suggested first PR scope

If this direction matches the project roadmap, I propose keeping the first PR limited to:

• a durable execution record for approved external side-effecting tools;
• one stable internal execution ID across live execution and resume;
• reconstruction of known successful results;
• conservative recovery_required behavior for unknown non-idempotent outcomes;
• an explicit recovery-policy seam owned by OpenWorker;
• audit fields for tool_call_id, execution_id, attempt, and recovery_decision;
• deterministic crash-injection tests.

Connector-specific idempotency and reconciliation implementations can follow separately. Low-risk read-only tools and the existing parallel execution path should remain unchanged.

Fault-injection matrix

Crash point	Durable state	Expected recovery
Before the dispatch record	No record	Execute once only when the session is known to use the new protocol; otherwise require recovery review
After dispatched, before/while the external call returns	dispatched	Do not silently replay; apply the declared recovery policy
After external success, before succeeded is stored	dispatched	Same unknown-outcome policy; no blind retry
After succeeded, before role="tool"/iteration_end	succeeded	Reconstruct the result; do not call the external system
After the normal tool result checkpoint	Completed	Preserve the current no-replay behavior

Related work

• Session permanently fails with 400/2013 after interrupt — history needs a load-side self-heal #331 / PR fix: repair tool-call/result pairing on load to prevent unrecoverable 400 errors #350 address malformed tool-call/result pairing on load and propose synthesizing placeholders for dangling calls.
• PR fix(engine): stop replaying an abandoned partial turn on retry #363 explicitly preserves trailing tool calls that represent a legitimate durable suspension.
• Approving a mirrored plan/directory prompt from chat is recorded as a rejection (and consumes the prompt) #265 / PR fix: don't resolve plan/directory items from chat replies, route through resolve_inbox #300 fix chat-reply resolution of mirrored plan/directory prompts being recorded as rejections; they touch the Inbox resolve path but not the replay window described here.

This issue is different from pairing repair: a missing tool result may mean "waiting for approval", "never dispatched", or "external side effect may already have succeeded". A durable execution record could provide the missing distinction; a generic placeholder alone cannot identify the external outcome.

Questions for maintainers

1. Does a small durable execution record fit the intended architecture, or would you prefer a smaller conservative no-replay fix first?
2. Should the first PR stop at recovery_required for unknown outcomes, leaving connector idempotency/reconciliation for follow-ups?
3. Is an OpenWorker-owned recovery policy on ToolSpec the right seam, rather than extending aisuite metadata?
4. What user-facing recovery flow would you prefer for an unknown outcome: an Inbox item, a turn-ending event/notice, or another mechanism?
5. Is the proposed conservative legacy default—recovery_required for resolved approvals without a tool result or execution record—acceptable?

I am happy to prepare the fault-injection test and a focused PR after aligning on the recovery semantics and scope.