Add crash-safe tool execution journaling and interrupted-turn recovery

[Charlie-Wang-03]

Summary

Moonshine can resume a session's message history, but it currently cannot distinguish a cleanly completed tool-bearing turn from a process interruption inside tool execution.

That creates a side-effect safety gap: a tool may have completed externally while its result or the enclosing turn was never durably completed. A resumed model can then request additional tools without knowing whether replay would duplicate prior external effects.

This PR adds a generic tool-execution journal with a deliberately conservative recovery contract:

1. Persist tool execution intent immediately before dispatch.
2. Persist a terminal ok / error marker before dispatching the next tool.
3. Treat process-level interruption after intent as completion-ambiguous.
4. Detect a prior tool-bearing turn that never reached durable turn_completed.
5. Fail closed on future tool dispatch rather than automatically replay potentially side-effecting operations.
6. Preserve the existing handle_function_calls result contract.

This does not claim exactly-once execution. The contract is instead:

Detect ambiguous execution state and never silently replay possible side effects.

Problem

Crash window A: interruption inside a multi-tool batch

tool A starts
→ side effect happens
→ tool A finishes
→ tool B starts
→ KeyboardInterrupt / process interruption

Previously, the batch could leave the normal result-persistence path without a generic durable execution-state contract.

The runtime could not reliably distinguish:

• a tool that never started,
• a tool that completed,
• and a tool whose completion became unknowable.

Crash window B: handler finishes before the turn commits

tool handler returns successfully
→ execution result exists
→ process dies before turn_completed
→ same session is resumed

A completed handler does not prove that the enclosing agent turn was durably completed.

If the resumed provider context does not reconstruct that exact interrupted tool trajectory, automatically allowing another tool request can duplicate an external side effect.

Design contract

The recovery policy is intentionally conservative:

durable intent
→ dispatch
→ durable terminal execution state
→ durable turn completion

If Moonshine cannot prove that this lifecycle completed cleanly, later tool dispatch for the same durable session is blocked.

The runtime therefore prefers a visible fail-closed state over silent replay of an operation that may already have produced an external side effect.

This PR deliberately does not introduce:

• exactly-once semantics,
• generic automatic retries,
• tool idempotency metadata,
• retry-safety declarations,
• or operation-specific compensation protocols.

Those would require a separate, stronger execution contract.

Implementation

agent_runtime/execution_journal.py

Adds a dedicated ToolExecutionJournal runtime component with four lifecycle events:

• tool_execution_started
• tool_execution_finished
• tool_execution_ambiguous
• tool_execution_blocked

The journal uses the existing session conversation-event store rather than introducing another persistence backend.

Large tool arguments and outputs are not duplicated into the journal. Bounded previews and SHA-256 fingerprints are stored instead.

Durable dispatch boundary

For each executable call:

tool_execution_started
→ registry.dispatch(...)
→ tool_execution_finished
→ next tool

A normal tool exception remains a terminal error and does not poison later dispatch.

For process-level interruption such as KeyboardInterrupt, Moonshine records the execution as ambiguous on a best-effort basis and re-raises the original interruption.

A hard process kill may prevent the explicit ambiguous event itself from being written, but the unmatched durable tool_execution_started record is sufficient for recovery detection after restart.

Interrupted-turn detection

Execution completion alone is not sufficient because the process may fail after the handler finishes but before the enclosing turn commits.

The journal therefore also derives a monotonic turn sequence from the existing:

• turn_started
• turn_completed

events.

Completed resumed turns are paired LIFO with starts. This prevents a later successfully completed turn from accidentally hiding an older interrupted tool-bearing turn and avoids relying on wall-clock timestamp precision.

Fail-closed recovery

If Moonshine detects either:

• an execution with no durable terminal marker,
• an explicitly ambiguous execution,
• or a prior tool-bearing turn that never completed,

future tool dispatch in that session is blocked.

The provider receives a structured blocked_interrupted_execution error instead of executing the handler.

A resumed turn can still answer without tools; only potentially unsafe additional dispatch is prevented.

Compatibility

handle_function_calls retains its existing result keys:

name
call_id
arguments
output
error

If no durable session_store / session_id is available, dispatch retains its legacy behavior.

The change is therefore scoped to durable-session recovery without changing the non-durable call contract.

Deterministic regression coverage

tests/test_interrupted_tool_recovery.py adds five offline tests requiring no real LLM or API:

1. A successfully completed first tool receives a durable terminal record before a later tool is interrupted.
2. A simulated hard crash leaving only tool_execution_started blocks dispatch after SessionStore restart.
3. A later completed resumed turn does not hide an older interrupted tool-bearing turn, including when timestamps are identical.
4. An ordinary tool RuntimeError is terminal and does not poison later tool dispatch.
5. Dispatch without durable session storage preserves the legacy result shape and behavior.

Validation on current upstream

PR #4 is refreshed onto current upstream main:

base: bb0e4aa025fdcc5297fff1025ab4876f120ed615
head: af181ed4395ce9901a0e59576fa91c9ec7cfa4c6

The refresh used a normal merge commit. No rebase, history rewrite, or force push was used.

Validation was run on GitHub Actions with Ubuntu 24.04 / Python 3.11.16.

Moonshine validation run:

https://github.com/Charlie-Wang-03/Moonshine/actions/runs/34313956509

Focused checks

• changed-file compilation: PASS
• interrupted-tool-recovery regression: 5/5 PASS
• merged PR Fix orphaned tool_calls when the tool-round limit is reached #1 provider/tool protocol regression: 14/14 PASS

This verifies that the execution journal coexists with the provider/tool-call protocol changes already merged through PR #1.

Current-main vs PR-branch full-suite A/B

Current main (bb0e4aa):

Ran 259 tests in 145.920s

FAILED (failures=18, errors=25, skipped=2)

This PR (af181ed):

Ran 264 tests in 146.168s

FAILED (failures=18, errors=25, skipped=2)

Therefore the branch adds exactly five passing deterministic recovery tests while preserving the same existing full-suite failure/error counts as current main.

No additional full-suite failure or error is introduced by this PR.

The existing 43 failure/error baseline is current upstream test/code drift being addressed separately in PR #3.

Downstream compatibility: Creative-Intelligence runner v2

The PR head was also validated against current DeepMathLLM/Creative-Intelligence main:

Creative-Intelligence main:
9301ad4f4f0a411e00d4b92091dd2cf6ab91faec

Moonshine PR #4 head:
af181ed4395ce9901a0e59576fa91c9ec7cfa4c6

The validation used Creative-Intelligence's deterministic real-runtime Moonshine integration suite.

Only the integration workflow was temporarily pinned to the PR #4 head; no Creative-Intelligence production logic was changed and no downstream upstream PR was created.

Downstream validation run:

https://github.com/Charlie-Wang-03/Creative-Intelligence/actions/runs/34313995187

Result:

Ran 7 tests in 0.560s

OK

The suite exercises the real Moonshine runtime while faking only provider responses and covers existing downstream contracts around:

• session persistence,
• stable session identity,
• Moonshine tool registration and dispatch,
• verification/tool-event persistence,
• restart/resume recovery,
• rejection of mismatched project/mode/agent identity,
• rejection of incomplete session identity,
• archive publication after interrupted execution,
• and idempotent publication of already-verified results.

No downstream compatibility bug was found.

Relationship to existing PRs

PR #1 — merged

PR #1 fixes provider-protocol validity around orphaned tool_calls when the tool-round limit is reached.

Its regression suite passes together with this PR:

14/14 PASS

There is no direct file overlap with this PR.

PR #2 — merged

PR #2 closes SQLite connections after store operations and is already part of the current upstream base.

No additional work from PR #2 is duplicated here.

PR #3 — still open

PR #3 addresses broader upstream test/code drift across research workflow, retrieval, tool exposure, prompts, session infrastructure, and related behavior.

Its current diff has no direct file overlap with this PR.

There is one complementary semantic area: PR #3 writes ordinary tool_result conversation events, while this PR records execution-lifecycle state:

tool_execution_started
tool_execution_finished
tool_execution_ambiguous
tool_execution_blocked

These serve different contracts:

• tool_result: result/trajectory traceability
• execution journal: crash/recovery safety

No PR #3 code is pulled into this PR.

Diff scope

The effective PR diff remains limited to:

• agent_runtime/execution_journal.py
• model_tools.py
• tests/test_interrupted_tool_recovery.py

No CI workflow, downstream production change, retry framework, or unrelated cleanup is included.

Downstream motivation

Creative-Intelligence already contains workload-specific recovery logic around persisted verifier results and archive publication.

The underlying question is generic:

After an interrupted Moonshine tool execution or tool-bearing turn, can the runtime prove that replay is safe?

That contract applies to any Moonshine workflow, so recovery belongs in the Moonshine runtime rather than in downstream-specific workarounds.

Limitations

This PR deliberately does not attempt to provide exactly-once semantics or automatic recovery of arbitrary side-effecting tools.

Safe automatic replay would require an additional contract such as:

• explicit tool idempotency metadata,
• retry-safety declarations,
• operation-specific reconciliation,
• or compensation/recovery protocols.

The present change establishes the prerequisite runtime invariant:

Ambiguous execution is durable, observable, and fail-closed.