Repository navigation
Conversation
The initial USER send_message_to_agent event may carry an event-level
"media" list (a sibling of "action") of {asset_id, mime_type, sha256}
refs. ScenarioLoader.task_media exposes it; malformed entries raise
ValueError naming the event. Text-only scenarios are unchanged.
When the initial user turn carries media, gaia2-eventd POSTs
{"message": text, "media": [...]} to /send_user_message; every other
body stays exactly {"message": text}.
handle_notify passes a body with a "media" list to an optional
send_turn callback; a backend without one (Hermes, mini-swe) answers 400
instead of dropping the audio. Legacy {"message": text} bodies take the
unchanged send_message path.
OpenClaw 2026.4.1 drops non-image chat.send attachments, so its send_turn
appends one [[gaia2-media:<asset_id>]] marker per asset to the message
text, for the launch.mjs fetch shim to resolve.
When $GAIA2_MEDIA_ROOT/manifest.json exists, launch.mjs replaces each
[[gaia2-media:<asset_id>]] marker in user messages of chat-completions
requests with an input_audio part. The asset must be in the manifest,
resolve inside the root and match its sha256; otherwise the request is
refused with HTTP 400 without calling upstream. Bodies without markers
pass through unchanged. The trace log records {asset_id, sha256} per
delivered asset instead of the base64 audio. Helpers live in
launch_request_utils.mjs, which the Dockerfile already copies.
When GAIA2_JUDGE_CONTEXT names a judge context file
({"version": 1, "turns": {"<USER event id>": {"text": ...}}}),
gaia2-eventd passes its texts to extract_oracle_data, which uses them as
the judge's task text for the listed USER turns (e.g. the original prompt
of a turn the agent received as audio or as an ASR transcript). The text
sent to the agent is unchanged; with no file the judge is unchanged.
OpenClaw 2026.4.1 (pi-ai 0.64.0 buildBaseOptions) requests min(model.maxTokens, 32000) output tokens on every call, and vLLM rejects any prompt longer than max_model_len minus the requested output. With a 65,536-token server that leaves ~33.5K tokens of prompt, while OpenClaw only compacts near its context window. OPENCLAW_MAX_TOKENS (unset -> unchanged 65,535) sets the openai-compat model's maxTokens so callers can make contextTokens + maxTokens fit the served window.
An empty "media" list is a malformed media turn (400), and a judge context file that exists but cannot be read or parsed raises instead of silently falling back to the agent-facing text.
gaia2-init-entrypoint.sh starts the agent with `su -s /usr/bin/bash agent`,
and su sets SHELL=/usr/bin/bash. OpenClaw's exec tool runs "$SHELL -c
<command>" (getShellConfig), so agent commands ran the real bash and
never went through /home/agent/bin/bash, the wrapper that preloads
libfaketime when /tmp/faketime.rc exists. OpenClaw also strips LD_* from
the exec env, so the gateway's own LD_PRELOAD does not reach them either.
Result: `date` in the agent shell printed the host date, not the
scenario's simulated date, and relative-date reasoning ("last Friday")
went wrong for every model.
Export SHELL=/home/agent/bin/bash in the agent env file, which is sourced
after su. The Gaia2 CLIs read /tmp/faketime.rc themselves and are
unaffected.
…enClaw completionToSSE forwards a non-streamed chat-completions message.reasoning to OpenClaw as delta.reasoning. pi-ai (openai-completions) stores it as a thinking block, re-sends it on every later request as `reasoning`, and OpenClaw's tool-result context guard (estimateMessageChars) and compaction estimate count it. With a chat template that never renders past reasoning (Qwen3 thinking, multi-step tool loop), the guard sees up to ~10x the real prompt, blanks tool results and throws "Preemptive context overflow" far below the served context. GAIA2_OPENCLAW_FORWARD_REASONING=0 (false/off/no) drops the reasoning chunk before it reaches OpenClaw. Unset keeps today's behaviour. The raw response in the trace still carries the reasoning.
…n empty The inlined HEARTBEAT.md tells the model to reply HEARTBEAT_OK when nothing needs attention. OpenClaw strips NO_REPLY to an empty reply but passes HEARTBEAT_OK through, and on_backend_response dropped it for every run, so a chat.send run that ended with it never wrote a send_message_to_user and eventd idled out (13 silence-variant hangs in pilot30). For runs in _own_run_ids the ack now closes the turn with an empty boundary, exactly as a stripped NO_REPLY does; acks from runs the adapter did not start (heartbeats, memory flushes) are still dropped.
The LLM retry loop in launch.mjs replaced init.signal with its own per-attempt timeout controller, so a request OpenClaw had already abandoned (llm.idleTimeoutSeconds, a turn abort) kept running and was retried up to 6 x 180 s in the background. Each attempt now aborts when either the caller's signal or the 180 s timeout fires; after a caller abort the loop neither retries nor sleeps out the backoff, and rethrows the caller's reason. The trace records such attempts as type caller_aborted. The timeout and retry policy are otherwise unchanged. test_launch_fetch_abort.mjs runs launch.mjs itself (image-only modules stubbed) and fails on the previous version.
The adapter's log (which runs it started, which finals it dropped or buffered, which turn boundaries it wrote) was the one container log the runner did not copy out, so a scenario that hung without a turn end could not be diagnosed afterwards. It is now extracted with the others, best effort like them.
gaia2-eventd logs one line per scenario event at startup (id, type AGENT, deps/succs) plus "Oracle data: N turns, M total oracle events", and the judge appends every judged turn's oracle events (tool, args) to judgments.jsonl. All three destinations were readable by the agent user: - /tmp/gaia2-eventd.log (eventd's console log) is created by touch as root with no chmod since a9e6435, so 0644 under podman and 0664 + gaia2 group under the FEH apptainer wrapper. - {state_dir}/eventd.log and {state_dir}/judgments.jsonl (a9e6435) were protected only by /var/gaia2/state being gaia2 0700. The FEH apptainer wrapper (FEH 8df1202, clause 2) opens the state dir to the agent through the gaia2 group, because the setuid gaia2-exec is inert there and the agent CLIs must write events.jsonl. In adaptability scenarios with 2-3 oracle turns, judgments.jsonl gains a record (oracle tool calls and args) after each judged turn, while the agent is still working on the next one. gaia2-eventd gets --log-dir for eventd.log and judgments.jsonl (default --state-dir, so other callers are unchanged). The OpenClaw entrypoint points it at /var/gaia2/logs, created gaia2:gaia2 0700, and makes /tmp/gaia2-eventd.log gaia2 0600 before eventd starts appending. events.jsonl, daemon_status.json and the other agent-facing files stay in the state dir. The runner reads judgments from /var/gaia2/logs, falling back to the state dir for older images, and still extracts both logs as root.
Under the FEH apptainer wrapper the state dir is group-writable by the agent (setgid, no sticky bit), so the agent could replace daemon_status.json and the runner would record a forged success. eventd now also writes the status copy into --log-dir (gaia2-only 0700 when the entrypoint passes it), and the adapter /status and the runner read the verdict from there, falling back to the state copy for older images. log_dir defaults to state_dir, so other callers are unchanged.
…e OpenClaw SSE shim launch.mjs completionToSSE dropped extra_content (Gemini-3's google thought_signature) when re-serializing tool_calls, so multi-turn tool use with a thinking model that signs its tool calls died: turn 1 returned a signed tool_call (200), but the turn-2 tool-result request echoed the tool_call WITHOUT the signature and the genai compat gateway rejected it (HTTP 400). launch.mjs retried only 429/5xx, so the agent gave up after one read-only exec. Capture extra_content into a module-level map (keyed by a normalized id, since OpenClaw core mutates the id) and re-attach it to outbound tool_calls on the follow-up request; forward extra_content in completionToSSE; and repair+retry a 400 that names a missing/invalid thought signature. Backends that do not populate extra_content are unchanged (map empty => no-op).
Author
|
Closing for now. This thought-signature round-trip is part of the OpenClaw gateway / audio-native enablement (input_audio injection + the OpenAI-compat SSE shim) that is not upstreamed yet, so it cannot be reviewed as an isolated change against main (its diff would carry the whole stage1 stack). The change is preserved on the fix/openclaw-gemini-thought-signature branch (commit f10b30a); we will repackage the gateway/audio enablement as a coherent, properly-isolated PR set once that feature stabilizes after the current benchmark run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change class: Harness compatibility — enables thinking-model (reasoning) agents via the OpenAI-compat gateway. Not an agent-exploit vector.
Summary
OpenClaw's
launch.mjsSSE shim droppedextra_content(the Gemini-3google.thought_signature) when re-serializing tool_calls, so a thinking model that signs its tool calls could not do multi-turn tool use through the OpenAI-compat gateway: turn 1 returned a signedtool_call(HTTP 200), but the turn-2 tool-result request echoed that tool_call without the signature and the genai compat gateway rejected it with HTTP 400.launch.mjsretried only 429/5xx, so the agent gave up after a single read-onlyexec.Fix
Mapcapturesextra_contentfrom responses (keyed by a normalized id, because OpenClaw core mutates the tool_call id) and re-attaches it to outboundtool_callson the follow-up request.completionToSSEforwardsextra_contenton the tool_call delta.extra_contentare unchanged (map stays empty → no-op).Impact
Lets reasoning/thinking models (e.g. Gemini-3 Pro/Flash via the Llama compat gateway, and any signed-tool-call backend) run multi-turn agent tasks. Root-caused from an end-to-end smoke where the audio pipeline worked but the model scored 0 with ~30-token no-op completions = this turn-2 400. Verified by code review +
node --check; the rebuilt-image smoke is the end-to-end confirmation.Shares
containers/openclaw/launch.mjswith the reasoning-context PR (sequence on merge). Part of the same isolated gaia2-cli fix set; draft, no reviewers yet.