Skip to content

gaia2-cli: Time scenarios end the turn only on the agent's real reply - #121

Draft
johnh2o2 wants to merge 1 commit into
facebookresearch:mainfrom
johnh2o2:fix/gaia2-time-turn-ends
Draft

johnh2o2 wants to merge 1 commit into
facebookresearch:mainfrom
johnh2o2:fix/gaia2-time-turn-ends

Conversation

@johnh2o2

@johnh2o2 johnh2o2 commented Nov 9, 2026 •

Copy link
Copy Markdown

Summary

Time scenarios score near zero on OpenClaw for two harness reasons. Like native ARE, the judge grades each turn only on the agent events before its send_message_to_user, and eventd ends the episode after nb_turns of them.

  1. Fake turn ends. With messages.queue.mode: "steer", a notification sent while the agent works is folded into the working run, and OpenClaw ends the notification's own run at once with an empty final, which the adapter wrote as a turn end. In scenario_universe_22_bv95nt (save any apartment whose price changes within 4 minutes), a price notification at +30 s closed the turn 70 ms later, so the save_apartment calls from +55 s went ungraded. The adapter now drops the empty final of a message sent while one of its runs was unfinished, if it arrives within 2 s (observed: 9-367 ms); a real run calls the model first.
  2. Early real replies. AGENTS_TEMPLATE.md says to reply now and let a notification bring the agent back, which ends a Time turn before its timed steps. Scenarios tagged Time now get a short section: reply early only where the user asks to be told something before a timed step (the 63 of 160 Time scenarios with 2-3 turns), otherwise once the timed parts are done or can no longer happen, waiting with sleep 15 and date.

The prompt section applies to every gaia2-cli scaffold; the adapter change is OpenClaw-only, and hermes and mini-swe were not re-run. Known gap: runs that OpenClaw starts by itself for queued messages are not tracked, so a notification steered into one could still end the turn; no recorded run showed this.

Depends on #112: without it, date in the OpenClaw shell prints host time rather than scenario time.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • Test addition or improvement

Testing Strategy

  • 4 adapter and 3 prompt unit tests; the make test suites pass (cli + runner 648, openclaw 55, hermes 17, node).
  • Prompts of 79,792 scenario files: 70,294 non-Time byte-identical, 9,498 Time differ only by the new section.
  • Adapter over recorded logs: 0 drops in 1,701 non-Time episodes, exactly the 1,051 fake boundaries in 1,036 Time episodes.
  • End to end (OpenClaw, Gemini 3.1 Pro, English Time scenarios, same judge): before, 60 fake turn ends in 18 of 30 episodes, reply count == nb_turns in 4/30, ungraded writes in 20/30; after, 0 fake turn ends, nb_turns matched in 7/10, ungraded writes in 1/10. Pass rate stays low (0/10), now from ordinary task errors.
  • Oracle replay of 5 Time scenarios through eventd and the judge passes 5/5 on both images (unchanged, as expected).

AI Usage Declaration

Written with an AI coding agent (Claude) under my direction; I reviewed the diff, tests and verification data.

Time scenarios score near zero on the OpenClaw scaffold, for two reasons
that come from the harness rather than the model.

The judge, like native ARE, grades only the agent events before each
send_message_to_user, and eventd ends the episode once their count reaches
nb_turns. The adapter writes one for every final of a run chat.send
started. With messages.queue.mode "steer", a notification that arrives
while the agent is working is folded into the working run, and OpenClaw
ends the notification's own run at once with an empty final. That final
became a turn boundary 9-367 ms after the notification, and everything the
agent did afterwards went ungraded. The adapter now drops the empty final
of a message it sent while one of its runs was unfinished, if it arrives
within 2 s; a run that does real work calls the model first.

The prompt also tells the agent to respond now and let a notification
bring it back, so in a Time scenario its real reply ends the turn before
the timed steps happen. Scenarios tagged "Time" now get a short section:
reply early only where the user asks to be told something before a timed
step, otherwise reply once the timed and conditional parts are done or can
no longer happen, and wait with short sleeps.

Prompts of all other scenarios are unchanged, and in recorded runs of the
other categories no notification ever arrived while a run was working.
@meta-cla meta-cla Bot added the cla signed label Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant