Repository navigation
Conversation
Time scenarios score near zero on the OpenClaw scaffold, for two reasons that come from the harness rather than the model. The judge, like native ARE, grades only the agent events before each send_message_to_user, and eventd ends the episode once their count reaches nb_turns. The adapter writes one for every final of a run chat.send started. With messages.queue.mode "steer", a notification that arrives while the agent is working is folded into the working run, and OpenClaw ends the notification's own run at once with an empty final. That final became a turn boundary 9-367 ms after the notification, and everything the agent did afterwards went ungraded. The adapter now drops the empty final of a message it sent while one of its runs was unfinished, if it arrives within 2 s; a run that does real work calls the model first. The prompt also tells the agent to respond now and let a notification bring it back, so in a Time scenario its real reply ends the turn before the timed steps happen. Scenarios tagged "Time" now get a short section: reply early only where the user asks to be told something before a timed step, otherwise reply once the timed and conditional parts are done or can no longer happen, and wait with short sleeps. Prompts of all other scenarios are unchanged, and in recorded runs of the other categories no notification ever arrived while a run was working.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Time scenarios score near zero on OpenClaw for two harness reasons. Like native ARE, the judge grades each turn only on the agent events before its
send_message_to_user, and eventd ends the episode afternb_turnsof them.messages.queue.mode: "steer", a notification sent while the agent works is folded into the working run, and OpenClaw ends the notification's own run at once with an empty final, which the adapter wrote as a turn end. Inscenario_universe_22_bv95nt(save any apartment whose price changes within 4 minutes), a price notification at +30 s closed the turn 70 ms later, so thesave_apartmentcalls from +55 s went ungraded. The adapter now drops the empty final of a message sent while one of its runs was unfinished, if it arrives within 2 s (observed: 9-367 ms); a real run calls the model first.AGENTS_TEMPLATE.mdsays to reply now and let a notification bring the agent back, which ends a Time turn before its timed steps. Scenarios taggedTimenow get a short section: reply early only where the user asks to be told something before a timed step (the 63 of 160 Time scenarios with 2-3 turns), otherwise once the timed parts are done or can no longer happen, waiting withsleep 15anddate.The prompt section applies to every gaia2-cli scaffold; the adapter change is OpenClaw-only, and hermes and mini-swe were not re-run. Known gap: runs that OpenClaw starts by itself for queued messages are not tracked, so a notification steered into one could still end the turn; no recorded run showed this.
Depends on #112: without it,
datein the OpenClaw shell prints host time rather than scenario time.Type of Change
Testing Strategy
make testsuites pass (cli + runner 648, openclaw 55, hermes 17, node).nb_turnsin 4/30, ungraded writes in 20/30; after, 0 fake turn ends,nb_turnsmatched in 7/10, ungraded writes in 1/10. Pass rate stays low (0/10), now from ordinary task errors.AI Usage Declaration
Written with an AI coding agent (Claude) under my direction; I reviewed the diff, tests and verification data.