You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Problemcomputer_action with {"type": "type"} fails verification on apps that don't render the raw input text (Windows Calculator, terminals with custom UIs, games, etc.), and the failure aborts the remaining actions in the batch — so a trailing press ENTER/= is silently dropped.Reproduction (Windows Calculator, standard mode):1. computer_app(launch, "计算器")2. computer_state → get revision3. computer_action with [{"type":"type","text":"1024*768"},{"type":"press","keys":["ENTER"]}]4. Result: status: error, code: input_not_verified; ENTER never executed.The calculator renders the expression as 1024 × 768 while verification searches the window OCR for the literal string 1024*768 — a match is impossible by construction.Observed side effects:- The automatic clipboard fallback (paste_unicode_text, i.e. Ctrl+A + Ctrl+V) fires even though Calculator does not support pasting expressions; in our environment it additionally surfaced WinError -2147221036 (CLIPBRD_E_CANT_OPEN) from CloseClipboard.- Some runs left a bogus history entry (1024×1024=1,048,576) in the calculator — consistent with the unreliable send_input + clipboard fallback + aborted-batch combination.## Root cause_verify_typed_text() in reasonix_computer_use/domain_tools.py verifies typed input by looking for the literal input string in the window OCR (find_text). For apps that display the input in a different format (operator symbols, digit grouping, localization), the literal string never appears, so verification always fails. On failure, computer_action returns input_not_verified and stops the rest of the batch (~line 1199), swallowing subsequent actions like ENTER.## Suggested fix (locally verified)1. Semantic-change verification — capture the window OCR text before type; after the input, treat it as verified if the OCR content changed (keep the existing exact-match as a fast path).2. Don't abort the batch when input cannot be verified — downgrade to a warnings field on the result and let subsequent actions run; a genuinely failed input will surface via the final state/UIA verification anyway.3. (optional) Only attempt the clipboard-paste fallback when the window exposes Edit/Document controls.With fixes 1+2 applied locally, type "1024*768" + press ENTER on Calculator completes in one batch: status: ok, warnings: [], displayed expression 1024 × 768 =, result 786,432 correct. Plugin test suite: 106 passed / 1 failed (the single failure is a pre-existing environment-only failure, unrelated to this change).## Environment- reasonix-computer-use v0.8.0-alpha.12- Windows 11, Windows Calculator (UWP)
Problem
computer_actionwith{"type": "type"}fails verification on apps that don't render the raw input text (Windows Calculator, terminals with custom UIs, games, etc.), and the failure aborts the remaining actions in the batch — so a trailingpress ENTER/=is silently dropped.Reproduction (Windows Calculator, standard mode):1.computer_app(launch, "计算器")2.computer_state→ getrevision3.computer_actionwith[{"type":"type","text":"1024*768"},{"type":"press","keys":["ENTER"]}]4. Result:status: error, code: input_not_verified; ENTER never executed.The calculator renders the expression as1024 × 768while verification searches the window OCR for the literal string1024*768— a match is impossible by construction.Observed side effects:- The automatic clipboard fallback (paste_unicode_text, i.e.Ctrl+A+Ctrl+V) fires even though Calculator does not support pasting expressions; in our environment it additionally surfacedWinError -2147221036(CLIPBRD_E_CANT_OPEN) fromCloseClipboard.- Some runs left a bogus history entry (1024×1024=1,048,576) in the calculator — consistent with the unreliable send_input + clipboard fallback + aborted-batch combination.## Root cause_verify_typed_text()inreasonix_computer_use/domain_tools.pyverifies typed input by looking for the literal input string in the window OCR (find_text). For apps that display the input in a different format (operator symbols, digit grouping, localization), the literal string never appears, so verification always fails. On failure,computer_actionreturnsinput_not_verifiedand stops the rest of the batch (~line 1199), swallowing subsequent actions likeENTER.## Suggested fix (locally verified)1. Semantic-change verification — capture the window OCR text beforetype; after the input, treat it as verified if the OCR content changed (keep the existing exact-match as a fast path).2. Don't abort the batch when input cannot be verified — downgrade to awarningsfield on the result and let subsequent actions run; a genuinely failed input will surface via the final state/UIA verification anyway.3. (optional) Only attempt the clipboard-paste fallback when the window exposesEdit/Documentcontrols.With fixes 1+2 applied locally,type "1024*768"+press ENTERon Calculator completes in one batch:status: ok,warnings: [], displayed expression1024 × 768 =, result786,432correct. Plugin test suite: 106 passed / 1 failed (the single failure is a pre-existing environment-only failure, unrelated to this change).## Environment- reasonix-computer-use v0.8.0-alpha.12- Windows 11, Windows Calculator (UWP)