fix(gate): fire failure nudge only on machine-confirmed status - #15
fix(gate): fire failure nudge only on machine-confirmed status#15AnswerTheo wants to merge 1 commit into
Conversation
PostToolUse gate flagged 'observed a tool failure' on successful commands whose output merely contained a failure keyword (No such file or directory, failed, error:) — greps, log reads, and SSM runs that exit 0. The host Bash payload exposes no exit code, so exit_success() always fell through to a text regex. Split explicit_status() (machine-readable exit code / success bool) from the text guess, and gate detect_failure() on ground truth only. exit_success() keeps its text fallback for verification bookkeeping, so should_block_stop() is unaffected. Verified: benign exit-0 + 'No such file' and grep 'failed' no longer fire; real exit_code:1 (pytest) and Edit success:false still fire; test_gate.py (6/6) and test_gate_robustness.py (12/12) pass. Closes fivetaku#14 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Confirming this from a second environment — I hit the same bug today and opened #16 before finding this PR, which predates mine by two weeks and diagnoses it better. Closing mine in favour of this one. Two things worth adding from my side. Your scope is right and mine was too narrow. I only exempted content tools ( Offering a regression test, since this PR is code-only. Branch: 16 checks over
Red-proofed: 11 of the 16 fail against Take it or leave it — no attribution needed, just paste it in if it saves you time. |
Closes #14.
Problem
The
PostToolUseobservation gate firesfablize gate observed a tool failureon successful commands whose output merely contains a failure-looking word (No such file or directory,failed,error:,N errors). In practice this misfires constantly on exit-0 commands:grepresults, log/file reads, and remote (SSM/ssh) runs.Root cause
The host's Bash
PostToolUsepayload carries noexit_code/exitCode/returncode/status/successfield, soexit_success()always falls through toFAILURE_RE.search(text)— a pure text guess.detect_failure()then flags any exit-0 command whose stdout contains a failure keyword.Edit/Writeare affected too when the edited text itself contains such words.Fix
Split machine-readable status (
explicit_status()) from the text guess, and gate the failure nudge on ground truth only.exit_success()keeps its text fallback for verification-command bookkeeping, soshould_block_stop()is unaffected. If a future host exposes a real exit code,explicit_status()picks it up automatically.Verification
hooks/gate_post_tool.py:No such file or directory→ was firing, now{}grepoutput containingfailed/error:→ was firing, now{}exit_code: 1(pytest) → still fires ✅Editsuccess: false→ still fires ✅tests/test_gate.py(6/6, incl. S3 pytest-exit-1 → BLOCK) andtests/test_gate_robustness.py(12/12).