Skip to content

fix(gate): fire failure nudge only on machine-confirmed status - #15

Open
AnswerTheo wants to merge 1 commit into
fivetaku:mainfrom
AnswerTheo:fix/gate-text-heuristic-false-positive
Open

fix(gate): fire failure nudge only on machine-confirmed status#15
AnswerTheo wants to merge 1 commit into
fivetaku:mainfrom
AnswerTheo:fix/gate-text-heuristic-false-positive

Conversation

@AnswerTheo

Copy link
Copy Markdown

Closes #14.

Problem

The PostToolUse observation gate fires fablize gate observed a tool failure on successful commands whose output merely contains a failure-looking word (No such file or directory, failed, error:, N errors). In practice this misfires constantly on exit-0 commands: grep results, log/file reads, and remote (SSM/ssh) runs.

Root cause

The host's Bash PostToolUse payload carries no exit_code/exitCode/returncode/status/success field, so exit_success() always falls through to FAILURE_RE.search(text) — a pure text guess. detect_failure() then flags any exit-0 command whose stdout contains a failure keyword. Edit/Write are affected too when the edited text itself contains such words.

Fix

Split machine-readable status (explicit_status()) from the text guess, and gate the failure nudge on ground truth only. exit_success() keeps its text fallback for verification-command bookkeeping, so should_block_stop() is unaffected. If a future host exposes a real exit code, explicit_status() picks it up automatically.

Verification

  • Before/after on hooks/gate_post_tool.py:
    • benign exit-0 + No such file or directory → was firing, now {}
    • grep output containing failed/error: → was firing, now {}
    • real failure exit_code: 1 (pytest) → still fires ✅
    • real Edit success: false → still fires ✅
  • Regression suites pass: tests/test_gate.py (6/6, incl. S3 pytest-exit-1 → BLOCK) and tests/test_gate_robustness.py (12/12).

PostToolUse gate flagged 'observed a tool failure' on successful
commands whose output merely contained a failure keyword (No such
file or directory, failed, error:) — greps, log reads, and SSM runs
that exit 0. The host Bash payload exposes no exit code, so
exit_success() always fell through to a text regex.

Split explicit_status() (machine-readable exit code / success bool)
from the text guess, and gate detect_failure() on ground truth only.
exit_success() keeps its text fallback for verification bookkeeping,
so should_block_stop() is unaffected.

Verified: benign exit-0 + 'No such file' and grep 'failed' no longer
fire; real exit_code:1 (pytest) and Edit success:false still fire;
test_gate.py (6/6) and test_gate_robustness.py (12/12) pass.

Closes fivetaku#14

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@simonbbby

Copy link
Copy Markdown

Confirming this from a second environment — I hit the same bug today and opened #16 before finding this PR, which predates mine by two weeks and diagnoses it better. Closing mine in favour of this one.

Two things worth adding from my side.

Your scope is right and mine was too narrow. I only exempted content tools (Edit/Write/…), reasoning that for Bash the response really is a command log so the heuristic belonged there. That was wrong for exactly the reason #14 documents: the Bash payload carries no exit_code at all, so the text grep is not a supplement to ground truth, it is the only signal. With my narrower fix applied the nudge kept firing all session on commands that exited 0 and merely printed the word "failed" — including my own test output. Your version is the one that actually stops it.

Offering a regression test, since this PR is code-only. Branch: simonbbby:tests/nudge-misfire-regression — a single tests-only commit, cherry-pickable onto this branch, following the existing standalone-script convention.

16 checks over detect_failure():

  • the everyday Bash misfires (grep output naming a missing file, a log line containing error:, stdout with the word failed, a traceback with no status)
  • the content-tool shape (a doc with a ## Failed attempts heading, a changelog line fixed the failure, a fixture with 3 errors)
  • and the cases that must keep firing: success: false, exit_code: 1

Red-proofed: 11 of the 16 fail against main today, all 16 pass with this PR applied, and the other 6 suites stay green.

Take it or leave it — no attribution needed, just paste it in if it saves you time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PostToolUse gate: false 'observed a tool failure' on successful commands (text-heuristic misfire)

2 participants