Skip to content

fix(gate): don't read file content as a tool failure - #16

Closed
simonbbby wants to merge 1 commit into
fivetaku:mainfrom
simonbbby:fix/content-tool-false-failures
Closed

fix(gate): don't read file content as a tool failure#16
simonbbby wants to merge 1 commit into
fivetaku:mainfrom
simonbbby:fix/content-tool-false-failures

Conversation

@simonbbby

Copy link
Copy Markdown

Editing a file whose content contains a failure word gets reported as a failed tool call.

detect_failure() text-greps the tool response against FAILURE_RE regardless of tool type. For Bash that response is a command log, so the heuristic is right. For Edit/Write the response is the file itself, so any of these trip it:

  • a doc with a ## Failed attempts section
  • a changelog line fixed the failure in the parser
  • a test fixture or note containing 3 errors or error: cannot find module

The agent then gets fablize gate observed a tool failure. Do not report completion until it is fixed... for an edit that succeeded. In one repo this fired on essentially every edit to a status doc that has a ## Failed attempts heading — enough that the agent started routinely disclaiming the gate output as spurious, which is the opposite of what the gate is for.

Fix

Content tools report failure structurally (success / ok / exit_code), never in prose, so they now consult only that signal:

  • new structural_success() reads explicit result fields only, never text
  • exit_success() delegates to it, then falls back to the text heuristic as before
  • detect_failure() returns early for CONTENT_TOOLS (Edit, Write, NotebookEdit, MultiEdit, Read, NotebookRead)

Bash behaviour is unchanged, including textual-only failures with no exit code. A genuinely failed Edit is still caught — via the structural signal rather than by accident.

Tests

New tests/test_content_tool_failures.py, following the existing standalone-script convention: 14 checks covering the false positives, structurally-failed content tools, and the unchanged Bash paths.

Red-proofed — against the unpatched parser, 7 of the 14 fail:

MISMATCH  expected_failure=False got=True  edit: '## Failed attempts' heading
MISMATCH  expected_failure=False got=True  edit: prose 'documents a failure mode'
MISMATCH  expected_failure=False got=True  edit: 'probe reports 1 error, 0 warnings'
MISMATCH  expected_failure=False got=True  edit: changelog 'fixed the failure'
MISMATCH  expected_failure=False got=True  edit: 'error: ' inside a code sample
MISMATCH  expected_failure=False got=True  write: same content via Write
MISMATCH  expected_failure=False got=True  read: same content via Read
RESULT: 7/14 mismatches

With the fix, all 7 existing suites plus the new one exit 0.

detect_failure() text-greps the tool response for failure words. For Bash
that response is a command log, so the heuristic is right. For Edit/Write it
is the file itself, so editing a doc containing 'Failed attempts', a
changelog line 'fixed the failure', or a fixture with '3 errors' was
reported as a failed tool call — and the agent was then told not to report
completion. In one repo this fired on essentially every edit to a status
doc that has a '## Failed attempts' section.

Content tools signal failure structurally (success / ok / exit_code), never
in prose, so they now consult only that signal:

- new structural_success() reads explicit result fields only
- exit_success() delegates to it, then falls back to the text heuristic
- detect_failure() returns early for CONTENT_TOOLS

Bash behaviour is unchanged, including textual-only failures with no exit
code. A genuinely failed Edit is still caught, via the structural signal.

tests/test_content_tool_failures.py covers it: 14 checks, and 7 of them fail
against the unpatched parser.
@simonbbby

Copy link
Copy Markdown
Author

Closing as a duplicate of #15, which predates this by two weeks and fixes the bug more completely. Also tracked as #11 and #14.

My version only exempted content tools (Edit/Write/…), on the assumption that for Bash the response really is a command log so the text heuristic belonged there. #14 shows that assumption is wrong: the Bash payload carries no exit_code, so the text grep is not a supplement to ground truth, it is the only signal — which is why it misfires on greps, log reads and any stdout containing "failed". #15 fires only on machine-confirmed failure for all tools, which is the correct scope.

The regression test from this PR has been adapted to #15 semantics and offered there: simonbbby:tests/nudge-misfire-regression.

@simonbbby simonbbby closed this Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant