Skip to content

fix(vendors): reconcile a verdict that contradicts its own reasoning - #57

Open
wazum wants to merge 1 commit into
nizos:mainfrom
wazum:fix/verdict-self-contradiction
Open

fix(vendors): reconcile a verdict that contradicts its own reasoning#57
wazum wants to merge 1 commit into
nizos:mainfrom
wazum:fix/verdict-self-contradiction

Conversation

@wazum

@wazum wazum commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Fixes #55.

Problem

enforceTdd's validator call runs with thinking: { type: 'disabled' } and a response schema that puts kind before reason. It's a plain-text completion, not tool-calling, so generation is strictly left-to-right — the model commits to kind before it's written a word of reason, because reason is the only place reasoning happens at all. On a borderline case it sometimes self-corrects mid-explanation, but by then kind is already locked in.

Two real captures (see #55 for the full write-up):

"...This is acceptable green. Correction: pass." — returned as violation.

"...Correcting: this is a single new test, which is a valid red step." — returned as violation.

A fix I tried and reverted

Reordering the schema to {"reason":...,"kind":...} seemed like the obvious fix. Measured it against a real scenario already in the test suite, 30 trials each side:

Schema Correct
original (kind first) 26/30 (87%)
reordered (reason first) 13/30 (43%)

More than triples the error rate on an unrelated judgment call — asking the model to justify itself with nothing committed yet seems to invite rationalization toward leniency. Not shipping that; details in #55.

Fix

reconcileSelfContradiction in to-verdict.ts, applied after schema validation. Never touches how the model reasons — just catches the contradiction after the fact. A violation whose reason ends with a correction marker ("Correction:", "Correcting:", "on second thought", "scratch that", "take that back", "never mind", "walk that back") that affirms pass gets flipped to pass, unless:

  • the correction still names a violation ("...actually a more serious violation"),
  • the affirming word is itself negated ("not valid"), or
  • a later correction in the same reason reverses an earlier one (uses the last marker).

Only flips toward pass, never the reverse — a wrongly-kept violation costs a retry, a wrongly-flipped pass would be a silent fail-open.

Deliberately not exhaustive: asked the model live for realistic self-correction phrasings that avoid the word "correction," got 8, caught 5 with the extended marker list (plus both real captures = 7 total). The 3 misses are principled: one has no stated conclusion on its own, one is a double-negative unsafe to parse with a simple regex, and one places the affirmation before the marker instead of after (checking backward risks false-flipping an ordinary violation that discusses a hypothetical "fine" case before explaining why it's actually blocked).

Tests

npm run checks clean — 550 tests, lint/format/typecheck pass.

  • Both real captures pinned verbatim.
  • Correction that still names a violation → stays violation.
  • Negated affirming word → stays violation.
  • Multiple corrections, later one reverses the earlier → uses the last, stays violation.
  • No correction language → unchanged.
  • pass verdicts never touched, regardless of reason text.
  • Case-insensitive ("CORRECTING" as well as "Correction").
  • Five additional real-world phrasings ("on second thought," "scratch that," "take that back," "never mind," "walk that back") gathered live and added one at a time via TDD — found and fixed a real bug along the way (the negation guard's word list included "never," which collided with the "never mind" marker itself).

enforceTdd's validator call runs with thinking disabled and a response
schema that puts kind before reason, so reason is the only place
reasoning happens. On a borderline case the model sometimes
self-corrects mid-explanation ("Correction: pass.") but kind is
already committed to violation by then, since JSON generation is
strictly left-to-right.

Reordering the schema (reason before kind) was tried and measured
live: it more than tripled the error rate on an unrelated real
judgment call (13% -> 57% over 30 trials each), so this reconciles
after the fact in to-verdict.ts instead. A violation whose reason ends
with a correction that affirms pass (and doesn't still name a
violation, and isn't itself negated) is flipped to pass. Only flips
toward pass, never the reverse, since a wrongly-kept violation costs a
retry but a wrongly-flipped pass is a silent fail-open.

Tests pin both real captured incidents verbatim, plus five additional
self-correction phrasings ("on second thought", "scratch that", "take
that back", "never mind", "walk that back") gathered by asking the
model for realistic alternatives to "correction" and checking which
ones the heuristic actually catches. Three plausible phrasings are
deliberately not handled: one has no stated conclusion, one is a
double-negative unsafe to parse with a simple regex, and one places
the affirmation before the marker instead of after, which would
require checking backward and risks false-flipping an ordinary
violation that discusses a hypothetical "fine" case before explaining
why it's actually blocked.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enforceTdd can block a write even when its own reasoning concludes pass

1 participant