fix(vendors): reconcile a verdict that contradicts its own reasoning - #57
Open
wazum wants to merge 1 commit into
Open
fix(vendors): reconcile a verdict that contradicts its own reasoning#57wazum wants to merge 1 commit into
wazum wants to merge 1 commit into
Conversation
enforceTdd's validator call runs with thinking disabled and a response
schema that puts kind before reason, so reason is the only place
reasoning happens. On a borderline case the model sometimes
self-corrects mid-explanation ("Correction: pass.") but kind is
already committed to violation by then, since JSON generation is
strictly left-to-right.
Reordering the schema (reason before kind) was tried and measured
live: it more than tripled the error rate on an unrelated real
judgment call (13% -> 57% over 30 trials each), so this reconciles
after the fact in to-verdict.ts instead. A violation whose reason ends
with a correction that affirms pass (and doesn't still name a
violation, and isn't itself negated) is flipped to pass. Only flips
toward pass, never the reverse, since a wrongly-kept violation costs a
retry but a wrongly-flipped pass is a silent fail-open.
Tests pin both real captured incidents verbatim, plus five additional
self-correction phrasings ("on second thought", "scratch that", "take
that back", "never mind", "walk that back") gathered by asking the
model for realistic alternatives to "correction" and checking which
ones the heuristic actually catches. Three plausible phrasings are
deliberately not handled: one has no stated conclusion, one is a
double-negative unsafe to parse with a simple regex, and one places
the affirmation before the marker instead of after, which would
require checking backward and risks false-flipping an ordinary
violation that discusses a hypothetical "fine" case before explaining
why it's actually blocked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #55.
Problem
enforceTdd's validator call runs withthinking: { type: 'disabled' }and a response schema that putskindbeforereason. It's a plain-text completion, not tool-calling, so generation is strictly left-to-right — the model commits tokindbefore it's written a word ofreason, becausereasonis the only place reasoning happens at all. On a borderline case it sometimes self-corrects mid-explanation, but by thenkindis already locked in.Two real captures (see #55 for the full write-up):
A fix I tried and reverted
Reordering the schema to
{"reason":...,"kind":...}seemed like the obvious fix. Measured it against a real scenario already in the test suite, 30 trials each side:kindfirst)reasonfirst)More than triples the error rate on an unrelated judgment call — asking the model to justify itself with nothing committed yet seems to invite rationalization toward leniency. Not shipping that; details in #55.
Fix
reconcileSelfContradictioninto-verdict.ts, applied after schema validation. Never touches how the model reasons — just catches the contradiction after the fact. Aviolationwhosereasonends with a correction marker ("Correction:", "Correcting:", "on second thought", "scratch that", "take that back", "never mind", "walk that back") that affirms pass gets flipped topass, unless:Only flips toward
pass, never the reverse — a wrongly-kept violation costs a retry, a wrongly-flipped pass would be a silent fail-open.Deliberately not exhaustive: asked the model live for realistic self-correction phrasings that avoid the word "correction," got 8, caught 5 with the extended marker list (plus both real captures = 7 total). The 3 misses are principled: one has no stated conclusion on its own, one is a double-negative unsafe to parse with a simple regex, and one places the affirmation before the marker instead of after (checking backward risks false-flipping an ordinary violation that discusses a hypothetical "fine" case before explaining why it's actually blocked).
Tests
npm run checksclean — 550 tests, lint/format/typecheck pass.violation.violation.violation.passverdicts never touched, regardless of reason text.