Skip to content

fix(eval): check last line for LLM judge verdict in fuzzy_match and ua_match - #272

Open
AUTHENSOR wants to merge 1 commit into
web-arena-x:mainfrom
AUTHENSOR:fix/llm-judge-last-line-verdict
Open

fix(eval): check last line for LLM judge verdict in fuzzy_match and ua_match#272
AUTHENSOR wants to merge 1 commit into
web-arena-x:mainfrom
AUTHENSOR:fix/llm-judge-last-line-verdict

Conversation

@AUTHENSOR

Copy link
Copy Markdown

Summary

llm_fuzzy_match and llm_ua_match in evaluation_harness/helper_functions.py extract the judge verdict by checking whether substrings like "correct", "incorrect", "same", or "different" appear anywhere in the judge response. When the judge reasons before its final verdict (common for GPT-4), the reasoning text can contain these words, causing the wrong verdict to be selected.

Example: a judge response like "The student's phrasing differs but is not incorrect. The answer is correct." matches "incorrect" first (via the substring scan over the full text), returning 0.0 when the verdict should be 1.0.

Additionally, both functions used assert "correct" in response / assert "same" in response, which crashes the benchmark if the judge uses a synonym the harness does not expect (e.g., "wrong" or "not matching").

Fix

  • Check only the last line of the judge response for the verdict token, so reasoning text does not override the judge's concluding statement.
  • Replace assert with a fail-safe return 0.0 when no recognized verdict token is found.

Verification

  • Diff touches only llm_fuzzy_match and llm_ua_match (14 lines changed).
  • Both functions now bind the verdict to the judge's last line, matching the convention used by other eval frameworks (e.g., inspect_ai, openai/evals cot_classify).
  • The assert crash path is eliminated.

…match

Previously these functions checked the entire judge response for
substring matches like 'correct' or 'different', which matched
reasoning text before the verdict. Also replaced assert (which
crashes on unrecognized synonyms) with a fail-safe return 0.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant