Fix per-domain eval args misalignment when polyglot is combined with other domains - #40
Open
rootkiller6788 wants to merge 1 commit into
Open
Conversation
…other domains When 'polyglot' is not the last domain in --domains, generate() received the full eval_subsets/eval_samples lists but a polyglot-filtered domain list, so the positional zip inside generate() paired each domain after polyglot with polyglot's eval subset and sample count (and shifted every other domain by one). Filter the per-domain eval arguments alongside the domain list before calling generate().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
generate_loop()evaluatespolyglotthrough a separate harness (run_harness_polyglot), so it passes a polyglot-filtered domain list intogenerate():but it passes the full
eval_subsets/eval_sampleslists unchanged. Insidegenerate(), the staged and full evaluation loops pair them positionally:Whenever
polyglotis not the last domain in--domains, the domain list is shorter than the argument lists, so every domain afterpolyglotis paired withpolyglot's eval subset and sample count, and the remaining domains shift by one.Reproduction
Before this change, the
generate()call for that run receives:So
search_arenawould be evaluated on the wrong subset/sample count, andgenesis_go2walkingwould inheritsearch_arena's arguments. Whenpolyglotis last (or absent),zip()truncation masks the problem, which is why it only shows up with polyglot in a non-final position.Fix
Filter the per-domain evaluation arguments alongside the domain list so each remaining domain keeps its own
eval_subset/eval_samples:and pass those to
generate(). Runs withoutpolyglotare unaffected (the lists pass through unchanged).Verification
generate_loop()(heavy dependencies stubbed) confirms the pre-patchgenerate()call is misaligned and the post-patch call receives correctly aligned per-domain arguments.python -m py_compile generate_loop.pypasses.