Skip to content

Fix per-domain eval args misalignment when polyglot is combined with other domains - #40

Open
rootkiller6788 wants to merge 1 commit into
facebookresearch:mainfrom
rootkiller6788:fix-polyglot-eval-args-alignment
Open

Fix per-domain eval args misalignment when polyglot is combined with other domains#40
rootkiller6788 wants to merge 1 commit into
facebookresearch:mainfrom
rootkiller6788:fix-polyglot-eval-args-alignment

Conversation

@rootkiller6788

Copy link
Copy Markdown

Problem

generate_loop() evaluates polyglot through a separate harness (run_harness_polyglot), so it passes a polyglot-filtered domain list into generate():

generate(docker_client, [d for d in domains if d != "polyglot"], ..., eval_samples=eval_samples, eval_subsets=eval_subsets, ...)

but it passes the full eval_subsets / eval_samples lists unchanged. Inside generate(), the staged and full evaluation loops pair them positionally:

zip(domains, eval_subsets, stagedeval_samples)

Whenever polyglot is not the last domain in --domains, the domain list is shorter than the argument lists, so every domain after polyglot is paired with polyglot's eval subset and sample count, and the remaining domains shift by one.

Reproduction

python generate_loop.py --domains polyglot search_arena genesis_go2walking --eval_samples 10 20 30

Before this change, the generate() call for that run receives:

domains=['search_arena', 'genesis_go2walking']
eval_samples=[10, 20, 30]      # search_arena gets 10 (polyglot's), genesis gets 20 (search_arena's)
eval_subsets=['', '_filtered_100_train', '']   # search_arena gets polyglot's empty subset

So search_arena would be evaluated on the wrong subset/sample count, and genesis_go2walking would inherit search_arena's arguments. When polyglot is last (or absent), zip() truncation masks the problem, which is why it only shows up with polyglot in a non-final position.

Fix

Filter the per-domain evaluation arguments alongside the domain list so each remaining domain keeps its own eval_subset / eval_samples:

non_polyglot_domains = [d for d in domains if d != "polyglot"]
non_polyglot_eval_subsets = [s for d, s in zip(domains, eval_subsets) if d != "polyglot"]
non_polyglot_eval_samples = [n for d, n in zip(domains, eval_samples) if d != "polyglot"]

and pass those to generate(). Runs without polyglot are unaffected (the lists pass through unchanged).

Verification

  • A mocked integration harness that runs generate_loop() (heavy dependencies stubbed) confirms the pre-patch generate() call is misaligned and the post-patch call receives correctly aligned per-domain arguments.
  • python -m py_compile generate_loop.py passes.

…other domains

When 'polyglot' is not the last domain in --domains, generate() received the
full eval_subsets/eval_samples lists but a polyglot-filtered domain list, so
the positional zip inside generate() paired each domain after polyglot with
polyglot's eval subset and sample count (and shifted every other domain by
one). Filter the per-domain eval arguments alongside the domain list before
calling generate().
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant