feat(tasks): ifeval/multi_if _fixed — three repaired constraint checkers

[ethan-scitix]

Three of IFEval's 25 constraint checkers grade something other than what their own
instruction says. This adds ifeval_0shot_gen_fixed and multi_if_0shot_gen_fixed,
which substitute repaired checkers and quantify the delta. Neither unqualified task
changes behaviour — ifeval_0shot_gen and multi_if_0shot_gen keep grading through
the vendored registries, defects included, because that is what reproduces a published
number.

Nothing here changes what an item asks. Every repair is to the code that decides
whether an answer complied.

The three defects

length_constraints:nth_paragraph_first_word — the list counted is not the list
indexed. Upstream splits on \n\n, decrements a count for each blank chunk, then
indexes the unfiltered list:

paragraphs = re.split(r"\n\n", value)
num_paragraphs = len(paragraphs)
for paragraph in paragraphs:
    if not paragraph.strip():
        num_paragraphs -= 1
...
paragraph = paragraphs[self._nth_paragraph - 1]

A blank chunk at or before index nth - 1 therefore checks the wrong paragraph — or
hands the check a blank one — while the total it is compared against was computed on
the other reading. The repair filters once and indexes what it counted.

This fires far more often than it looks like it should, because whole runs open with a
blank chunk uniformly depending on the chat template: 540 of 541 responses in one
stored Intern-S2-Preview run open with a space then a blank line, and all 100 responses
in a Qwen3-30B-A3B thinking-on run open with \n\n outright. A second stored full-541
run does it in 0.2%, and its flips at this id are correspondingly zero — so the blast
radius is run-dependent, not universal.

Second, Multi-IF-only half: 32 of its 749 slots store a first_word spanning more than
one token (Hindi 12, Spanish 9, French 3, Portuguese 3, Russian 3, Italian 2, English
0), and the prompts carrying them ask for that same phrase. Upstream compares against
paragraph.split()[0], one token, so such a slot returns FAIL for every possible
response — a check that cannot pass measures nothing. The repair compares as many
tokens as the constraint's own value spans, each normalised by upstream's own routine.
It does not relax anything: the phrase must still open the paragraph, in order. For a
single-token value — every IFEval slot and 714 of Multi-IF's — it reduces to upstream's
expression token for token.

keywords:letter_frequency — the item is graded against a letter nobody asked for.
Upstream accepts a letter only in [a-z]; anything else is silently replaced by
random.choice(string.ascii_letters), drawn freshly on every call. So an item
naming # or ! is graded against a different letter each time it is scored. The
repair keeps the character the item names, and consumes no draw, so the global RNG
stream other checkers default arguments from is unperturbed. Values that are genuinely
unusable (None, more than one character after stripping) still fall back to
upstream's behaviour — there the item states nothing gradable, and inventing a rule
would be worse.

change_case:english_capital — langdetect is called on input it cannot read.
Upstream is value.isupper() and langdetect.detect(value) == "en". langdetect's
profiles are built from lowercase text, so ALL-CAPS input is off-distribution for every
profile and the verdict wanders between the Latin-script languages. The checker's own
precondition is that the response is all capitals, so every response it ever sees is
in exactly the state the detector handles worst. Only the argument changes: the repair
detects on a case-folded copy. isupper() is still upstream's, still first, and still
short-circuits, so the set of responses that reach the detector is exactly the set
upstream sends there.

The sibling change_case:english_lowercase is deliberately not repaired — it calls
the detector on text that is already lowercase, which is the condition the profiles
were built for.

Measured score impact

Measured by running each task's own feedback()/report() over the same stored
responses — these are the numbers a run reports, not a grader-level approximation —
with langdetect's factory seeded identically per arm.

ifeval_0shot_gen_fixed, strict prompt-level:

set	upstream → fixed
Intern-S2-Preview, full 541	92.79 → 95.38 (+2.59)
a second stored full-541 run	94.82 → 95.19 (+0.37)
Qwen3-30B-A3B thinking-on, 100 prompts	65.00 → 67.00 (+2.00)

Loose prompt-level +0.55 / +0.55 / +2.00. Coverage: the three ids account for 70 of 834
constraint slots over 68 of 541 prompts (12.6%).

Flips are bidirectional — the second run's loose reading moves english_capital 2
slots FAIL→PASS and 1 PASS→FAIL. A well-posed detector removes false passes as well
as false failures.

multi_if_0shot_gen_fixed, over 160 stored Qwen3-30B-A3B thinking-on conversations
(1,218 constraint slots):

turn	overall	strict instruction-level
1	75.45 → 77.51 (+2.06)	78.14 → 80.97 (+2.83)
2	69.09 → 70.74 (+1.65)	79.46 → 81.44 (+1.98)
3	62.67 → 63.30 (+0.63)	78.75 → 79.64 (+0.89)

score moves 69.07 → 70.52 (+1.45). Coverage: 1,121 slots across 515 of 4,501
conversations (11.4%).

The strict/loose asymmetry is mechanical, and there is a test for it rather than a
claim: the loose reading re-tries each response with its first line stripped, which
deletes the very blank chunk that shifts the index. Loose was accidentally masking
the paragraph defect, not immune to it — which is why it is the weaker evidence of the
two.

It also removes a source of run-to-run nondeterminism

Re-grading each IFEval set under five different langdetect seeds, strict
prompt-level spans:

set	upstream	fixed
Intern-S2-Preview 541	92.61–92.79 (0.18)	95.38–95.38 (0.00)
second full-541 run	94.45–94.82 (0.37)	95.19–95.19 (0.00)
Qwen3 100-prompt	63.00–65.00 (2.00)	65.00–67.00 (2.00)

On both full sets the repaired task is seed-invariant: not one of the 834 slots
changes verdict across all five seeds. The 100-prompt subset keeps its 2.00 spread, and
the attribution is exact — its two unstable slots are one
change_case:english_lowercase (deliberately unrepaired) and one english_capital
whose response is a 94,152-character repetition loop that reads as Welsh on one seed in
five. The repair makes detection well-posed; it does not make langdetect deterministic,
and on a degenerate response nothing would.

Multi-IF keeps a 0.18 score spread after repair — the same 0.18 upstream has, because
that residue is language routing on genuinely multilingual text, which no repair here
touches and none should.

Reproduction note. langdetect.detect is randomized and sieval never sets
DetectorFactory.seed, so an unseeded A/B flips checkers these tasks do not touch:
change_case:english_lowercase flipped between arms during this measurement before its
factory was pinned, and looked for a while like the repair leaking through a shared
RNG. Pin the seed in both arms before attributing any flip to a fix.

Design

One overridden method, and it is asserted, not asserted-in-a-docstring. Each
_fixed task overrides exactly _instruction_dict() — a new seam on the base tasks
returning None, meaning "the vendored registry". Prompting, both graders, the
per-sample record and the pooled report are all inherited, so the two tasks cannot
diverge anywhere except at the registry. A test checks the override set is literally
{"_instruction_dict"} and that preprocess/infer/postprocess/feedback/report
are the same function objects.

The repairs are mixins over the upstream classes, so build_description's defaults
and validation, get_instruction_args and the description pattern all stay upstream's
by construction rather than by review.

Registries are built fresh per call and the vendored globals are never mutated —
samples grade concurrently, and swapping a global would change how the unqualified
task grades in the same session. _build also refuses to mount a fix under an id not
already in the base registry: after an upstream rename the overlay would otherwise land
on a key nothing looks up, and _fixed would quietly grade identically to the
unqualified task while still claiming a delta.

Why the repair module lives in community/ as original code, which that directory's
CLAUDE.md asks to be argued for: the same three checkers are vendored twice — under
instruction_following_eval/ (google-research) and under multi_if/ (Meta's fork of
the same file). For these three classes the two copies differ only in one import path
and one logging call, neither of which a repair touches. Holding the fix outside both
vendored packages serves upstream alignment rather than working against it: the
alternative is two copies of every repair inside files whose value is being diffable
against upstream, and a fix duplicated into two files is a fix that will eventually
exist in only one of them.

The one change inside the vendored files is a keyword-only instruction_dict
parameter on their four grader functions, defaulting to the vendored registry. Omitting
it reproduces upstream exactly, which is what the unqualified tasks do. It was preferred
over re-implementing the grading loops in the repair module: those loops are ~25 lines
each of upstream logic and a copy would drift silently at the next re-sync, whereas a
parameter cannot.

Both tasks are experimental, deliberately. By construction neither can reproduce a
published anchor — they grade differently on purpose — and neither has been run end to
end under its own name. _fixed owes a quantified divergence, which is above and in
reference_impl.notes; a live run is what promotion owes.

Tests

95 new tests.

• tests/unit/community/test_instruction_following_eval_fixed.py (68) asserts every
  repair twice: once that it reduces to upstream on the inputs upstream already
  handled — an 8-response × 3-nth matrix for the paragraph checker, description-string
  equality for letter frequency, a held-constant stubbed detector for english_capital
  — and once that it changes the verdict on the input that exposed the defect. The
  reduction half is what makes the divergence provably the defect and not a rewrite
  riding along.
• Scope is bounded from both sides: a blank chunk after the target index is asserted
  harmless, a wrong word and a wrong paragraph count still fail, a blank first_word
  stays ungradeable in both arms, and the fixed letter-frequency checker is asserted to
  consume no random draw (random.getstate() unchanged).
• tests/unit/tasks/test_{ifeval,multi_if}_0shot_gen_fixed.py (13 + 14) pin
  registration, the fresh-interpreter import discipline (evaluation_lib, the repair
  module, langdetect and nltk all absent after importing the task), the one-method
  claim, byte-identical judgements on unrepaired-only samples, and that upstream cannot
  pass a multi-token first_word with any response at all.
• change_case:* fixtures stub the detector rather than calling it: a test that flipped
  with the process RNG would be no evidence at all.

Multi-IF fixtures are synthetic throughout — no prompt or kwarg text from that dataset
appears anywhere in this PR.

Verification

ruff check + ruff format --check clean repo-wide; ty check reports zero
diagnostics in sieval/ and none in any file this PR adds or touches;
scripts/check_preflight.py passes every check with both tasks registered (including
check_report_declarations, check_reference_kind and check_meta_index_sync);
sync_meta_index.py and sync_package_stubs.py re-run.

Full unit suite against a clean worktree at the same base commit: 13 failed / 5262
passed at base, 13 failed / 5357 passed on this branch. The two failure sets are
identical, and the passed count differs by exactly 95 — the tests this PR adds. The 13
are ModuleNotFoundError: No module named 'rouge_score' in the IHEval modules, an
optional dependency absent from the measuring environment and untouched by this PR.

🤖 Generated with Claude Code