Skip to content

refactor(tasks): share auxiliary model-role resolution on Task - #97

Merged
ethan-scitix merged 3 commits into
mainfrom
refactor/task-role-resolution
Aug 13, 2026
Merged

refactor(tasks): share auxiliary model-role resolution on Task#97
ethan-scitix merged 3 commits into
mainfrom
refactor/task-role-resolution

Conversation

@ethan-scitix

@ethan-scitix ethan-scitix commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Type

  • refactor — code restructuring, no behavior change (one unreachable error-ordering corner, called out below)

Summary

  • Eight tasks each carried their own _resolve_grader / _resolve_extractor. Normalising the role name away, seven are byte-identical; AGIEval's differs only because its builder also needs the candidate model.
  • The declaration half of the role contract was already shared (Task._bind_role_requirement); only the resolution half was copied. So the models_by_role protocol — pooled role model, both-supplied rejection, missing-role message — had eight owners that must move together, and the next grader-bearing benchmark would have added a ninth.
  • Adds Task._resolve_role_model(role, configured, models_by_role, build=...) and routes all eight through it. build takes configured and returns the Model, so each task keeps its own constructor and its own "you must supply one" message — the part that legitimately differs (AGIEval admits the self sentinel and explains its 26-point extractor spread; each grader task explains its own scoring dependency). The seven grader tasks pass self._build_grader directly; AGIEval wraps it only to add the candidate model.
  • A type parameter ties the two arguments together — configured: T against build: Callable[[T], Model] — so configured is typed by whatever the builder accepts rather than object, and the value is not threaded through the call twice. Threading it twice was how a mistyped configured could slip past the both-supplied guard and let a pooled model win silently; ty now rejects a mismatched pairing.
  • Both halves of the contract get direct unit tests. _bind_role_requirement had none, and its missing-binding branch is unreachable from model_requirements_for (which picks the role out of the bindings it was handed), so that message and its a/an agreement were entirely unasserted — even though all eight tasks name a literal role and can miss.
  • Retires a third copy of requires must be TaskRequirements: TaskModelRequirement.__post_init__ owns it, and the copies in _bind_model_requirements / _bind_role_requirement each sat in front of a construction that re-checks the same thing. The two class-named variants stay — they validate the requires ClassVar, a different contract.
  • Net −76 lines of production code (+166 in tests).

The one behaviour exception

Everything about the shared helper is byte-identical to the code it replaces. The exception is the retired type check: a call invalid in two ways at once now reports the binding problem instead of the type problem — _bind_role_requirement(ctx, "grader", <not TaskRequirements>) with no grader binding raises the missing-binding ValueError, and model_requirements_for with a bad requires ClassVar and no bindings raises the no-candidate ValueError.

Every singly-invalid call is unchanged, including a bad requires with a resolvable binding, which still raises the identical TypeError from __post_init__. Neither corner is reachable in production: requires always arrives as cls.requires, already rejected earlier by the two class-named guards (sieval/core/tasks/meta.py at decoration, _validate_model_requirements at construction).

Related Issues

Refs #25 — follow-up from the PR #45 review.

Test Plan

Automated

  • Lint/format clean (ruff check && ruff format --check)
  • Type check clean (ty check)
  • Unit tests pass (pdm run pytest) — 5186 passed; preflight 25 PASS, no FAIL/WARN; sync_package_stubs.py --check and sync_meta_index.py --check both clean

Manual

  • Behaviour pinned against the pre-refactor code. All eight tasks × six supply combinations (both supplied / missing role / pooled present / wrong role key / direct / neither) driven through both trees and diffed: 48/48 rows identical, exception chaining included. Both resolution entry points are classmethods, so this needs no dataset or model fixtures.

  • Mutation-tested the shared helper, because a green suite after a move refactor proves only that nothing imports the dead code. Three mutations against Task._resolve_role_model, all caught:

    mutation result
    remove the both-supplied guard 9 failed — one per task, plus the core test
    make it ignore models_by_role 27 failed — three per task, plus three core
    mis-key the role lookup 9 failed — one per task, plus the core test

    The per-task failure counts are the point: all eight tasks genuinely execute the shared path, rather than passing on leftovers of their own.

  • Mutation-tested the declaration half, now that it has tests. Dropping the vowel branch from the a/an article, letting source_task inherit a parent's meta, dropping the role guard, dropping the context guard, and collapsing source_task to the bare class name are 5/5 caught, each by the intended test.

  • Verified no lingering references to the removed methods anywhere in the tree, including .pyi stubs.

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring — no new modules; all touched files already carry theirs
  • No new upper-layer dependencies added to core/ — the helper lives on Task in core/tasks/ and imports nothing new beyond collections.abc.Callable
  • Deleted code verified — no remaining call sites depend on it

ethan-scitix and others added 3 commits August 13, 2026 20:40
Eight tasks each carried their own `_resolve_grader` / `_resolve_extractor`,
seven of them byte-identical once the role name is normalised away. The
declaration half of the contract was already shared as
`Task._bind_role_requirement`; only the resolution half was copied, so the
`models_by_role` protocol — pooled role model, both-supplied rejection,
missing-role message — had eight owners that had to move together.

Add `Task._resolve_role_model(role, configured, models_by_role, build=...)`
and route all eight through it. `build` is a thunk so each task keeps its own
constructor and its own "you must supply one" message, which is the part that
legitimately differs: AGIEval's extractor needs the candidate model and admits
the `self` sentinel, and each grader task explains its own scoring dependency.

Behaviour is unchanged; every error message is byte-identical to the copy it
replaces. Verified by mutation: removing the both-supplied guard fails 8 tests
(one per task), making the helper ignore `models_by_role` fails 24, and
mis-keying the role lookup fails 8 — so all eight tasks genuinely exercise the
shared path rather than passing on their own leftovers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-ups from the review of this PR.

`_resolve_role_model` takes a type parameter, so `configured` is typed by
whatever `build` accepts rather than `object`, and `build` receives it instead
of the call site closing over the same value a second time. The seven grader
tasks pass `self._build_grader` directly — the empty lambda is gone — and
AGIEval wraps it only to add the candidate model. Threading the value twice was
how a mistyped `configured` could slip past the both-supplied guard and let a
pooled model win silently; ty now rejects a mismatched pairing. Docstring cut
to the two things the code does not already say.

Both halves of the role contract get direct unit tests. The declaration half,
`_bind_role_requirement`, had none: its missing-binding branch is unreachable
from `model_requirements_for`, which picks the role out of the bindings it was
handed, so the message and its a/an agreement were unasserted even though all
eight tasks name a literal role and can miss.

Behaviour is unchanged — all 48 supply combinations across the eight tasks are
byte-identical to the pre-refactor code. The resolution mutations now fail 9,
27 and 9 tests (up from 8, 24 and 8) as the core tests catch each break
independently. Five of six mutations to the declaration half are caught by the
intended test; the sixth is a `requires` type guard that
`TaskModelRequirement.__post_init__` already enforces with the same message, so
no test can pin it as the sole enforcer — recorded in that test rather than
papered over.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`requires must be TaskRequirements` was raised from three places with the same
wording: `TaskModelRequirement.__post_init__`, which is the one with a direct
test, plus copies in `_bind_model_requirements` and `_bind_role_requirement`.
Both copies sit in front of a construction that re-checks the same thing, so
they bought nothing but a third place for the wording to drift. Removed.

The two class-named variants stay — `meta.py` at decoration and
`_validate_model_requirements` at construction validate the `requires` ClassVar,
a different contract, and their messages name the class.

Not a pure no-op, so stating it: with the copies gone, a call that is invalid in
*two* ways at once reports the binding problem instead of the type problem —
`_bind_role_requirement(ctx, "grader", <not TaskRequirements>)` with no `grader`
binding now raises the missing-binding ValueError, and `model_requirements_for`
with a bad `requires` ClassVar *and* no bindings raises the no-candidate
ValueError. Every singly-invalid call is unchanged, including a bad `requires`
with a resolvable binding, which still raises the identical TypeError from
__post_init__. Neither corner is reachable in production: `requires` always
arrives as `cls.requires`, already rejected earlier by the two class-named
guards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ethan-scitix
ethan-scitix merged commit 45d7a6c into main Aug 13, 2026
9 checks passed
@ethan-scitix
ethan-scitix deleted the refactor/task-role-resolution branch August 13, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant