fix(models): compute usage totals, and reject bad builder defaults at bind time

[ethan-scitix]

Type

• fix — bug fix or alignment correction

Summary

Four fixes to what a per-call record may contain, and to when a call is worth failing over.

• Usage totals are computed, not read. One concept had three stances: openai_chat raised when the reported total did not decompose, openai_completions accepted whatever the server sent, and the sglang transport never read a reported total at all — it summed. Both dialects now agree with sglang.
• default_params rejects values that cannot round-trip. _named_json_value took any non-str/bytes Iterable, where the same-package copy_json_value has always taken only list/tuple. Narrowed to match.
• The same rule now holds at call time. _kwargs_to_request pops stop before the JSON check runs and handles it with its own isinstance(stop, Iterable) branch, so narrowing _named_json_value closed the bind-time door and left the call-time one open.
• Builder defaults are checked at bind time. meta() runs per response, so it was the first thing to notice a bad default — raising only after a call had been billed.

Why the usage check was the risky one. OutputContractError is documented as "a successful wire reply violated its declared output contract" — the tokens were billed. It is never caught anywhere in sieval, so it reaches the runner's generic except Exception, consumes a retry, and lands the sample in fails scoring 0. A server one token off discarded a good reply, and in the streaming path only after the rollout had finished. Computing the total also makes the identity structural, so _validate_input_scoring_boundary's total-token comparison can no longer fire and is removed; the remaining boundary checks compare usage against echoed logprob positions and still have teeth.

Why the Iterable gap mattered. Two admitted shapes corrupt ModelMeta.default_params — which its own docstring calls "persisted" — without raising: a set serializes in hash order, against the resume strict-match contract; a generator is consumed by the first call, so every later meta() persists []. Verified before changing anything: {"c","a","b"} → ['a','c','b'], and a generator gives ['a','b'] then [].

Why the call-time half had to follow. stop is echoed into request_params, which is persisted, so the bind-time fix on its own produced a rule that held at one entry point and not the other — ChatModel(stop={"a","b"}) raised while agenerate(stop={"a","b"}) was accepted and sent the set in hash order. Measured across four processes, the same five-element set produced a different order each time. tools / tool_choice are deliberately left taking any iterable: a set of tool definitions is impossible (mappings are unhashable) and the elements already go through _named_json_value.

Related Issues

Refs #25. Follow-ups from the review of #98, which flagged both as belonging in their own PR.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit tests pass (pdm run pytest) — 5332 passed. Three tests/performance gates (efficiency/memory ratios) fail only when the whole suite runs; all three pass in isolation on this branch. Load sensitivity, not a regression — the diff adds one isinstance check.
• sieval/core/models/model.py line+branch coverage 99% (coverage run --source=sieval, per sieval/core/CLAUDE.md) — no uncovered line falls in the changed region.

Manual

• Mutation-tested every new test — each reverted change fails only the tests that should catch it:

  mutation	result
  usage reads the server-reported total again	2 failed
  _named_json_value widened back to Iterable	5 failed
  call-time stop widened back to Iterable	2 failed
  bind-time validation turned into a no-op	6 failed (both write sites)

• Reachability scan — both narrowings close a trap, not a live defect. _kwargs has two writes (_initialize, with_args); builder_defaults comes only from the ChatModel/GenModel constructors. All eight construction sites are ChatModel(**config) from task YAML, which cannot express a set or generator. On the call-time side, every in-tree stop= passes a list(...) or a module-level tuple (human_eval, gsm8k, gsm1k, mbpp, livecodebench, openbookqa, hendrycks_math, theoremqa), and every other agenerate(...) argument is a scalar. The exposure is to out-of-tree task authors, whose symptom would have been a resume abort reporting a diff it cannot describe.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring — no new modules
• No new upper-layer dependencies added to core/
• Deleted code verified — the removed total-token check is unreachable once the total is computed; its test case goes with it

If: Breaking Change

• Described what breaks and migration path in Summary
• Existing tests updated to reflect new behavior

Programmatic callers only, and all of it is unreleased — there is no migration path to write, and no CHANGELOG entry. sieval/core/models/dialects/ and _named_json_value do not exist at v0.7.0, where Model.meta() was a bare dict(self._kwargs) with no JSON validation of default_params and self._kwargs = kwargs with no copy. Every behaviour below arrived in #45 after that tag, so no released version can have depended on it.

• ChatModel(model=..., stop={"a","b"}) raises at construction instead of persisting hash-ordered output, and a non-finite default raises there rather than on the first meta().
• agenerate(stop={"a","b"}) raises instead of sending the set in hash order.
• The narrowing also reaches _named_json_value's other callers — tools[i], tool_choice, the structured-output schema, and dialect options — so a set or generator nested in any of those now raises where it used to be silently flattened into a list.

No in-repo caller is affected and YAML config cannot produce any of these. Two existing tests asserted the old timing, not the old outcome, and were updated accordingly.