feat(models): record the full token breakdown, not just the totals

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

UsageStats has declared reasoning_tokens and cached_tokens since the binding plane landed, and no dialect ever populated them. So a converted or requantized checkpoint that held its score while burning twice the reasoning budget was invisible — the delivery regression no accuracy metric catches. This collects the breakdown, persists it, and reports it.

• Both OpenAI-shaped dialects lift prompt_tokens_details / completion_tokens_details (reasoning, cached, accepted/rejected speculative-decoding tokens) through one shared dialects/_usage.py; chat and completions receive the same wire object, so the reader has one owner rather than two copies that drift the next time OpenAI adds a field. sglang surfaces the cached_tokens it was already reading twenty lines below for its radix-cache guard.
• The optional counts are int | None, not int = 0. Most OpenAI-compatible servers omit the detail objects entirely, so a zero default makes "no cache hits" and "never said" the same number, and any average over a mixed fleet silently wrong. Carried to disk: ModelUsage gains NotRequired keys written only where the server reported one, so an absent key means unreported and a present 0 is a real measurement.
• No subset relation is enforced, and the reported total is now recorded rather than dropped. reasoning <= completion is an OpenAI convention, not a wire guarantee — a server counting reasoning outside its completion count is exactly the one whose reported total exceeds the computed one. reported_total_tokens is kept only where it disagrees with prompt+completion, so it is absent on almost every call and, when present, is the only evidence that a provider's accounting differs from ours. fix(models): compute usage totals, and reject bad builder defaults at bind time #100 had nowhere to put that difference and dropped it silently; this closes that.
• The profiler reports shares against the calls that reported them, never against every call (which understates a share by exactly the non-reporting fraction), and prints that denominator on every line rather than only when partial — a coverage note given sometimes reads as full coverage the rest of the time. Breakdowns are never folded into total_tokens, which stays prompt+completion.

One coupling fixed on the way. record_model_usage and aggregate_token_usage carried the same accumulation logic twice — the live path and the resume rebuild. They now share one accumulator, because a field added to the live path alone would have vanished on the next resume. The rebuild also clears the new accumulators, which are additive and would otherwise double-count everything they re-read.

Deliberately not collected: audio_tokens. Nothing in sieval sends or receives audio, so it would ship as a permanently absent key.

Related Issues

Refs #25. Follow-up from the review of #100, which established the computed total and flagged that a server/computed disagreement was being discarded with no trace.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit tests pass (pdm run pytest) — 5359 passed. Two tests/performance gates (efficiency/memory ratios) fail only when the whole suite runs and pass in isolation; pre-existing load sensitivity, unrelated to this diff.
• Coverage over every changed module, all above the 95% sieval/core gate: _usage.py 100%, transports/sglang.py 100%, model.py 99%, ir.py 98%, openai_completions.py 97%, openai_chat.py 96%, profiler.py 96%.

Manual

• Mutation-tested every new test — each reverted decision fails only the tests that should catch it:

  mutation	result
  absent detail container reads as 0 instead of None	6 failed
  reported_total_tokens recorded even when it agrees	5 failed
  share denominator diluted with non-reporting calls	2 failed
  resume rebuild stops clearing the breakdown accumulators	1 failed
  sglang sums cached_tokens across the n samples	2 failed
  _model_usage writes 0 instead of omitting an unreported key	2 failed

• The None vs 0 distinction is pinned from both directions, since it is the whole reason the fields are optional: sglang's _meta fixture really does report cached_tokens: 0, and that 0 survives to the record; a meta with the key deleted yields None. Same pair at the dialect layer and across the ModelUsage record boundary.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring — new dialects/_usage.py carries it
• No new upper-layer dependencies added to core/
• Deleted code verified — nothing deleted; the duplicated accumulation in aggregate_token_usage moved into the shared _accumulate_usage and is covered by the rebuild test

If: Breaking Change

• Described what breaks and migration path in Summary
• Existing tests updated to reflect new behavior

On-disk record shape only, and all of it is unreleased — no migration path, no CHANGELOG entry. sieval/core/models/dialects/ does not exist at v0.7.0; the whole binding plane arrived in #45 after that tag. Two existing dialect tests asserted that a divergent reported total was discarded, and one sglang pair asserted cached_tokens was dropped; all four now assert the new, deliberate behaviour rather than the old absence.