[RFC]: Uncertainty on the headline — a clustered standard error, and the paired delta that acceptance actually needs

[ethan-scitix]

Motivation.

As of #93, report.json says which column the headline came from (score_key)
and which population it was averaged over (denominator_policy). It still says
nothing about how much that number would move if the same run were repeated.

Every question SiEval exists to answer is a difference of two numbers:

• did this converted / quantized / re-served checkpoint degrade against the
  bf16 baseline?
• does our number reproduce the published one?
• did a grader fix move scores, and by how much?

A difference is unreadable without a noise scale, so today every such judgement
is made by hand, in a PR description, and the arithmetic is redone from
scratch each time. Three places where that has already bitten:

1. A saturated set cannot rank, and nothing in the report says so. An
   earlier coverage survey found several MathArena competitions whose top-N
   spread at the frontier is smaller than a plausible run-to-run σ — ranking on
   them is a coin flip, while other sets in the same family spread by tens of
   points. That is a property of (set, model population) that a reader
   currently cannot see from report.json at all.
2. Repeats do not sharpen a paper comparison. Raising n shrinks the
   sampling variance of our estimate, but the published number carries its own
   error that we cannot reduce, so the comparison's σ has a floor of
   roughly sqrt(σ_paper² + σ_live²). Without this written down, "run it again
   with more repeats" keeps being proposed as a way to resolve a disagreement it
   cannot resolve.
3. pass@k moves for two different reasons. It is an upper bound that
   rises with sampling variance, which is why [RFC]: Sampled-answer metric family for math and MCQ — avg@k / pass@k / pass^k, consistency, and health signals #74 shipped pass^k beside it.
   Neither can be read as "better" or "worse" without knowing the width.

What exists today is one task's worth, borrowed from upstream:

• hle_0shot_gen reports confidence_interval — a 95% Wald half-width,
  1.96 * sqrt(p(1-p)/n), computed in sieval/community/hle.py over total
  attempts ((finals + fails) * n). It is HLE's own convention, vendored, and
  not comparable with anything else.
• ugmathbench_0shot_gen_fixed's reference_impl.notes reason about a binomial
  standard error in prose to argue a 0.198 pp delta is immaterial — correct
  reasoning, computed by hand, reproducible by nobody.
• docs/guide/metrics.md does not mention uncertainty at all.

RFC #74 deferred this deliberately, called it "arguably the highest-value
addition of all", and named its precondition: a follow-up RFC "once the
estimator module in A exists". That module is sieval/core/tasks/metrics.py,
and #93 finished the last of #74's in-scope items — so the precondition is met.

Proposed Change.

A. Bootstrap at the seam that already exists

metrics.aggregate() already reduces a list[dict] of per-problem metric
values to the report's means. Those per-problem values are exactly the input a
resampling interval needs, so this is an addition at an existing seam rather
than a new pipeline: resample problems with replacement, recompute the same
reduction, report a percentile interval.

This works uniformly for every key the module produces — pass@1, avg@n,
pass@k, pass^k, maj@k, self_consistency — which matters because three of
those are nonlinear in c and have no convenient closed form.

B. The cluster is the problem, not the rollout

At n > 1 the rollouts of one problem are strongly correlated. Treating
N = problems × n as independent Bernoulli trials — which is what HLE's Wald
half-width does — understates the interval, and understates it more the more
the model varies per problem. The resampling unit must be the problem.

C. Name and shape

Proposal: one key per interval-bearing metric, suffixed, e.g.
avg@n_ci95 / pass@k_ci95 (half-width, in the same unit as the metric), plus
score_ci95 mirroring whichever key score_key names. Additive, so stored
reports stay readable — the same compatibility argument #74 F relied on, and
cli/leaderboard/scanner.py still reads only report["score"].

Open: half-width vs an explicit [lo, hi] pair (the latter survives an
asymmetric interval, which a percentile bootstrap near 0 or 100 produces).

D. The paired difference is the primary deliverable

"Do the two intervals overlap" is the wrong test for every question in the
Motivation, and shipping only per-run bars would actively mislead: on the same
problem set, the paired per-problem difference has far smaller variance than two
independent intervals suggest, so overlapping bars routinely hide a real,
consistent regression.

So the RFC proposes the paired form as a first-class output, not an afterthought:
given two runs over the same problem set, bootstrap the per-problem delta and
report its interval. Whether that lives in sieval leaderboard (comparing two
result dirs) or in a new verb is the main open design question here.

E. A pipeline failure is not a draw

Under denominator_policy: requested a failed sample contributes a
deterministic zero, not a sampled outcome. Resampling it as though it were a
Bernoulli trial inflates the interval; dropping it silently changes the estimand.
The proposal: resample over the declared denominator, treating failures as the
fixed zeros they are, and say so in the guide. This is a concrete payoff of #93 —
the policy is now machine-readable, so the interval can key off it instead of
guessing.

F. Determinism — the hard constraint, stated up front

A bootstrap needs randomness, and #74 D explicitly refused a seed in the metric
layer (that is why maj@k requires k == n rather than sub-sampling: "a seed
in the metric layer is a new source of irreproducibility"). This RFC cannot quietly
reverse that. Three candidate resolutions, in rough order of preference:

1. Closed form where one exists (avg@n, pass@1) — a clustered standard
   error needs no randomness at all, and covers the headline of most tasks.
2. A data-derived seed — derive it from the sorted problem ids, so the
   interval is a pure function of the stored per-problem values and recomputing
   it gives the identical number on any machine. Requires stating the resample
   count as part of the contract.
3. Deterministic jackknife instead of a bootstrap for the nonlinear keys —
   no randomness, weaker in small samples.

Whichever is chosen, the requirement is that two readers of the same
report.json compute the same interval.

Feedback Period.

One week.

CC List.

(none)

Any Other Things.

Scope

In scope: every task whose report goes through metrics.aggregate(), plus a
documented story for the ones that do not (the clp/ppl family is
deterministic per sample, so its only uncertainty is problem sampling —
still real, and arguably the cheapest case).

Out of scope, deliberately:

• A pass/fail gate. This RFC produces a number, not a verdict. Turning
  "the paired delta excludes zero" into an acceptance criterion is a separate
  decision with its own failure modes.
• An anomaly rule keyed on the interval. Adding one rotates rules_hash
  and marks every stored anomalies.json stale fleet-wide, so if it is wanted
  it should be bundled deliberately rather than trickled in — same caveat [RFC]: Sampled-answer metric family for math and MCQ — avg@k / pass@k / pass^k, consistency, and health signals #74
  recorded.
• Retrofitting stored runs. Intervals are computed inline at report time,
  consistent with the decision not to ship a recompute command ([RFC]: Sampled-answer metric family for math and MCQ — avg@k / pass@k / pass^k, consistency, and health signals #74 G). Old runs
  keep their columns; new runs gain these.

Compatibility

• report.json — additive. scanner.py reads report["score"] only.
• profile.json / meta.json / effective_config.yaml — untouched.
• anomalies.json — unaffected unless a rule is added (see above).
• hle_0shot_gen's existing confidence_interval conflicts on meaning: same
  concept, different estimator (Wald over attempts, not clustered over problems).
  It cannot simply be renamed — it is HLE's published convention and a stored
  column. Proposal: keep it, and let the shared key sit beside it, exactly as
  first_rollout_correct's accuracy sits beside pass@1 today.

Also needs updating

• docs/guide/metrics.md — a new section, and the pairs rule extended: an
  interval must be read with its denominator_policy, and a paired delta must
  never be inferred from two per-run intervals.
• scripts/check_preflight.py — if intervals become mandatory for some class of
  task, check_report_declarations (added in feat(metrics): declare score_key / denominator_policy on every task report #93) is where that would be
  enforced.

Refs

Follow-up to #74 (its "Deliberately not proposed here" section) and #93, which
made denominator_policy machine-readable and is what §E leans on.