chore(evals): migrate baseline equivalence to Vally 0.10

[WilliamBerryiii]

Summary

Migrate baseline-equivalence evaluation from the Vally 0.6 pairwise-grader contract to native Vally 0.10 persisted rubrics and structured comparison JSONL.

Implement this independently of Dependabot PR #2491 so schema, parser, policy, ADR, and validation changes can be reviewed and rolled back without unrelated dependency updates or the ESLint 10 peer conflict.

Related work: #2479 remains independent and cross-referenced. It is not a parent, sub-issue, blocker, or duplicate.

Branch and delivery

• Branch: feat/vally-0-10-migration, based on current main.
• Delivery: one focused PR containing dependency, schema, parser, legacy cleanup, policy, fixtures, tests, reporting, ADR, documentation, and calibration work.
• PR workflow: open as draft, calibrate through CI, add summary evidence to the PR description, retain detailed output in temporary CI artifacts, commit approved policy values, then mark ready.

Scope

• Pin @microsoft/vally-cli to 0.10.0 without unrelated dependency upgrades.
• Keep evals/baseline-equivalence/stimuli.yml as the canonical prompt, tag, policy, and rubric source, with deterministic synchronization validation.
• Remove unsupported type: pairwise declarations.
• Use a separate required documented-divergence gate; it is advisory in devloop and blocking in CI.
• Inventory all 39 per-agent signature files, migrate every unique rule into canonical assertions, verify parity, then delete the old compare spec, signature directory, signature generator, dedicated generator tests, renderer code, dashboard signature-root logic, and signature-linked documentation.
• Replace --run-a / --run-b and console parsing with --baseline / --treatment and comparison JSONL.
• Remove the nonfunctional StimulusFilter parameter and the unusable eval:compare npm script.
• Rename harness modes from pr / nightly to devloop / ci without compatibility aliases; preserve current workflow triggers and changed-path eligibility.
• Default devloop executor to gpt-5.6-luna with explicit override.
• Use CI executors gpt-5.6-luna and claude-haiku-4.5; use claude-haiku-4.5 as comparison judge.
• Start devloop and initial report-only CI calibration at one trial per stimulus.
• Start with a 2% judge-error target; report exact counts/rates and use devloop evidence to propose rounding semantics for PR approval. Malformed, missing, duplicate, or unmatched evidence remains fail-closed.
• Keep equivalence scoring report-only until migration PR reviewers approve a calibrated threshold.
• Replace A/B reporting atomically with baselineWins, treatmentWins, ties, native Vally statistics, and separate gate results.
• Version the new result contract as schemaVersion: 2.0.0; reject V1 with rerun guidance and add no compatibility reader or migration script.
• Commit only test-required sanitized fixtures and no separate fixture manifest or calibration report.
• Make narrow factual maintenance updates to accepted ADR Add repository foundation and documentation files #2.
• Reconcile proposed ADR Add linting and validation scripts #10 with Vally 0.10 mode, model, result-schema, and compatibility policy.
• Add a new accepted Vally 0.10 evaluation-policy ADR related bidirectionally to ADRs Add repository foundation and documentation files #2 and Add linting and validation scripts #10. Do not supersede or semantically rewrite ADR Add repository foundation and documentation files #2.
• Revalidate baseline-equivalence documentation and Vally grader guidance against 0.10 capabilities.

Out of scope

• ESLint 10 compatibility work.
• Indefinite Vally 0.6 compatibility support.
• A broad adapter that emulates the removed Vally 0.6 API.
• V1 result migration or deprecated field/mode aliases.
• Expanding CI trigger or changed-path criteria.
• Opportunistic policy changes without calibration evidence and PR approval.

Acceptance criteria

• A focused branch based on main contains no unrelated Dependabot or ESLint changes.
• Root installation reproducibly resolves Vally 0.10.
• Baseline-equivalence inputs contain no unsupported pairwise graders.
• Canonical, baseline, and treatment prompt/rubric fields are synchronized by validation.
• Native comparison writes and parses structured JSONL.
• Equivalent and documented-divergence stimuli use separate, required, tested gates.
• All 39 signature files are inventoried and former signature coverage has canonical assertion parity before the complete signature subsystem is deleted.
• Structural evidence defects cannot pass CI; judge-error policy is calibrated and reviewer-approved.
• Devloop is advisory and CI is authoritative without changing existing trigger/path criteria.
• Initial calibration uses the selected low-cost executors, judge, and one-trial baseline.
• Reports use schema V2 baseline/treatment fields, native statistics, and split gates with no A/B aliases.
• Sanitized Vally 0.10 fixtures cover eval and comparison machine contracts.
• ADR Add repository foundation and documentation files #2 receives factual maintenance only, ADR Add linting and validation scripts #10 is reconciled, and the new accepted policy ADR is related bidirectionally to both.
• Pester, eval lint/schema/text/safety, PowerShell lint, ADR consistency, Markdown/frontmatter, skill validation, npm audit, and full repository validation pass.
• Calibration evidence and proposed enforcement values are reviewed in the draft migration PR before it is marked ready.

Estimated effort

Two to three engineering days plus credentialed model runtime and review.