infra(ci): consider adding "qa-report --asr" Whisper sanity check to CI (deferred — Tier-3 from #87)

[shaypal5]

Status: deferred — opened to track the Tier-3 ASR sanity policy decision in #87 so we revisit it when conditions change.

Background

PR #88 (closes #87) introduced the effective-prosody cap in synthbanshee/tts/renderer.py and a paired qa-report --asr Whisper sanity check. The check detects clips that synthesize fine to a listener but trip Whisper-large-v3's silence-detection / segmentation heuristic — the failure mode on sp_it_a_0001 that produced the residual #87 regression.

Per CLAUDE.md "ASR sanity check policy", this check is currently local-only:

• Tier 1 (cap unit tests, no Whisper) → CI on every PR.
• Tier 2 (Whisper-on-fixture) → not implemented; deferred.
• Tier 3 (qa-report --asr real-render) → run locally once per audio-touching PR, after all fixes.

Why deferred (2026-05-06 decision)

1. Cost. Each real Tier-3 invocation pays ~$1–$2 of Azure rendering for one Tier-A scene plus Whisper download / inference. At current scene volume (single-digit Tier-A clips per PR review cycle), local manual runs are perfectly tractable.
2. Author-triggered labelling does not add value yet. The only "audio-touching PR" author is currently Claude/agent-driven — labelling is a self-decision, so a tts-full-eval label gate would just shift the same decision into a different surface.
3. Repo secrets churn. Adding AZURE_TTS_KEY to repo secrets is non-trivial (rotation, scoping, observability) and we'd want a proper Azure spend cap before letting CI render.

Re-evaluation triggers

Reopen for active design when any of these hold:

• Tier-A clip volume per PR cycle exceeds ~5 — manual runs become a bottleneck.
• The repo gains additional human contributors regularly opening audio-touching PRs.
• Azure spend telemetry / cost caps are available so a scheduled CI workflow has bounded cost exposure.
• Whisper inference becomes 10× cheaper (e.g., a faster CT2/quantized backbone is validated against the investigate(tts): #83 residual — Whisper WER regression on high-intensity (I3+) Tier A clips #87 fingerprint).

What a CI implementation would look like (rough sketch — not committed)

• New scheduled workflow .github/workflows/asr-sanity-nightly.yml that runs nightly on main.
• Reuses the existing eval-asr extra (uv pip install -e ".[eval-asr]").
• Renders one fixed Tier-A scene (sp_it_a_0001 — the investigate(tts): #83 residual — Whisper WER regression on high-intensity (I3+) Tier A clips #87 evidence-base scene) end-to-end through Azure, runs Whisper, asserts length_ratio ≥ 0.85 and WER ≤ 0.10.
• On failure, opens an issue with the result and a link to the run.

Alternative: a tts-full-eval label-gated pull_request workflow, fired only when an author labels a PR. Same content; different trigger.

What is already in place (so the deferral doesn't lose institutional knowledge)

• qa-report --asr is a one-line invocation that runs the full check locally.
• The cap activations are recorded in clip metadata (generation_metadata.effective_prosody_caps) and surfaced by qa-report without requiring --asr, so a static "did the cap fire" signal is available cheaply on every QA pass.
• The cap thresholds and the rationale for the policy are documented in CLAUDE.md ("ASR sanity check policy").
• This issue is the canonical reminder; do not delete it on merge.

References

• investigate(tts): #83 residual — Whisper WER regression on high-intensity (I3+) Tier A clips #87 — root cause + remediation + bisect data.
• PR infra(ci): consider adding "qa-report --asr" Whisper sanity check to CI (deferred — Tier-3 from #87) #88 — the cap + qa-report extension that introduced this policy.
• CLAUDE.md "ASR sanity check policy" section.