bench: categorical failure-mode metrics + Mandarin corpus expansion - #20
Draft
larryxiao wants to merge 2 commits into
Draft
bench: categorical failure-mode metrics + Mandarin corpus expansion#20larryxiao wants to merge 2 commits into
larryxiao wants to merge 2 commits into
Conversation
…SPEC-021) Addresses issue #17 where [SPEAKING CHINESE] hallucinations are invisible to WER/CER alone — the score looks bad but can't tell the user what went wrong or which fix to try. Three new per-clip metrics classify each output categorically: - failureMode: .placeholder (bracket annotations like [SPEAKING CHINESE]), .silentTranslation (CJK audio → Latin output), .garbled (WER > 200%), .ok - outputScriptMatch: true when reference and hypothesis share the dominant script - hallucRateRaw: fraction of output chars inside [...] brackets Aggregate report gains a Failures column (e.g. "PH:3 ST:1") so problems surface at a glance. Per-clip detail shows a Mode column; CSV adds three new fields. Mandarin corpus expanded from 2 to 8 clips (zh_001–zh_008) covering conversational speech, numerals, proper nouns, and instruction sentences — enough to distinguish failure rate from noise on a single-language bucket. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers issue #17 end to end: categorical failure-mode metrics (PR-A, this branch), then bracket-hallucination suppression + script-match retry fix (PR-B). Defines success criteria so the fix PR has measurable acceptance tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
larryxiao
added a commit
that referenced
this pull request
May 13, 2026
- SPEC-021 (Mandarin fix) promoted to 🟡 — PR #20 draft open - Per-app tone profiles added as ⚪ (#24), downstream of SPEC-007 - Launch at login added as ⚪ (#29), S-sized via SMAppService - SPEC-018 (send-feedback menu) moved to 🟢 — merged in #5 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
larryxiao
added a commit
that referenced
this pull request
May 16, 2026
alpha.11 composition. fn / Globe key (#28) and launch-at-login (#33 + #39) are fully shipped. The overlay multi-monitor fix (#27) had been merged without a 🟢 row — adding it now under SPEC-004 since it's the same spec. Adoption focus band remaining: SPEC-025 / SPEC-026 / SPEC-027 (specs in, implementations gated on the user's Developer Program + EdDSA + GIF capture), plus SPEC-021 Mandarin fix (PR #20 still draft), plus the two uncreated ⚪ items (awesome-list tracking, quarterly bench refresh). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SPEC: SPEC-021 (Mandarin auto-detect failure: benchmark coverage + fix)
Milestone: M1
Why
Issue #17 shows Mandarin audio transcribed as
[SPEAKING CHINESE]. WER/CER alonecan't distinguish three distinct failure modes (placeholder hallucination, silent
translation, garbled output) — they all show elevated numbers but require different
fixes. This PR adds the metrics we need to measure each mode independently and
establishes a baseline before any fix lands.
Change
Sources/OpenQuackBench/BenchRunner.swiftFailureModeenum:.placeholder,.silentTranslation,.garbled,.okClipMetricsgains three fields:failureMode,outputScriptMatch,hallucRateRawBenchResult.failureCountscomputed property for aggregate reportingclassifyFailureMode()— priority-ordered classifier:[SPEAKING CHINESE],[FOREIGN LANGUAGE], etc.)silentTranslationgarbledoutputScriptMatch()— true when reference and output share the dominant scripthallucRateRaw()— fraction of output chars inside[...]bracketsSources/OpenQuackBench/Report.swiftFailurescolumn showingPH:N ST:N G:Norall okModecolumn (✓for ok, bold name for failures)failure_mode,output_script_match,halluc_rate_rawbench/corpus/fetch.sh+bench/corpus/multilingual/zh_00{3-8}.txtsay -v Tingting, covering conversationalspeech, numerals, proper nouns, and instruction sentences
Tests
swift build && swift testgreen (CI: macOS-15 + Xcode, where#Previewmacros resolve)bash bench/corpus/fetch.shgenerateszh_003–zh_008WAVs cleanly ✓Out of scope
Closes part of issue #17 (baseline measurement); fix lands in PR-B.