Skip to content

bench: categorical failure-mode metrics + Mandarin corpus expansion - #20

Draft
larryxiao wants to merge 2 commits into
mainfrom
worktree-feat+spec-021-zh-bench-metrics
Draft

bench: categorical failure-mode metrics + Mandarin corpus expansion#20
larryxiao wants to merge 2 commits into
mainfrom
worktree-feat+spec-021-zh-bench-metrics

Conversation

@larryxiao

Copy link
Copy Markdown
Owner

SPEC: SPEC-021 (Mandarin auto-detect failure: benchmark coverage + fix)

Milestone: M1

Why

Issue #17 shows Mandarin audio transcribed as [SPEAKING CHINESE]. WER/CER alone
can't distinguish three distinct failure modes (placeholder hallucination, silent
translation, garbled output) — they all show elevated numbers but require different
fixes. This PR adds the metrics we need to measure each mode independently and
establishes a baseline before any fix lands.

Change

Sources/OpenQuackBench/BenchRunner.swift

  • New FailureMode enum: .placeholder, .silentTranslation, .garbled, .ok
  • ClipMetrics gains three fields: failureMode, outputScriptMatch, hallucRateRaw
  • BenchResult.failureCounts computed property for aggregate reporting
  • classifyFailureMode() — priority-ordered classifier:
    1. Bracket annotation regex ([SPEAKING CHINESE], [FOREIGN LANGUAGE], etc.)
    2. CJK reference + Latin-dominant output → silentTranslation
    3. WER > 200% → garbled
  • outputScriptMatch() — true when reference and output share the dominant script
  • hallucRateRaw() — fraction of output chars inside [...] brackets
  • Unicode block helpers for CJK (Unified + Ext-A + Compat, Hangul, Hiragana, Katakana) and Latin

Sources/OpenQuackBench/Report.swift

  • Aggregate table: new Failures column showing PH:N ST:N G:N or all ok
  • Per-clip detail: new Mode column ( for ok, bold name for failures)
  • CSV: three new fields failure_mode,output_script_match,halluc_rate_raw

bench/corpus/fetch.sh + bench/corpus/multilingual/zh_00{3-8}.txt

  • Mandarin corpus: 2 → 8 clips via say -v Tingting, covering conversational
    speech, numerals, proper nouns, and instruction sentences

Tests

  • swift build && swift test green (CI: macOS-15 + Xcode, where #Preview macros resolve)
  • bash bench/corpus/fetch.sh generates zh_003zh_008 WAVs cleanly ✓
  • Bench delta: run before landing PR-B to capture baseline failure rates

Out of scope

  • The fix (token suppression, script-match retry) — PR-B, depends on this baseline
  • Japanese, Korean, and other-script failure tracking — follow-up once zh fix validated
  • Full multilingual Settings UI — M2/M2.5

Closes part of issue #17 (baseline measurement); fix lands in PR-B.

larryxiao and others added 2 commits May 10, 2026 07:40
…SPEC-021)

Addresses issue #17 where [SPEAKING CHINESE] hallucinations are invisible to
WER/CER alone — the score looks bad but can't tell the user what went wrong
or which fix to try.

Three new per-clip metrics classify each output categorically:
- failureMode: .placeholder (bracket annotations like [SPEAKING CHINESE]),
  .silentTranslation (CJK audio → Latin output), .garbled (WER > 200%), .ok
- outputScriptMatch: true when reference and hypothesis share the dominant script
- hallucRateRaw: fraction of output chars inside [...] brackets

Aggregate report gains a Failures column (e.g. "PH:3 ST:1") so problems surface
at a glance. Per-clip detail shows a Mode column; CSV adds three new fields.

Mandarin corpus expanded from 2 to 8 clips (zh_001–zh_008) covering
conversational speech, numerals, proper nouns, and instruction sentences —
enough to distinguish failure rate from noise on a single-language bucket.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers issue #17 end to end: categorical failure-mode metrics (PR-A,
this branch), then bracket-hallucination suppression + script-match
retry fix (PR-B). Defines success criteria so the fix PR has measurable
acceptance tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
larryxiao added a commit that referenced this pull request May 13, 2026
… ⚪ ideas

- SPEC-021 (Mandarin fix) promoted to 🟡 — PR #20 draft open
- Per-app tone profiles added as ⚪ (#24), downstream of SPEC-007
- Launch at login added as ⚪ (#29), S-sized via SMAppService

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
larryxiao added a commit that referenced this pull request May 13, 2026
- SPEC-021 (Mandarin fix) promoted to 🟡 — PR #20 draft open
- Per-app tone profiles added as ⚪ (#24), downstream of SPEC-007
- Launch at login added as ⚪ (#29), S-sized via SMAppService
- SPEC-018 (send-feedback menu) moved to 🟢 — merged in #5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
larryxiao added a commit that referenced this pull request May 16, 2026
alpha.11 composition. fn / Globe key (#28) and launch-at-login (#33 + #39)
are fully shipped. The overlay multi-monitor fix (#27) had been merged
without a 🟢 row — adding it now under SPEC-004 since it's the same spec.

Adoption focus band remaining: SPEC-025 / SPEC-026 / SPEC-027 (specs in,
implementations gated on the user's Developer Program + EdDSA + GIF
capture), plus SPEC-021 Mandarin fix (PR #20 still draft), plus the two
uncreated ⚪ items (awesome-list tracking, quarterly bench refresh).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant