Repository navigation
feat(bench): per-backend RTFx with per-stage runtime breakdown - #34
Merged
Merged
Conversation
The EP work is only provably valuable if realtime factor is reproducibly measurable per backend. polyvoice-bench recorded rt_factor but could not select an execution provider, attributed nothing to a backend, and gave no per-stage picture. - polyvoice-bench: --execution-provider auto|cpu|coreml|nnapi|cuda|xnnpack. Omitted keeps each pipeline's shipped default (legacy embedder cpu, v2 auto) so committed DER baselines stay reproducible; an explicit value applies to the selected pipeline. Reports carry resolved_execution_provider and host_cpus. - pipeline_v2: new StageTimings + Pipeline::run_with_timings (four Instant reads; run() delegates — public signature unchanged). The bench records stage_timings per file and a stage_totals aggregate for v2 runs; VAD has no separate stage in v2 (it lives inside the powerset segmenter). - scripts/bench-backends.sh: one dataset, one run per backend, one RTFx comparison table; extra bench flags via BENCH_ARGS. Verified on VoxConverse (3 files): cpu and CoreML rows are directly comparable (identical 11.78% micro DER, RTFx 11.17 vs 11.15) and the breakdown shows embedding as the dominant stage (33.9s vs 22.6s segmentation). - Report schema bumped to polyvoice-bench-v0.10, strictly additive over v0.8; src/bin/README.md documents the workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ekhodzitsky
added a commit
that referenced
this pull request
Jul 31, 2026
The EP work is only provably valuable if realtime factor is reproducibly measurable per backend. polyvoice-bench recorded rt_factor but could not select an execution provider, attributed nothing to a backend, and gave no per-stage picture. - polyvoice-bench: --execution-provider auto|cpu|coreml|nnapi|cuda|xnnpack. Omitted keeps each pipeline's shipped default (legacy embedder cpu, v2 auto) so committed DER baselines stay reproducible; an explicit value applies to the selected pipeline. Reports carry resolved_execution_provider and host_cpus. - pipeline_v2: new StageTimings + Pipeline::run_with_timings (four Instant reads; run() delegates — public signature unchanged). The bench records stage_timings per file and a stage_totals aggregate for v2 runs; VAD has no separate stage in v2 (it lives inside the powerset segmenter). - scripts/bench-backends.sh: one dataset, one run per backend, one RTFx comparison table; extra bench flags via BENCH_ARGS. Verified on VoxConverse (3 files): cpu and CoreML rows are directly comparable (identical 11.78% micro DER, RTFx 11.17 vs 11.15) and the breakdown shows embedding as the dominant stage (33.9s vs 22.6s segmentation). - Report schema bumped to polyvoice-bench-v0.10, strictly additive over v0.8; src/bin/README.md documents the workflow. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ekhodzitsky
added a commit
that referenced
this pull request
Jul 31, 2026
The EP work is only provably valuable if realtime factor is reproducibly measurable per backend. polyvoice-bench recorded rt_factor but could not select an execution provider, attributed nothing to a backend, and gave no per-stage picture. - polyvoice-bench: --execution-provider auto|cpu|coreml|nnapi|cuda|xnnpack. Omitted keeps each pipeline's shipped default (legacy embedder cpu, v2 auto) so committed DER baselines stay reproducible; an explicit value applies to the selected pipeline. Reports carry resolved_execution_provider and host_cpus. - pipeline_v2: new StageTimings + Pipeline::run_with_timings (four Instant reads; run() delegates — public signature unchanged). The bench records stage_timings per file and a stage_totals aggregate for v2 runs; VAD has no separate stage in v2 (it lives inside the powerset segmenter). - scripts/bench-backends.sh: one dataset, one run per backend, one RTFx comparison table; extra bench flags via BENCH_ARGS. Verified on VoxConverse (3 files): cpu and CoreML rows are directly comparable (identical 11.78% micro DER, RTFx 11.17 vs 11.15) and the breakdown shows embedding as the dominant stage (33.9s vs 22.6s segmentation). - Report schema bumped to polyvoice-bench-v0.10, strictly additive over v0.8; src/bin/README.md documents the workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the measurement leg of the execution-provider track: the EP work (CoreML today, XNNPACK/NNAPI/CUDA later) is only provably valuable if realtime factor is reproducibly measurable per backend.
What changed
polyvoice-bench --execution-provider <auto|cpu|coreml|nnapi|cuda|xnnpack>. Omitted = each pipeline's shipped default (legacy embeddercpu, v2auto), so committed DER baselines stay reproducible; an explicit value applies to the selected pipeline. Every report is labeled withresolved_execution_providerandhost_cpus.pipeline_v2::StageTimingsandPipeline::run_with_timings(...) -> (DiarizationResult, StageTimings)— fourInstantreads;run()delegates to it, public signature unchanged. Bench reportsstage_timingsper file +stage_totalsaggregate (segmentation / embedding / clustering / resegmentation; VAD lives inside the powerset segmenter in v2, so no fake zero field).scripts/bench-backends.sh— one dataset, one run per backend, one comparison table (RTFx, micro DER, runtime); extra bench flags viaBENCH_ARGS. This reproducibility is the task's deliverable.polyvoice-bench-v0.8→polyvoice-bench-v0.10, strictly additive (all existing fields unchanged). Workflow documented insrc/bin/README.md.Measured (deliverable proof, VoxConverse 3 files, M1)
Identical DER across backends (numerics unaffected), directly comparable RTFx, and the breakdown surfaces embedding as the dominant stage (33.9s vs 22.6s segmentation) — exactly the signal the EP track needs.
Verification
cargo nextest run --all-features: 425 passed (bench proptests included); clippy-D warnings+ fmt clean.stage_totals/resolved_execution_provider/ schema verified in the emitted JSON.🤖 Generated with Claude Code