Skip to content

feat(pipeline_v2): optional dense embedding (sub-window per segment) - #25

Merged
ekhodzitsky merged 1 commit into
masterfrom
accuracy/dense-embed
Jun 22, 2026
Merged

ekhodzitsky merged 1 commit into
masterfrom
accuracy/dense-embed

Conversation

@ekhodzitsky

Copy link
Copy Markdown
Owner

What

Adds opt-in dense embedding for pipeline v2: PipelineConfig.embed_window_secs: Option<f32> (CLI --v2 --embed-window <secs>, bench --embed-window). Some(w) slides a w-second window (hop w/2) inside each primary segment so a speaker run produces several embeddings — like the legacy pipeline's dense windows — instead of one. More embeddings per speaker → more robust centroids → lower confusion. Pure compute, no extra model weights.

Why — measured (collar 0.25, macro DER)

The triangle (legacy / v2+AHC / v2+VBx) showed legacy wins clean VoxConverse largely because of its dense sliding-window embeddings (confusion 5.9 vs v2's 12.1). This closes that gap for v2.

set v2+AHC sparse v2+AHC + dense(1.5)
VoxConverse-30 16.4% (conf 12.1) 14.7% (conf 9.7)
VoxConverse-10 (harder) 19.8% 15.7%
AMI-4 (overlap) 32.8% 32.5% (beats legacy 35.5%)

w = 1.5 is the sweet spot — 1.0 over-fragments (18.7%, worse than sparse).

Honest scope

  • Real v2 improvement (VoxConverse −1.7pp, harder subsets −4pp, beats legacy on AMI).
  • Does not overtake legacy on clean VoxConverse (13.5%), and slightly hurts the VBx clusterer on AMI (VBx-sparse stays the meeting champion).
  • So it ships opt-in (None default = current behavior, one embedding per segment). It's a v2/AHC booster, not a default change.
  • Cost: more embedder calls (≈ legacy's density), so slower than sparse v2.

Tests

  • expand_embed_units unit tests (splits long segments, keeps short, None = identity, sub-windows inherit the local speaker index and end at the segment boundary).
  • CLI --help snapshot updated.
  • e2e verified via bench (--embed-window 1.5).
  • fmt clean · clippy --all-targets clean · 310 lib tests green.

Breaking (v2 API)

PipelineConfig gained embed_window_secs — struct-literal constructors add embed_window_secs: None (or use ..default()). Covered by the 0.9.0 minor bump.

🤖 Generated with Claude Code

Add PipelineConfig.embed_window_secs: Option<f32>. When Some(w), each primary
segment is split into overlapping w-second sub-windows (hop w/2) before
embedding, so a speaker run yields several embeddings (like the legacy
pipeline's dense windows) instead of one — more robust centroids, lower
confusion. Pure compute, no extra model weights.

Measured on v2+AHC (collar 0.25 macro): VoxConverse-30 16.4% → 14.7%
(confusion 12.1 → 9.7), and it beats legacy on overlap-heavy AMI. w=1.5 is the
sweet spot (1.0 over-fragments → worse). It does not overtake legacy on clean
VoxConverse (13.5%) and slightly hurts VBx on AMI, so it is opt-in (None
default = unchanged, one embedding per segment).

Wired through the v2 builder, `polyvoice diarize --v2 --embed-window <secs>`,
and `polyvoice-bench --embed-window`. Breaking: PipelineConfig gained a field
(add embed_window_secs: None or use ..default()).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 92.30769% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/bin/polyvoice.rs 25.00% 3 Missing ⚠️
src/bin/polyvoice-bench.rs 0.00% 1 Missing ⚠️
src/pipeline_v2/mod.rs 98.30% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ekhodzitsky
ekhodzitsky merged commit bad4bd1 into master Jun 22, 2026
44 checks passed
@ekhodzitsky
ekhodzitsky deleted the accuracy/dense-embed branch June 22, 2026 20:50
@ekhodzitsky ekhodzitsky mentioned this pull request Jun 22, 2026
ekhodzitsky added a commit that referenced this pull request Jul 31, 2026
…25)

Add PipelineConfig.embed_window_secs: Option<f32>. When Some(w), each primary
segment is split into overlapping w-second sub-windows (hop w/2) before
embedding, so a speaker run yields several embeddings (like the legacy
pipeline's dense windows) instead of one — more robust centroids, lower
confusion. Pure compute, no extra model weights.

Measured on v2+AHC (collar 0.25 macro): VoxConverse-30 16.4% → 14.7%
(confusion 12.1 → 9.7), and it beats legacy on overlap-heavy AMI. w=1.5 is the
sweet spot (1.0 over-fragments → worse). It does not overtake legacy on clean
VoxConverse (13.5%) and slightly hurts VBx on AMI, so it is opt-in (None
default = unchanged, one embedding per segment).

Wired through the v2 builder, `polyvoice diarize --v2 --embed-window <secs>`,
and `polyvoice-bench --embed-window`. Breaking: PipelineConfig gained a field
(add embed_window_secs: None or use ..default()).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ekhodzitsky added a commit that referenced this pull request Jul 31, 2026
…25)

Add PipelineConfig.embed_window_secs: Option<f32>. When Some(w), each primary
segment is split into overlapping w-second sub-windows (hop w/2) before
embedding, so a speaker run yields several embeddings (like the legacy
pipeline's dense windows) instead of one — more robust centroids, lower
confusion. Pure compute, no extra model weights.

Measured on v2+AHC (collar 0.25 macro): VoxConverse-30 16.4% → 14.7%
(confusion 12.1 → 9.7), and it beats legacy on overlap-heavy AMI. w=1.5 is the
sweet spot (1.0 over-fragments → worse). It does not overtake legacy on clean
VoxConverse (13.5%) and slightly hurts VBx on AMI, so it is opt-in (None
default = unchanged, one embedding per segment).

Wired through the v2 builder, `polyvoice diarize --v2 --embed-window <secs>`,
and `polyvoice-bench --embed-window`. Breaking: PipelineConfig gained a field
(add embed_window_secs: None or use ..default()).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants