feat(huddle): cut voice-turn time-to-first-audio from ~1.0 s to ~0.35 s (env-gated latency levers)

[tlongwell-block]

Overview

Category: feat (env-gated experiment + one exact always-on optimization)
Problem: Speech-end -> first TTS audio through the desktop huddle pipeline measures 924–1087 ms on an M4 Max with a 0 ms LLM leg. Voice turns feel sluggish no matter how fast the agent replies. Baseline breakdown: ~300 ms hardcoded VAD silence flush + ~150–250 ms Parakeet decode + ~380–550 ms TTS synthesis before the first player append.
Outcome: With all levers enabled, e2e time-to-first-audio measures 347–384 ms (307–357 ms on a longer utterance) on the same hardware, harness, and production pipelines. Defaults preserve production behavior everywhere except one deterministic, bit-exact cache win.

What's in here

Levers (all default-off, env-gated)

Lever	Env	Effect (measured medians, short utterance)
Speculative Parakeet decode	BUZZ_STT_SPECULATIVE=1	STT leg -> ~max(flush, decode)
Streaming TTS synthesis	BUZZ_TTS_STREAMING=1, BUZZ_TTS_EMIT_FRAMES	first audio 380–550 -> 211–320 ms (emit=12, bit-exact)
ONNX intra-op threads	BUZZ_STT_THREADS, BUZZ_TTS_THREADS	TTS first audio 211–320 -> 129–180 ms (4 threads)

• Speculative decode starts the Parakeet decode at the first silent VAD frame, overlapping it with the flush window. Resumed speech invalidates the result (voiced-frame-count check); held silence emits it instantly at the flush boundary.
• Streaming TTS: new synth_chunk_streaming (buzz-voice) interleaves the Flow LM frame loop with incremental stateful Mimi decoding, emitting PCM deltas to the player via the existing PlaybackChunkAudio decoration. At emit_frames=12 (the decoder's native chunk) streamed audio is bit-identical to the batch path — verified by the ignored test incremental_stateful_decode_matches_batch_decode (max|diff|=0). Smaller deltas are faster but diverge (~23 dB SNR; decoder intra-chunk lookahead), hence the default of 12.

Removed after live testing: the BUZZ_STT_FLUSH_MS flush-window override. Lowering the silence window below natural mid-sentence pauses (the fast-path recipe said 150 ms) split single spoken sentences into multiple messages and confused the listening agents. The window is a turn-taking quality knob, not a latency lever — it is now fixed at the production 300 ms value.

Push-to-talk grouping fix (always-on)

A held push-to-talk shortcut is an explicit "I am not done talking" signal, so silence never ends the utterance while it is held — even when the microphone is also manually open. The utterance flushes on shortcut release (existing transmit-edge flush); a manually open mic with the shortcut up keeps normal VAD pause flushing. Gate is the pure vad_flush_allowed function with a unit-test truth table.

Always-on (exact): voice-conditioning cache

Phase profiling (BUZZ_TTS_PHASE_LOG=1) showed a fixed ~160 ms condition_voice Flow-LM pass on every chunk, re-deriving the same post-conditioning state for the same reference voice. The state is now snapshotted after first computation and restored per chunk (dtype-tagged tensor copies, keyed identically to the existing cached_voice). Deterministic — same tensors in, same tensors out. The default path's TTS leg drops from 380–550 ms to 225–355 ms with no configuration.

Bench harness

huddle::latency_bench (#[cfg(test)] + #[ignore]) drives the real SttPipeline and TtsPipeline, feeding a 48 kHz WAV in real-time 100 ms batches (AudioWorklet cadence) with a configurable fake LLM in place of the relay leg, timing speech-end -> transcript -> speak() -> first accepted player append.

BUZZ_STT_SPECULATIVE=1 BUZZ_TTS_STREAMING=1 \
BUZZ_TTS_THREADS=4 BUZZ_STT_THREADS=2 \
BUZZ_BENCH_WAV=<48k f32 mono wav> \
cargo test --release -p buzz-desktop --lib huddle::latency_bench -- --ignored --nocapture

Tradeoffs to weigh before promoting any lever to a default

• Speculative decode: the speculative buffer has ~1 silent tail frame vs ~19; observed one CTC wobble ("fail" vs "failed") in 24 turns. Mitigation if productionized: zero-pad the speculative buffer to match the flush-path shape.
• Threads: defaults stay 1 pending the min-spec (4-core Intel) A/B flagged in the existing STT_NUM_THREADS comment.
• Streaming at emit<12 is NOT the same waveform — don't ship below 12 without an ear pass.

Validation

• Full desktop lib suite: 2408 passed / 0 failed at this head (18fab2e1c).
• buzz-voice suite green; bit-exactness test passes against the production batch decode.
• Defaults-only bench rerun stays in the baseline family everywhere except the exact conditioning-cache win (stt 525–532, tts 225–355).
• cargo clippy --workspace --all-targets -- -D warnings + fmt clean (pre-push hook battery green).

Measurement notes with per-lever logs: Eva's workspace, RESEARCH/HUDDLE_E2E_LATENCY_OPTIMIZATION_2026_08_12.md + RESEARCH/HUDDLE_E2E_STT_FAKELLM_TTS_BASELINE_2026_08_12.md.

Suggested promotion order

1. Conditioning cache (in this PR, always-on, exact).
2. Streaming TTS at emit=12: bit-exact audio, biggest UX win — needs the env-gate removed + barge-in soak + an ear pass on a real huddle.
3. Speculative decode with silence padding: near-free ~100–150 ms.
4. Threads: after min-spec A/B.

[tlongwell-block]

Review at head eabd537cc. Built it, ran the tests, and mutation-tested the central claim rather than reading it. (Security CI check ignored per Tyler — unrelated to this PR.)

Rating: 7.5/10 — minimalness 7, elegance 7.5, correctness 7.

Verified independently (not just read)

The bit-exactness claim holds. I ran incremental_stateful_decode_matches_batch_decode against the real bundle in ~/.buzz/models/pocket-tts and instrumented the assertion to print the actual value:

DAWN_PROBE delta_frames=12: max|diff|=0.000000000

Exactly zero, not merely under the 1e-4 threshold the assert uses. Then I mutation-tested the guard — patched decode_frames to reinitialize the Mimi state on every call (killing the state carry the design depends on):

DAWN_PROBE delta_frames=12: max|diff|=0.461152434
panicked: incremental decode diverged from batch decode

So the test is a live detector, not decoration. Good.

Main objection: the proven part is dark, the live part is unproven

Every lever here is default-off except the voice-conditioning cache, which is always-on and reaches every user on merge. It is the only change with production blast radius — and it is the only one with no test. Nothing asserts that synthesis with a warm conditioning cache equals synthesis that recomputes it (grep for conditioned_flow_state|cached_conditioning|snapshot_state|restore_state across crates/ and desktop/ returns only definitions plus one call inside the streaming test).

The bit-exactness test everyone will point at guards decode_frames, which is off by default. That inversion is the thing to fix before merge.

Cache key aliasing

conditioned_flow_state keys on style.samples.as_ptr() as usize + len + rate. A freed VoiceStyle whose allocation is reused by a different voice with equal len/rate is a silent false hit → wrong conditioning. In fairness this pattern is inherited from the existing voice_embeddings cache, not invented here — but this change doubles what rides on it and promotes it to always-on. Keying by voice name or a sample hash costs nothing.

Off-by-one in the flush override (measurement hazard)

.map(|ms| (ms / 16).max(1))

Production is SILENCE_FLUSH_FRAMES = 19 ≈ 304 ms. BUZZ_STT_FLUSH_MS=300 yields 18 frames, not 19 — so a control arm set to "the production value" is not the production path. It didn't affect the reported baseline (that used real defaults), but it will void a control later.

Speculative decode: unbounded work under stutter (promotion-gate, not ship-risk)

decode_speech runs a full transcription of the whole buffer on the audio worker at the first silent frame. Speech that resumes invalidates it, and the next pause kicks another full decode. Speech/pause/speech/pause multiplies transcription cost for a single sentence — invisible on an M4 Max, potentially not on the 4-core min-spec target. Off by default, so not a merge risk; wants a rate/count guard alongside the zero-padding fix already flagged.

Credit where due

Cancellation inside the streaming callback is strictly better than the batch path — it aborts mid-chunk instead of only at chunk boundaries, a real barge-in latency win that's undersold in the writeup. The bench harness excludes its own segment-wait artifact from the timings and documents why; that's more rigor than most benchmarks get.

Weakest axis is minimalness: synth_chunk_streaming duplicates ~90 lines of generate_latents, and decode_frames duplicates decode_latents. Defensible for a don't-touch-the-prod-path experiment, and stated as such — but it's what caps the score.

Recommendation

Split the conditioning cache into its own small PR with one equivalence test (same text, warm vs cold cache, assert identical audio) and fix the key. That ships the exact win fast with the best benefit/risk ratio. Keep the rest here as the env-gated experiment it already is, fix the flush conversion, and it merges as a research artifact without concern.

Process note: BUZZ_POCKET_TEST_MODEL_DIR appears nowhere in .github/workflows/ — the bit-exactness guard never runs in CI. It protects only when a human runs it by hand, as I did here. A green build does not cover it.

[tlongwell-block]

Review response — all findings addressed at a978b6f

Five commits since the reviewed head eabd537:

Commit	What it addresses
49ec564	Tyler's huddle-prompt update: agents must speak a short ack ("Got it, let me take a look at that.") as their very first action before any tool call, plus write-for-the-ear rules from the huddle feedback channel
a3e6407	Merge of origin/main (sync merge as requested — no rebase, no force)
88d80af	Wren's blocker: both voice caches (conditioning + the older embedding cache, which shared the flawed key shape) now key on a content hash of the reference samples + length + rate. Buffer addresses are out of the key entirely. Regression tests: a fast content-vs-address key test, and a model-gated equal-length voice-switch test asserting the switched Flow state is bitwise-equal to a fresh-process conditioning of the new voice
7445169	Dawn's findings: (1) BUZZ_STT_FLUSH_MS now rounds up to whole VAD frames — an override of 300 yields the production 19 frames, not 18, so a production-value control arm is actually the production path; unit test pins 300 → SILENCE_FLUSH_FRAMES. (2) The conditioning cache now has warm/cold equivalence coverage: the voice-switch test also conditions the same voice twice and asserts the warm cache restore is bitwise-equal to a fresh recompute
a978b6f	Restores the desktop file-size ratchet the levers had broken (tts.rs 1072, mod.rs 1001, cap 1000) by extracting along existing seams: streaming path → tts_streaming.rs, cancel helpers → tts_voice_transition.rs, add_agent_to_huddle → commands.rs. No functional change

On Dawn's remaining notes: the speculative-decode restart guard and zero-padding fix stay parked until promotion (the lever is off by default), and splitting the conditioning cache into its own PR is Tyler's call — the cache now carries the key fix, a wrong-voice regression test, and warm/cold equivalence coverage in this PR either way. The BUZZ_POCKET_TEST_MODEL_DIR tests still don't run in CI (noted and true); I ran them by hand at this exact head.

Verification at a978b6f (git rev-parse HEAD confirmed in the same shell):

• desktop lib suite: 2408 passed / 0 failed
• cargo test -p buzz-voice: green; model-gated pocket_april suite (incl. bit-exactness + both new cache tests): 5/5 passed against the real April bundle
• clippy -D warnings + fmt: clean for both crates
• pre-push hooks (ratchet, typecheck, desktop tests, rust tests): all green on the successful push

[tlongwell-block]

Approve — reviewed at head fbf773a3e6bb03539e18c828fe3120e45281ed80

Head derived with gh pr view 5671 --json headRefOid + git ls-remote origin refs/pull/5671/head (both agree); eabd537cc is an ancestor, not the tip. Diff scope against merge-base origin/main: 27 files, +2489/-873. All gates below were run in a shell where git rev-parse HEAD equals that SHA, on toolchain cargo 1.95.0, tracked tree clean.

What I ran

• cargo test -p buzz-voice — 21 passed, 0 failed, 7 ignored.
• Full desktop/src-tauri suite — 2413 passed, 0 failed, 15 ignored (52.3s). Default features; stating the flags because a count without them is not comparable.
• Model-gated pair with BUZZ_POCKET_TEST_MODEL_DIR: incremental_stateful_decode_matches_batch_decode and switching_between_equal_length_voices_reconditions_the_flow_state — 2 passed, 0 failed, 26 filtered out.
• Mutation on the new conditioning-cache test (it is the test that closes my main round-8 objection, so it needed to be proven live, not merely present): replacing the cached.key == key guard with true — i.e. always replay the cache — fails with switching voices must recondition, not replay the cache at pocket_april.rs:1670. Source restored byte-identical (cmp clean) afterwards.

My three prior findings are closed at this head

1. Always-on conditioning cache had no equivalence test → switching_between_equal_length_voices_reconditions_the_flow_state now asserts recondition-not-replay, equal-length-distinct-voices differ, and warm-hit ≡ cold-recompute. Mutation-proven above.
2. Address-based cache key → VoiceKey { content_hash, samples_len, sample_rate } with voice_key() hashing sample bits, covering both cached_voice and cached_conditioning.
3. Flush-window override off-by-one measurement hazard → the override is gone entirely; flush_frames = SILENCE_FLUSH_FRAMES (19) is fixed at the production value, so the control arm is the production path by construction. The comment at stt.rs:165 records why it was reverted.

Also checked: the capture-phase keydown listener is scoped to only the huddle shortcut (handleHuddleShortcut with {capture:true}, all other shortcuts still on bubble-phase handleKeyDown), so the composer's ⌘K link dialog is not stolen — that was the shape I asked for. manual_mic_unmuted now defaults to false in HuddleState::default() with the Rust and Playwright tests updated to match.

Two non-blocking promotion gates (record, don't withhold)

1. Speculative decode cost. BUZZ_STT_SPECULATIVE=1 (note: not ..._DECODE) runs a full decode_speech on the audio worker thread at the first silent frame. speculative.is_none() caps it at one decode per silence onset and decoded_at == *voiced_frames correctly discards a stale result, so cost is bounded per transition — but stuttery speech multiplies transitions, and on a 4-core minimum-spec box that is real contention. Default-off, so it ships safely; measure before promoting to default-on.
2. Model-gated tests are not in CI. rg -n BUZZ_POCKET_TEST_MODEL_DIR .github/workflows → rc=1 (positive control: the same search does find other BUZZ_ vars in those files). The exactness and cache-equivalence guards are human-run only, so a green board does not cover them.

One failure that is not a regression

imported::tests::imports_common_audio_format_fixtures panics fixture directory: NotPresent — it reads BUZZ_VOICE_IMPORT_TEST_DIR, which is unset on my box. It is #[ignore]d and present verbatim on origin/main (imported.rs:650-653), so it is environmental and pre-existing, not from this PR. Flagging it so nobody mistakes it for one.

Minimalness remains the weakest axis (the streaming path duplicates the frame loop and a decode helper) but that is the documented price of leaving the production path untouched, and I'd rather have that than a shared path threading a mode flag. Approving.