Qwen3-TTS PR 5: 1.7B CustomVoice, VoiceDesign and Base (voice clone)

[merceod]

One Qwen3TTSModel class now serves every 12 Hz Qwen3-TTS checkpoint. The variant is read from config.json. Registry keys: qwen3_tts (0.6B CustomVoice, as before), qwen3_tts_1p7b, qwen3_tts_voicedesign, qwen3_tts_base.

Depends on #298, #299, #300 and #301 (their commits are cherry-picked here and will drop out on rebase once they merge).

Graph

API tensors: text_inputs, prompt_layout, speaker_id, language_id (+ audio_inputs, ref_frames for Base)

talker_prefill        Talker(text, layout, speaker, language) -> talker_input_embeds (persist), codec_tokens (stream)
talker_prefill_clone  RefEncoder(audio, layout) -> speaker_embed, ref_codes -> Talker(...)            [Base]
talker_decode         Loop(Talker(talker_input_embeds) -> talker_input_embeds, codec_tokens), stops on codec EOS / max frames
codec_chunk[_clone]   Codec(codec_tokens[, ref_frames]) -> audio_chunk (emitted to the client)

Partitions: Talker --codec_tokens, ScheduledLeftContextChunkPolicy((1,3,8,16), 25, 25)--> Codec
Resources (Talker): paged KV, FlashInfer attention, RoPE positions, two samplers (group 0 with repetition penalty, code predictor)

What changed vs the 0.6B port

• 1.7B CodePredictor small_to_mtp_projection (Talker 2048 -> predictor 1024) and the reference's prompt layouts (instruct turn, speaker slot, whole-text vs streamed text, in-context clone).
• Base voice clone. ECAPA x-vector + Mimi codec frames from the reference clip, memoised by clip content so a reused voice (or the sentence chunks of one long request) is encoded once. Only the last 25 reference frames go into the codec stream, which is all a window can use as context.
• Codec streaming. Chunk ramp 1 | 1+3 | 4+8 | 12+16 | 25+25 (first audio one Talker step after prefill, the decoder is causal so chunking does not change the audio), one CUDA graph per window bucket x batch 1..32, requests at different points of the ramp batch together (padded to the widest window). Each pass carries its own trim geometry, and the single-request forward returns per-request samples. That last one fixes truncated audio under load, see below.
• Talker prefill replays a packed CUDA graph (token buckets 32..1024). Decode already did.
• /v1/audio/speech adapter (voice, instructions, language, ref_audio / ref_text / x_vector_only_mode), sentence chunking for inputs of 600+ characters, error status instead of an empty WAV on rejected requests. SDK client.tts(..., reference_audio=...).
• Deployment knobs. model_kwargs: {codec_dtype: bfloat16} (fp32 default keeps the codec bit-exact) and configs/qwen3tts_1p7b_split.yaml (codec on a second worker on the same GPU).
• Weight loading checks coverage in both directions.

Parity vs qwen-tts 0.1.1 (teacher-forced logits, 63 frames, bf16 Talker, fp32 codec)

Talker and CodePredictor logits are within 0.5-0.8% of the logit scale and agree on 100% of confident positions (reference top-2 margin > 1) for all six modes. The codec is bit-exact against the fp32 reference. Raw argmax agreement and identical greedy frames are reported but not gated, bf16 near-ties flip them.

checkpoint	mode	frames	Talker argmax / confident agreement / rel. logit diff	CodePredictor argmax / confident agreement / rel. logit diff	identical greedy frames	codec max-abs-diff	greedy audio max-abs-diff	clone (x-vector cosine, ref-code agreement)
Qwen3-TTS-12Hz-0.6B-CustomVoice	0p6b_customvoice	63	0.9841 / 1.0000 / 0.0078	0.9407 / 1.0000 / 0.0059	2/63	0.00e+00	1.277	-
Qwen3-TTS-12Hz-1.7B-Base	1p7b_base_icl	63	1.0000 / 1.0000 / 0.0051	0.9862 / 1.0000 / 0.0057	4/63	0.00e+00	0.642	cos 1.0000, codes 0.514
Qwen3-TTS-12Hz-1.7B-Base	1p7b_base_xvec	63	0.9841 / 1.0000 / 0.0055	0.9630 / 1.0000 / 0.0044	4/63	0.00e+00	0.615	cos 1.0000, codes nan
Qwen3-TTS-12Hz-1.7B-CustomVoice	1p7b_customvoice	63	0.9683 / 1.0000 / 0.0055	0.9450 / 1.0000 / 0.0053	1/63	0.00e+00	0.578	-
Qwen3-TTS-12Hz-1.7B-CustomVoice	1p7b_customvoice_instruct	63	1.0000 / 1.0000 / 0.0059	0.9376 / 1.0000 / 0.0053	2/63	0.00e+00	0.645	-
Qwen3-TTS-12Hz-1.7B-VoiceDesign	1p7b_voicedesign	58	1.0000 / 1.0000 / 0.0058	0.9368 / 1.0000 / 0.0083	2/58	0.00e+00	0.745	-

Base clone codes match the fp32 reference encoder bit for bit (the qwen-tts package runs its encoder in bf16 and agrees with its own fp32 run on 51% of codes).

Benchmarks (one H100, same node, back to back, 200 sentences x 3 repeats, 3 warmup requests)

system	c	TTFA p50 / p95 ms	RTF	audio-s / s	WER %	errors
mstar_1p7b	1	22 / 24	0.089	11.4	0.44	0
mstar_1p7b	8	63 / 99	0.185	43.0	0.41	0
mstar_1p7b	32	153 / 218	0.404	76.4	0.50	0
mstar_1p7b_split_mps_bf16codec	1	25 / 27	0.101	9.9	0.41	0
mstar_1p7b_split_mps_bf16codec	8	44 / 68	0.175	45.4	1.25	0
mstar_1p7b_split_mps_bf16codec	32	70 / 149	0.300	100.5	0.53	0
vllm_omni_1p7b	1	27 / 29	0.085	11.8	0.37	0
vllm_omni_1p7b	8	49 / 79	0.125	63.2	0.47	0
vllm_omni_1p7b	32	135 / 231	0.254	119.7	1.03	0
sglang_omni_1p7b	1	24 / 26	0.097	10.4	0.41	0
sglang_omni_1p7b	8	40 / 53	0.135	58.5	0.53	0
sglang_omni_1p7b_adm32	32	56 / 201	0.234	127.9	0.94	0

• Audio completeness was checked for every row (last-repeat WAVs average 6.8-7.0 s at every concurrency for every engine). WER is whisper-large-v3-turbo on those WAVs. The default M* rows stay at 0.4-0.5%, the baselines go up to 0.9-1.0% at c=32.
• First audio. M* has the lowest p50 at c=1 and the lowest p95 at every concurrency. At c=8/32 its p50 sits between SGLang-Omni's and vLLM-Omni's.
• Throughput at c>=8 is where the baselines win. Per-request stats show the worker is CPU-bound there, a Talker decode step spends 6.5-9 ms of CPU in the engine launch path (plan/admit, sampler buffers, staging, replay) against ~8 ms of GPU time, plus codec passes on the same loop. The 0.6B checkpoint lands at the same c=8 ceiling (46 audio-s/s). The split-worker deployment lowers latency, and under a user-level CUDA MPS daemon it also overlaps the two workers' kernels, which is the 100 audio-s/s row.
• sglang_omni_1p7b at c=32 with its default recipe rejected 160 of 600 requests with 503 (16 running + 16 queued), so the c=32 row uses --tts_engine.engine.max_running_requests 32 --tts_engine.engine.max_queued_requests 64.
• Served smoke with the final code: CustomVoice warm first audio 22 ms (26 ms with an instruction, 32 ms for a 41 s sentence-chunked input), VoiceDesign 23 ms, Base clone 106-112 ms, bad voice / empty input give 400, seeded runs repeat byte for byte.

All 27 rows (fp32 / bf16 codec, single worker / split / split + MPS, 0.6B, SGLang-Omni default recipe)

System (version)	concurrency	TTFA p50 / p95 ms	RTF	audio-s / s	WER %	errors	notes
mstar_0p6b mstar 643d958	1	22 / 23	0.084	12.0	0.59	0	200 sentences x 3 repeats
mstar_0p6b mstar 643d958	8	62 / 90	0.174	46.0	0.50	0	200 sentences x 3 repeats
mstar_1p7b mstar db3427e	1	22 / 24	0.089	11.4	0.44	0	200 sentences x 3 repeats
mstar_1p7b mstar db3427e	8	63 / 99	0.185	43.0	0.41	0	200 sentences x 3 repeats
mstar_1p7b mstar db3427e	32	153 / 218	0.404	76.4	0.50	0	200 sentences x 3 repeats
mstar_1p7b_bf16codec mstar db3427e	1	22 / 24	0.088	11.4	0.37	0	200 sentences x 3 repeats
mstar_1p7b_bf16codec mstar db3427e	8	61 / 87	0.177	44.9	0.44	0	200 sentences x 3 repeats
mstar_1p7b_bf16codec mstar db3427e	32	129 / 215	0.359	84.9	0.66	0	200 sentences x 3 repeats
mstar_1p7b_split mstar db3427e	1	22 / 24	0.094	10.7	0.41	0	200 sentences x 3 repeats
mstar_1p7b_split mstar db3427e	8	53 / 78	0.198	40.5	1.34	0	200 sentences x 3 repeats
mstar_1p7b_split mstar db3427e	32	115 / 198	0.432	70.8	0.34	0	200 sentences x 3 repeats
mstar_1p7b_split_bf16codec mstar db3427e	1	22 / 24	0.093	10.8	0.75	0	200 sentences x 3 repeats
mstar_1p7b_split_bf16codec mstar db3427e	8	52 / 72	0.191	41.6	1.15	0	200 sentences x 3 repeats
mstar_1p7b_split_bf16codec mstar db3427e	32	109 / 183	0.399	76.6	0.66	0	200 sentences x 3 repeats
mstar_1p7b_split_mps mstar 643d958	1	25 / 27	0.102	9.9	0.50	0	200 sentences x 3 repeats
mstar_1p7b_split_mps mstar 643d958	8	45 / 70	0.181	44.0	0.62	0	200 sentences x 3 repeats
mstar_1p7b_split_mps mstar 643d958	32	75 / 151	0.335	91.0	0.50	0	200 sentences x 3 repeats
mstar_1p7b_split_mps_bf16codec mstar 643d958	1	25 / 27	0.101	9.9	0.41	0	200 sentences x 3 repeats
mstar_1p7b_split_mps_bf16codec mstar 643d958	8	44 / 68	0.175	45.4	1.25	0	200 sentences x 3 repeats
mstar_1p7b_split_mps_bf16codec mstar 643d958	32	70 / 149	0.300	100.5	0.53	0	200 sentences x 3 repeats
sglang_omni_1p7b sglang-omni 0.1.5	1	24 / 26	0.097	10.4	0.41	0	200 sentences x 3 repeats
sglang_omni_1p7b sglang-omni 0.1.5	8	40 / 53	0.135	58.5	0.53	0	200 sentences x 3 repeats
sglang_omni_1p7b sglang-omni 0.1.5	32	1176 / 1429	0.352	88.5	0.76	160	200 sentences x 3 repeats
sglang_omni_1p7b_adm32 sglang-omni 0.1.5	32	56 / 201	0.234	127.9	0.94	0	200 sentences x 3 repeats
vllm_omni_1p7b vllm-omni 0.28.0	1	27 / 29	0.085	11.8	0.37	0	200 sentences x 3 repeats
vllm_omni_1p7b vllm-omni 0.28.0	8	49 / 79	0.125	63.2	0.47	0	200 sentences x 3 repeats
vllm_omni_1p7b vllm-omni 0.28.0	32	135 / 231	0.254	119.7	1.03	0	200 sentences x 3 repeats

Reproduce:

mstar serve qwen3_tts_1p7b --gpus 0 --port 8000                      # or --config configs/qwen3tts_1p7b_split.yaml
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --deploy-config <site-packages>/vllm_omni/deploy/qwen3_tts.yaml --port 8002
sgl-omni serve --model-path Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --config examples/configs/qwen3_tts_1_7b_customvoice.yaml --port 8004
python -m benchmark.tts_speech_bench --engine {mstar|vllm-omni|sglang-omni} --url http://127.0.0.1:<port> --model <id> \
    --sentences sentences_200.txt --voice vivian --language English --concurrency {1|8|32} --repeats 3 --warmup 3 \
    --label <label> --out <label>_c<N>.json --save-audio-dir <label>_c<N>_wav
python -m benchmark.tts_wer --audio-dir <label>_c<N>_wav --sentences sentences_200.txt --out <label>_c<N>_wer.json
python -m benchmark.tts_report --results <dir> --out REPORT.md
python test/qwen3-tts/parity_qwen3_tts.py --repo Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --frames 64 [--instruct "..."]
python test/qwen3-tts/parity_qwen3_tts.py --repo Qwen/Qwen3-TTS-12Hz-1.7B-Base --frames 64 --voice "" --language auto --ref-audio clone.wav --ref-text "..."
python test/qwen3-tts/smoke_qwen3_tts.py --url http://127.0.0.1:8000 --variant {custom_voice|voice_design|base}

Known gaps

• Throughput at c>=8. Engine-level follow-ups, in order of expected gain: reuse the decode attention plan across steps, batch the sampler's per-request seen-token updates and the per-request output stores, run codec passes on a second CUDA stream inside the worker, and a native incremental codec decoder on the recurrent-state pool ([hold until after rust transition] Gdn and qwen3 5 #258) to drop the 25-frame left-context recompute.
• The clone prefill stays eager (it also feeds the reference tail into the codec stream), so Base first audio is ~110 ms vs 22 ms for CustomVoice.
• Named, persisted voices wait for the shared voice registry. Reference clips are per request (memoised by content).
• Greedy decoding of Base clone prompts degenerates into repeated near-silent frames in the reference implementation too. Sampling is the supported mode and max_new_tokens (8192, the checkpoint default) bounds a runaway request.
• Server start is about 180 s, the decode graph's torch.compile runs for ~2 min before any capture line is logged.

References read, not copied: qwen_tts 0.1.1 modeling/inference files, vllm_omni qwen3_tts prompt builder + deploy/qwen3_tts.yaml, sglang_omni qwen3_tts request builders, streaming vocoder and cookbook.