Repository navigation
Qwen3-TTS PR 3: sentence chunking and error status for /v1/audio/speech - #300
Conversation
Gaurav-Shah05
left a comment
Gaurav-Shah05
left a comment
There was a problem hiding this comment.
One thing to fix: pieces submitted ahead aren't aborted when the stream ends early (below). On a live Qwen3-TTS server, closing a chunked stream cancelled the piece being read, and the two lookahead pieces ran to the end.
| if requested is None: | ||
| requested = getattr(req, "sentence_chunking", None) | ||
| min_chars = getattr(adapter, "speech_chunk_min_chars", None) | ||
| if requested is False or (requested is None and (min_chars is None or len(text) < min_chars)): |
There was a problem hiding this comment.
Any client can send sentence_chunking: true and nothing bounds input, so a 1.2 MB input becomes over 3,000 engine requests where the unchunked request stops at the model's max length. I think the override needs a cap.
There was a problem hiding this comment.
Added a speech_chunk_max_pieces to the adapter, which bounds the number of chunks and raises a 400 if it's exceeded.
| async for c in iterator: | ||
| # Mid-stream the status is already sent; closing the stream is the | ||
| # only honest signal left, so raise rather than end quietly. | ||
| _raise_if_error(c) |
There was a problem hiding this comment.
The request is already parked in recently_completed when its error chunk arrives, so raising here (and at the up-front _raise_if_error(first)) leaves iter_result_chunks early and its finally aborts a request that has already ended. #330 fixes the same abort in _stream_wav, so these two will conflict. I will rebase #330 once this merges.
There was a problem hiding this comment.
Now, the error is tracked and the stream is read to the end, with the error being raised after
|
|
||
| lookahead = max(1, int(getattr(adapter, "speech_chunk_lookahead", 2))) | ||
| if req.stream: | ||
| pending = [submit(i) for i in range(min(lookahead, len(chunks)))] |
There was a problem hiding this comment.
On a disconnect or a failed piece only the piece being read is aborted, so the lookahead pieces keep generating, and the non-streaming path goes on to submit and abort every remaining piece. I think the loop should stop on disconnect and abort every id still in pending in a finally.
There was a problem hiding this comment.
Now, both the streaming and non-streaming paths keep track of the number of pending chunks that have been processed and aborts the lookahead requests in finally
|
@Gaurav-Shah05 addressed the comments, can you take another look? |
3 participants
Long inputs to
/v1/audio/speechare synthesized as ordered sentence chunks.speech_chunking.split_sentencesgroups sentences (Latin and CJK terminators, paragraph breaks, over-long sentences wrap at clauses and then spaces) into pieces of aboutspeech_chunk_max_chars.serving_speechsubmits one engine request per piece, keepsspeech_chunk_lookaheadpieces in flight while the current one streams, and concatenates the PCM in order for both streaming and non-streaming responses. Each autoregressive request stays within the length its model was trained for, the engine batches the pieces like independent requests, and a client seed advances per piece.Opt-in per adapter (
speech_chunk_min_chars, off by default) and overridable per request withsentence_chunking: true|false. Adapters that keep the default behave as before. Qwen3-TTS turns it on for inputs of 600+ characters.Also in here: a streaming request now looks at its first engine result before answering 200, so a rejected request (bad voice, dead worker) comes back with its error status instead of an empty WAV, and a mid-stream error closes the stream.
Six commits, one per file. Tests in
test/modular/test_speech_chunking.pycover the splitter and the handler on the stubbed router (ordered ids and playback, per-chunk seeds, streaming concatenation, opt-out, forced chunking, error status). Fulltest/modularpasses.