Support Cosyvoice3 TRT-LLM Inference

[yuekaizhang]

Quick Start

docker compose -f docker-compose.cosyvoice3.yml up

Using a single L20 GPU:

Offline TTS (CosyVoice3 0.5B LLM + Token2Wav FP16 with TensorRT)

Backend	LLM Batch Size	llm_time (s)	token2wav_time (s)	pipeline_time (s)	RTF
TRTLLM	1	13.21	5.72	19.48	0.1091
TRTLLM	2	8.46	6.02	14.91	0.0822
TRTLLM	4	5.07	5.95	11.43	0.0630
TRTLLM	8	2.98	6.11	9.53	0.0562
TRTLLM	16	2.12	6.27	8.83	0.0501

Streaming TTS

First Chunk Latency

Concurrent Tasks	Average (ms)	50th Percentile (ms)	90th Percentile (ms)	95th Percentile (ms)	99th Percentile (ms)
4	750.42	740.31	941.05	977.55	1002.37

[whaozl]

👍👍👍