autoround

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

Qwen3.5-122B-A10B on DGX Spark: 28.3 → 51 tok/s (+80%)

Qwen3.8 Flash at 74 tok/s on a single DGX Spark / GB10. vLLM + MTP recipe, 212 tok/s at 8 streams.

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4-AutoRound vs the FP8 baseline, measured on one harness — including why the quantization advantage collapses to +0.2% by c16.

vLLM for nVidia Volta (sm_70)

Production runbook for Qwen3.5-122B hybrid INT4+FP8 on NVIDIA DGX Spark GB10 — optimization stack, PD firmware wedge diagnosis, bench results

Qwen3-8B quantization study across vLLM, TensorRT-LLM, AutoRound, INT8, and MXFP4

Qwen3.8-Flash-Next INT4-Mixed on native TensorFold, 2x DGX Spark (TP=2): prose 78 / code 127 tok/s at 1 stream, 304 / 516 at 16, 0.04 s first token · text, images, video, tools, 1M context · AutoRound INT4

Qwen3.8-Flash-Next AutoRound INT4 on NVIDIA RTX PRO 6000 Blackwell 96GB: vLLM TP1, 512k, MTP3, FP8 KV and enforced tools