[Roadmap] CPU Backend Optimization (2025 H2)

[mingfeima]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Continue optimize CPU backend on SGLang after #2807, we target at providing optimal performance for intel Xeon with AMX support. For CPUs without AMX, we will only guarantee functionality but not performance.

Single Node CPU Optimization

(deploy with single node CPU, with 1/2/4 sockets)

• Graph Mode Support: enable graph mode with torch.compile, to remove the python overhead. This roughly give extra 10% performance improvement for decoding, Add graph runner support with torch compile on CPU #7843.
• Prefill Optimization: add thread blocking and cache blocking for gemm and moe modules, the current implementations exist duplicate reads for weights for large input M, will give 50~80% improvements for prefilling phase. @mingfeima Optimize prefill performance on cpu backend #8750
• INT4 Support: 4bit mode is more important for computationally constrained hardware e.g. CPUs. Enable awq INT4 (w4a8), reuse frontend from CUDA implementation (also decouple with vLLM) and inject high performance kernels for CPU devices. [CPU][INT4] Add AWQ frontend support for CPU  #8225, [CPU][INT4] Add INT4 kernels for CPU  #8226.
• FP4 Support: follow WoQ pattern to enable FP4 support and run GPU model weights OOBox.
• FP8 KV Cache Support: enable usage of FP8 kv cache, fallback to compute with BF16. @shiyang-weng
• FP8 Bmm Support: enable FP8 BMM in MLA weight absorb, the current BMM is computed with BF16. As CPU hardware doesn't support FP8 computation, prefer to use WoQ (A:BF16, B:FP8, C:BF16). @blzheng the improvement for decoding will only be marginal, estimated 1.5% top.
• SplitK: GEMM with small OC (output channels, or N for GEMM) suffers from low performance since not be able to fully parallel, implement splitK for gate proj. @mingfeima The current gate proj only has ~50% memory bandwidth utilization.
• MTP Support: Enable Multi-Token Prediction @Yuxingwang-intel

Multi Node CPU Optimization

(deploy with multi node CPU, each with 1/2/4 sockets)

• Data Parallel MLA and Expert Parallel MoE: map DP-MLA and EPMoE from multi-GPUs to multi-CPUs @chunyuan-w @mingfeima @yanbing-j
• Distributed Collectives: expand collectives, e.g. broadcast, all-reduce to multi node. Evaluate use of oneCCL implementation from DeepSeed. @yanbing-j @blzheng

Hybrid MoE Offloading

(CPU handles MoE, GPU handles Attention, etc.)

• GPU-CPU Offloading: GPU and CPU run sequentially, @Quallyjiang @jianan-gu.
• GPU-CPU Pipelining: GPU and CPU run in parallel with a pipeling pattern.

Hybrid PD Disaggregation

(PoC stage)

• Scheduling with heterogeneous decoding nodes.

User Experience and Testing Enhancement

• Documentation: fulfill documentation, including installing, usage for prioritized models. [CPU] Add tutorial docs for SGL on CPU #8000
• PyPI Release: add sgl-kernel-cpu-amx release package, let user can install with pip @ZailiWang
• Model Coverage: constantly enable popular models, such as kimi-k2: [CPU] Fix TP padding case with weight block size #8243, [CPU][sgl-kernel] biased_grouped_topk: fix correction_bias dtype to float32 #8212.
• Proxy Model: use smaller proxy model to enable model level test on SGLang to enhance product quality on prioritized models, with a balance for test time consumption in CIs. @yanbing-j @DiweiSun
• Fallback Path Enabling: add fallback paths for non-AMX machines, Support MLA in Torch Native Attention Backend #3475
• Test on Non-AMX Machines: either use environment variables or provide new CI machines to test on non-AMX machines, to enhance product quality @DiweiSun
• Profiler Argument Enabling: add an argument from sglang CLI, -- profile to enable torch.profile and generate trace and logs. This would be very helpful for performance tunning. @yanbing-j
• Bug Tracking: actively track bugs from SGLang community @DiweiSun

Innovation

• super long context prefill/decode performance optimization
• expert load balancing with numa aware

Related resources

No response

[whatever1983]

Instead of FP8, running FP4 safetensors directly with CPU with either AMX or BF16 is going to be a kick ass feature:
NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4

[mingfeima]

Instead of FP8, running FP4 safetensors directly with CPU with either AMX or BF16 is going to be a kick ass feature: NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4

Wow, this really reminds me ... i was stuck with INT4 since spontaneously x86 doesn't support FP4 yet. However use FP4 with a WoQ pattern is absolutely doable, technically we can convert 4bit (INT4, or FP4) to INT8 and compute with amx-int8. And 4bit->8bit conversion can be done with shuffle intrisincs with a table lookup, this is much faster than FP8 conversion.

Thanks for the remainder, roadmap updated and definitely will have it implemented!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.