Has anyone successfully reproduced the throughput result?

[HsChen-sys]

Using model relaxml/Llama-2-7b-E8PRVQ-4Bit
On A6000, I only got ~82 toks/s which doesn't match 95 toks/s in the paper.
On 6000 Ada, I got ~109 toks/s while in the paper it's 140 toks/s.

command: python eval/eval_speed.py --hf_path relaxml/Llama-2-7b-E8PRVQ-4Bit

I also tried python interactive_gen.py --hf_path relaxml/Llama-2-7b-chat-E8PRVQ-4Bit but the throughput is strangely slow, only at 5.77 toks/s

The 3rd party repository QuIP-for-all contains some bugs so couldn't be run.

[tsengalb99]

Try python interactive_gen.py --hf_path relaxml/Llama-2-7b-chat-E8PRVQ-4Bit --compile

[tsengalb99]

Also, where are you getting 95 tok/s from?

[HsChen-sys]

Sorry for not making this clear. The data is from QTIP. QTIP's code doesn't provide a fused qkv_proj and it is recommended to used Quip# for performance benchmarking.

[tsengalb99]

This repo (quip-sharp) has kernels and inference code for QuIP#, which is different from QTIP. If you want to benchmark QTIP with matrix fusion you'll need to implement that yourself and quantize models accordingly (or just generate garbage weights if all you care about is benchmarking speed). To do that, you'll need to replace q_proj, k_proj, and v_proj in https://github.com/Cornell-RelaxML/qtip/blob/main/model/llama.py with a single qkv_proj and likewise for up_proj and gate_proj. The llama.py code in this repo already does that and has a wrapper class (https://github.com/Cornell-RelaxML/quip-sharp/blob/main/lib/linear/fused_quantized_linear.py) for splitting the output, so you can probably copy that over to QTIP pretty easily.