LLaMA-3 support and questions

[catid]

This seems like one of the best options for quantization for the important new LLaMA-3 70B model so that it can be run on 1-2 consumer grade GPUs. However it looks like support for MQA is not present in llama.py so it will not work I think.

Are you planning to add support for LLaMA-3?

[catid]

Looks like you have a fork of transformers code in llama.py for some reason.. is that because you had to patch some stuff?

[catid]

Currently working on generating offline Hessians for LLaMA3-8B

[tsengalb99]

I haven't taken a close look at the updated llama.py files for Llama 3 but I'd imagine the projection layers still work the same. I don't think there should be any off behavior with MQA but I'll find out when I try quantizing Llama 3 in the coming weeks. If you want to patch the current codebase to work with Llama 3 the easiest thing to do is probably upgrade to the latest transformers, copy over the llama.py file, and change all the linear layers to QuantizedLinear layers without doing any fusing (what we do in the current llama.py). You'll also get working cuda graph support so inference should be a lot faster than it is now. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Chris Taylor ***@***.***> Sent: Monday, April 22, 2024 1:59:30 AM To: Cornell-RelaxML/quip-sharp ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [Cornell-RelaxML/quip-sharp] LLaMA-3 support and questions (Issue #53) Currently working on generating offline Hessians for LLaMA3-8B — Reply to this email directly, view it on GitHub<#53 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6WZSHOKVBNSHT36L3IZADY6SRMFAVCNFSM6AAAAABGQYSAXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYGU2TCMJYGE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[catid]

@tsengalb99 What should devset_size be for the 70B model for calculating the Hessians?

[tsengalb99]

4096 is a nice number because it means the activations take around 1T of cpu memory. I actually generated some Hessians for Llama 3 last week but haven't had a chance to verify that they're correct yet. It took me around 20-30 hours on 8 A100s for 4096 and a context length of 8192. I'm planning on uploading them once I check them so you may want to wait for that instead of generating your own.

[gunho1123]

Thank you for sharing the Hessians for LLaMA 3!
Could you let me know the devset_size and ctx_size used for this release?

[gunho1123]

@tsengalb99 I tried using the downloaded Hessians to quantize the LLaMA-3.1-8B model, but I ran into the following error. Could you help me troubleshoot it?

File "/mnt/home/quip-sharp/lib/algo/quip.py", line 70, in incoherence_preprocess
Lhr = torch.linalg.cholesky(Hr) # Try again after regularization
^^^^^^^^^^^^^^^^^^^^^^^^^
torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 636 is not positive-definite).
Cholesky failed: linalg.cholesky: The factorization could not be completed because the input is not positive-definite