Skip to content

Fix TypeError when loading model in MoE router without bitsandbytes#690

Merged
cg123 merged 1 commit into
arcee-ai:mainfrom
umran666:fix/moe-router-load-in-4bit
Jun 13, 2026
Merged

Fix TypeError when loading model in MoE router without bitsandbytes#690
cg123 merged 1 commit into
arcee-ai:mainfrom
umran666:fix/moe-router-load-in-4bit

Conversation

@umran666

@umran666 umran666 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

I was trying to run mergekit-moe to merge some Qwen models on my CPU. Even though I wasn't using any 4-bit or 8-bit quantization flags, the merge kept crashing with:
TypeError: Qwen2ForCausalLM.__init__() got an unexpected keyword argument 'load_in_4bit'

It turns out that mergekit was passing load_in_4bit=False and load_in_8bit=False by default to the Hugging Face from_pretrained method. In newer versions of transformers, passing these options when you don't have a CUDA GPU or bitsandbytes configured causes a crash because the library doesn't know what to do with them.

Instead of always passing load_in_4bit and load_in_8bit as False, this change constructs the arguments dynamically. They are only sent to from_pretrained if they are set to True.
This keeps CPU merges (and any merge not using quantization) running smoothly without crashing!


Note

Low Risk
Narrow change to optional kwargs for model loading; behavior unchanged when 4/8-bit flags are explicitly enabled.

Overview
Fixes CPU / non-quantized MoE routing by only passing load_in_4bit and load_in_8bit to AutoModelForCausalLM.from_pretrained when those flags are True, instead of always forwarding them as False.

In get_gate_params (hidden / hidden_avg / hidden_last modes), quantization options are collected in a kwargs dict and spread into from_pretrained, so models like Qwen2 no longer receive unsupported keyword arguments on CPU or without bitsandbytes.

Reviewed by Cursor Bugbot for commit 07bae36. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@umran666

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I hereby sign the CLA

@cg123 cg123 merged commit e5b2b96 into arcee-ai:main Jun 13, 2026
6 of 7 checks passed
@cg123

cg123 commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

@github-actions github-actions Bot locked and limited conversation to collaborators Jun 13, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants