I have a question regarding the bash files in the EMoE/Language/scripts folder. It appears that you applied MoEfication to only one [10th] layer of the model. Additionally, since you set expert_repeat to be greater than or equal to num_experts, it seems that each expert is initialized with the original weights. As a result, the dimensionality isn't effectively reduced, correct?
I have a question regarding the bash files in the EMoE/Language/scripts folder. It appears that you applied MoEfication to only one [10th] layer of the model. Additionally, since you set expert_repeat to be greater than or equal to num_experts, it seems that each expert is initialized with the original weights. As a result, the dimensionality isn't effectively reduced, correct?