Add renormalize parameter for FusedMOE's & modify experts_max arg of mixture_of_experts() #70
Conversation
tangleintel
commented
Jan 9, 2025
tangleintel
commented
- Add renormalize parameter for FusedMOE cause whether to normalize routing_weights depends on the norm_topk_prob attrib in model's config.json file. Some models such as Qwen2-MoE is set to false.
- The experts_max param is inclusive according to the habana's doc
…torch.ops.hpu.mixture_of_experts()
| @@ -357,30 +357,32 @@ def forward(self, state, expert_id, w): | |||
| return torch.matmul(state, w[expert_id].transpose(0, 1)) | |||
|
|
|||
|
|
|||
| def calculate_routing_tensors(score, topk, hidden_states_dtype): | |||
| def calculate_routing_tensors(score, topk, hidden_states_dtype, renormalize: bool = True): | |||
There was a problem hiding this comment.
Can you add a small test for this to vllm-fork ci?
There was a problem hiding this comment.
I found it's hard to demonstrate this with a UT cause it's depends on the real model with trained weights and see what's the different of the generated contents with and without this PR. We can refer to optimum habana's results and CUDA's results. But I think its's not realistic to put them into the UT. Do you have any suggestions ?
There was a problem hiding this comment.
We define tests in https://github.com/HabanaAI/vllm-fork/blob/habana_main/.jenkins/test_config.yaml please refer to configuration file to see possible invocation of a small test for fusedMoE with renormalize True and False
| @@ -357,30 +357,32 @@ def forward(self, state, expert_id, w): | |||
| return torch.matmul(state, w[expert_id].transpose(0, 1)) | |||
|
|
|||
|
|
|||
| def calculate_routing_tensors(score, topk, hidden_states_dtype): | |||
| def calculate_routing_tensors(score, topk, hidden_states_dtype, renormalize: bool = True): | |||
There was a problem hiding this comment.
We define tests in https://github.com/HabanaAI/vllm-fork/blob/habana_main/.jenkins/test_config.yaml please refer to configuration file to see possible invocation of a small test for fusedMoE with renormalize True and False
