Skip to content

Commit 27547a1

Browse files
[MM][Bugfix] Add MoE verification for multi-modal models (#3897) (#4027)
### What this PR does / why we need it? Fix #3891. The empty of `moe_comm_method` in the above issue is due to the wrong check for MoE models. To be specific, the method `is_moe_model` only checks whether a text-only model is a MoE model, without considering multi-modal models, e.g., `VL` and `Omni`. Check the config dict recursively to find if it has a key contains "expert", without checking the model architecture. It is worth noting that, we can't verify a model by if it contains `FusedMoE` module because `is_moe_model` is called somewhere before the model loading, e.g., it's called when updating the ACLGraph config in platform initialization. - vLLM version: v0.11.0 - vLLM main: vllm-project/vllm@83f478b --------- Signed-off-by: shen-shanshan <[email protected]>
1 parent 3db53d1 commit 27547a1

File tree

1 file changed

+13
-3
lines changed

1 file changed

+13
-3
lines changed

vllm_ascend/utils.py

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -624,14 +624,24 @@ def shared_expert_dp_enabled() -> bool:
624624

625625

626626
def is_moe_model(vllm_config: VllmConfig):
627+
"""Checks if the model is a MoE model by config"""
627628
global _IS_MOE_MODEL
628629
if _IS_MOE_MODEL is None:
629-
config = vllm_config.model_config.hf_config
630-
_IS_MOE_MODEL = any('experts' in key.lower()
631-
for key in config.to_dict())
630+
model_configs = vllm_config.model_config.hf_config.to_dict()
631+
_IS_MOE_MODEL = _is_contain_expert(model_configs)
632632
return _IS_MOE_MODEL
633633

634634

635+
def _is_contain_expert(config: Any):
636+
if isinstance(config, dict):
637+
for k, v in config.items():
638+
if "expert" in str(k):
639+
return True
640+
if _is_contain_expert(v):
641+
return True
642+
return False
643+
644+
635645
def weak_ref_tensor(tensor: Any) -> Any:
636646
"""
637647
Create a weak reference to a tensor.

0 commit comments

Comments
 (0)