[aice/v.1.22] refactor chunk size code #354
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* enable chunk moe * add fix * fix wrong name
Summary This PR fixes the scaling issue for models like Hunyuan: w2_scale_fp8 is provided as a scalar, but should be expanded to match the per-channel size. w13_scale_fp8 is given in a combined form (two values for W1/W3) and needs to be reshaped and repeated to the correct size for per-channel quantization. Ensures that w2_input_scale is stored as a list (one per expert) instead of a single tensor.
* add calibration and conversion for GLM-4.5 fp8 models * set VLLM_DISABLE_MARK_SCALES_AS_CONST=true for scale_format=const * add conversion scripts for GLM-4.5 fp8 models * use torch.finfo for fp8 max
ranzhejiang
requested review from
afierka-intel,
jikunshang,
kzawora-intel,
madamczyk-intel,
mgawarkiewicz-intel,
michalkuligowski,
mswiniarsk,
tzielinski-habana and
xuechendi
as code owners
ranzhejiang
changed the title
ranzhejiang
changed the title
hlin99
approved these changes
Sep 1, 2025
|
what's the relationship between |
Updated in PR description |
|
@ranzhejiang , pls let me know if this PR is still valid. If not, pls close it. Or resolve the conflict. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@czhu15 @yangulei @Wei-Lin-Intel please help to review, thanks a lot.
1.Set env
PT_HPU_MOE_CHUNKfor chunk moe, which determines the chunk size sequence.2.Add env
PT_HPU_MOE_TOKEN_BOUNDARYfor chunk moe, which helps to select chunk size when meet different tokens numbers.For example:
PT_HPU_MOE_TOKEN_BOUNDARYis [64,128,1536,1736,2048,3072,4096],PT_HPU_MOE_CHUNKis [64,128,512,1024,1536,2048,4096]token_numberis 1025, we first look for the interval inPT_HPU_MOE_TOKEN_BOUNDARY. We find that 128 < 1025 ≤ 1536, so we select the position index of the interval (128, 1536], which is 2. Next, we choose the corresponding chunk size value inPT_HPU_MOE_CHUNK, which is 512. Therefore, the chunk size is set to 512.