pass in tensor_id for calculate_qparam #1709
Conversation
Signed-off-by: shanjiaz <[email protected]>
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @shanjiaz, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
I've addressed two key issues impacting FP8_BLOCK quantization performance. First, I've corrected the problem of shared statistics across blocks by ensuring that each block's quantization parameters are calculated using a unique identifier. Second, I've updated the example configuration to prevent the quantization of critical Mixture-of-Experts (MoE) gates, which were previously degrading model performance. These changes collectively lead to more accurate and performant quantized models.
Highlights
- Improved FP8_BLOCK Quantization Accuracy: I've resolved an issue where FP8_BLOCK quantization produced poor lm_eval results by ensuring that each block uses a unique tensor_id for calculate_qparams, preventing incorrect running statistics.
- Preventing MoE Gate Quantization: I've updated the fp8_block_example.py to properly ignore critical MoE gates (mlp.gate and mlp.shared_expert_gate) during quantization, which previously degraded performance.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
Signed-off-by: shanjiaz <[email protected]>
…roject/llm-compressor into hz-fix-block-quantization
There was a problem hiding this comment.
Code Review
This pull request addresses two issues with FP8_BLOCK quantization: shared statistics across blocks and incorrect quantization of MoE gates. The changes involve passing a unique tensor_id for each block when calculating quantization parameters, which correctly isolates their statistics. Additionally, the example script is updated to properly ignore MoE gate layers during quantization. The changes appear correct and directly address the described problems. I have one suggestion regarding code maintainability related to a method signature in the base observer class.
There was a problem hiding this comment.
@shanjiaz does this mean that it was only an issue for the MoE models? if you use a llama 3 model does FP8_BLOCK still create gibberish output?
Signed-off-by: shanjiaz <[email protected]>
Sure! Added in PR descriptions. Uploaded to huggingface under: |
I tried llama as well. It worked!
|
Signed-off-by: shanjiaz <[email protected]>
Signed-off-by: shanjiaz <[email protected]>
Signed-off-by: shanjiaz <[email protected]>
Signed-off-by: shanjiaz <[email protected]>
…roject/llm-compressor into hz-fix-block-quantization
Signed-off-by: shanjiaz <[email protected]>
### Issue:
FP8_BLOCK quantization produced poor `lm_eval` results due to two
issues:
1. **Shared statistics across blocks**: All blocks used the same
`tensor_id`, causing incorrect running statistics
2. **MoE gates being quantized**: Critical routing layers were
quantized, degrading performance
### Solution
- **Fixed block statistics**: Pass unique `tensor_id=f"block_{i}_{j}"`
to `calculate_qparams` for each block
- **Updated example**: Set proper ignore layers
### Changes
- `src/llmcompressor/observers/base.py`: Added unique tensor IDs for
block-wise statistics
- `examples/quantization_w8a8_fp8/fp8_block_example.py`: Fixed ignore
patterns for MoE gates
### Test:
Produced models:
```
shanjiaz/Qwen3-30B-A3B-FP8-BLOCK
shanjiaz/Qwen3-0.6B-FP8-BLOCK
```
Quantized models now get exact same result as Michael's original
```
lm_eval --model vllm --model_args pretrained=shanjiaz/Qwen3-30B-A3B-FP8-BLOCK --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto
vllm (pretrained=shanjiaz/Qwen3-30B-A3B-FP8-BLOCK,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
```
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8324|± |0.0103|
| | |strict-match | 5|exact_match|↑ |0.8848|± |0.0088|
```
lm_eval --model vllm --model_args pretrained=shanjiaz/Qwen3-0.6B-FP8-BLOCK --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto
vllm (pretrained=shanjiaz/Qwen3-0.6B-FP8-BLOCK,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
```
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3995|± |0.0135|
---------
Signed-off-by: shanjiaz <[email protected]>
Signed-off-by: Domenic Barbuzzi <[email protected]>
|
@dsikka @shanjiaz @brian-dellabetta I ran the code in examples/quantization_w8a8_fp8/fp8_block_example.py to generate a Qwen3 MoE model, but inference with vLLM failed. The vLLM version is 0.10.0, and the error log is as follows: |
@wangwenmingaa Thanks for reaching out! We're still working on supporting that on the vllm side. Meanwhile can you try this wip branch? Thanks! Let us know : ) |
@shanjiaz Thank you for your guidance. However, I've tried using this branch but am still encountering an error. The error log is as follows: |
You will also need to use compressed-tensors main until the next compressed-tensors release (which will be this week). vLLM is using an older compressed-tensors |
Issue:
FP8_BLOCK quantization produced poor
lm_evalresults due to two issues:tensor_id, causing incorrect running statisticsSolution
tensor_id=f"block_{i}_{j}"tocalculate_qparamsfor each blockChanges
src/llmcompressor/observers/base.py: Added unique tensor IDs for block-wise statisticsexamples/quantization_w8a8_fp8/fp8_block_example.py: Fixed ignore patterns for MoE gatesTest:
Produced models:
Quantized models now get exact same result as Michael's original