[Usage]: When will the weights of deepseek-r1-w4a8 be uploaded to modelscope/vllm-ascend?

[tt545571022]

When will the weights of deepseek-r1-w4a8 be uploaded to modelscope/vllm-ascend? I'm not quite sure about the weights I quantified through msmodelslim.

I want to run inference of deepseek-r1-w4a8, and failed，I suspect that there might be a problem with the weight quantification I did myself

[Yikun]

@22dimensions

[22dimensions]

We will upload the weights this week. By the way can you paste how you convert the model and the error message of running inference.

[tt545571022]

We will upload the weights this week. By the way can you paste how you convert the model and the error message of running inference.
I reference #2172

git clone https://gitee.com/ascend/msit.git
cd msit
git checkout 298e175d69b3b855111a1e09bbe2fcd12fdb4e24
cd ./msmodelslim
bash install.sh
cd /example/DeepSeek
python3 quant_deepseek_w4a8.py --model_path /workspace/models/DeepSeek-R1/ --save_path /workspace/models/DeepSeek-R1-w4a8  --mindie_format --batch_size 16 | tee out.log 2>&1

then

Since mindie_format generates mindie format, some adaptation modifications are needed for vllm-ascend to use it:
quant_model_description_w8a8_dynamic.json rename to quant_model_description.json, and add "group_size": 256
Modification in config.json："model_type":deepseekv2 is changed to "model_type":deepseek_v3; quantization_config is removed;

the quant log :

2025-08-12 03:05:21,977 - msmodelslim-logger - INFO - Save safetensors files to /workspace/models/DeepSeek-R1-w4a8-tjl successfully
2025-08-12 03:05:21,977 - msmodelslim-logger - INFO - Path of quant_model_description_json is '/workspace/models/DeepSeek-R1-w4a8-tjl/quant_model_description_w8a8_dynamic.json' 
2025-08-12 03:05:22,825 - msmodelslim-logger - INFO - Save quant_model_description_json success!
2025-08-12 03:05:22,825 - msmodelslim-logger - INFO - Safetensors weight saved successfully
2025-08-12 03:05:22,825 - msmodelslim-logger - INFO - Save successfully!
2025-08-12 03:05:24,363 - msmodelslim-logger - WARNING - '/workspace/models/DeepSeek-R1-w4a8-tjl/quant_model_description_w8a8_dynamic.json' already exist. The original file will be overwritten.
2025-08-12 03:05:26,073 - msmodelslim-logger - WARNING - '/workspace/models/DeepSeek-R1-w4a8-tjl/quant_model_description_w8a8_dynamic.json' already exist. The original file will be overwritten.
2025-08-12 03:05:26,236 - msmodelslim-logger - WARNING - '/workspace/models/DeepSeek-R1-w4a8-tjl/quant_model_description_w8a8_dynamic.json' already exist. The original file will be overwritten.

Total Process:  80%|████████  | 4/5 [2:59:29<56:11, 3371.53s/it]
Total Process: 100%|██████████| 5/5 [2:59:29<00:00, 2153.83s/it]

the offline infer script:

import os

os.environ["VLLM_USE_MODELSCOPE"] = "True"
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

from vllm import LLM, SamplingParams


def main():
    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]*50

    # Create a sampling params object.
    sampling_params = SamplingParams(max_tokens=100, temperature=0.0)
    # Create an LLM.
    llm = LLM(
            model="/workspace/models/DeepSeek-R1-w4a8",
              tensor_parallel_size=16,
              enforce_eager=True,
              trust_remote_code=True,
              quantization="ascend",
              max_model_len=1024,
              )

    # Generate texts from the prompts.
    outputs = llm.generate(prompts, sampling_params)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")


if __name__ == "__main__":
    main()

the stderr log:

(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 382, in __init__
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     self.worker.load_model()
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 248, in load_model
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     self.model_runner.load_model()
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2087, in load_model
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/__init__.py", line 59, in get_model
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     return loader.load_model(vllm_config=vllm_config,
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/base_loader.py", line 50, in load_model
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     process_weights_after_loading(model, model_config, target_device)
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/utils.py", line 116, in process_weights_after_loading
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     quant_method.process_weights_after_loading(module)
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm-ascend/vllm_ascend/quantization/quant_config.py", line 339, in process_weights_after_loading
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     self.quant_method.process_weights_after_loading(layer)
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm-ascend/vllm_ascend/quantization/w4a8_dynamic.py", line 385, in process_weights_after_loading
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     layer.w2_weight_scale_second.data, bias1 = self.process_scale(
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]                                                ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]   File "/vllm-workspace/vllm-ascend/vllm_ascend/quantization/w4a8_dynamic.py", line 341, in process_scale
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]     weight_high = weight.to(torch.float32).reshape([group_num, quantgroup_num, -1, n]) * \
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=640858) ERROR 08-12 03:38:26 [multiproc_executor.py:511] RuntimeError: shape '[256, 0, -1, 7168]' is invalid for input of size 234881024

[tt545571022]

Besides, I found that the model before quantization was 642GiB in size, and the model after quantization was 658GiB in size. wuwuwu~~~
@22dimensions

[tt545571022]

@22dimensions 麻烦你帮忙看看，我使用modelscope上vllm-ascend/DeepSeek-R1-w4a8-pruning的权重，tp=16时会报一样的错误。