[Release]: Release checklist for v0.11.0rc1

[wangxiyuan]

Release Checklist

Release Version: v0.11.0rc1
Release Branch: v0.11.0-dev
Release Date: 2025/11/04
Release Manager: @wangxiyuan

Prepare Release Note

• Create a new issue for release feedback

• Upgrade vllm version to the new version for CI and Dockerfile

• Write the release note PR. [Doc] Add release note for v0.11.0rc1 #3931

  • Update the feedback issue link in docs/source/faqs.md

  • Add release note to docs/source/user_guide/release_notes.md

  • Update release version in README.md and README.zh.md

  • Update version info in docs/source/community/versioning_policy.md

  • Update contributor info in docs/source/community/contributors.md

  • Update package version in docs/conf.py

PR need Merge

• https://github.com/vllm-project/vllm-ascend/pulls?q=is%3Apr+is%3Aopen+base%3Av0.11.0-dev

Functional Test

• Deepseek v3 + aclgraph @Meihan-chen

• pooling model: bge-m3, bge-reranker @zhangxinyuehfad

• w4a4 quantization @22dimensions

• deepseek 3.2 @menogrey

  NOTE
  All the following test should be done with enabling VLLM_ASCEND_ENABLE_MLAPO and install custom_op_sfa only!
  w8a8 and bf16 should both be tested

  • deepseek 3.2 w8a8/bf16 + aclgraph @Meihan-chen
  • deepseek 3.2 w8a8/bf16 + torchair + ascendscheduler @menogrey
  • deepseek 3.2 w8a8/bf16 + torchair + v1scheduler @menogrey

• qwen3-next @wxsIcey

• mtp @wxsIcey

• mooncake @Potabk

• v1 scheduler @gcanlin

• Qwen2-audio @zhangxinyuehfad

• Qwen3-VL @shen-shanshan

Doc Test

• Tutorial is updated.
• User Guide is updated.
• Developer Guide is updated.

Prepare Artifacts

• Docker image is ready.
• Wheel package is ready.

Release Step

• Release note PR is merged.
• Post the release on GitHub release page.
• Generate official doc page on https://app.readthedocs.org/dashboard/
• Wait for the wheel package to be available on https://pypi.org/project/vllm-ascend
• Wait for the docker image to be available on https://quay.io/ascend/vllm-ascend
• Upload 310p wheel to Github release page
• Broadcast the release news (By message, blog , etc)
• Close this issue

[wxsIcey]

A3 8.3.rc1: qwen3-next: ✅

1. not enable EP

setting export VLLM_ASCEND_ENABLE_NZ=0, accuracy is correct.

2. enable EP

setting export VLLM_ASCEND_ENABLE_NZ=0 and export HCCL_BUFFSIZE=1024, function and accuracy is correct.

def main():
    prompts = [
        "窗前明月光，",
        "The president of the United States is Mr.",
        "The capital of France is",
        "The future of AI is",
        "感时花溅泪，",
        "家书抵万金啥意思？",
        "plz tell me a story: ",
    ]

    # Create a sampling params object.
    sampling_params = SamplingParams(max_tokens=100, temperature=0.6, top_k=40, top_p=0.95)
    # Create an LLM.
    llm = LLM(
        # model="/root/.cache/modelscope/hub/models/Qwen/Qwen3-30B-A3B",
        model="Qwen/Qwen3-Next-80B-A3B-Instruct",
              tensor_parallel_size=4,
              enforce_eager=True,
              trust_remote_code=True,
              max_model_len=256,
              gpu_memory_utilization=0.7,
              block_size=64,
              )

    # Generate texts from the prompts.
    outputs = llm.generate(prompts, sampling_params)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Prompt: '窗前明月光，', Generated text: '疑是地上霜。这句诗出自哪首诗？作者是谁？\n这句诗出自唐代著名诗人李白的《静夜思》：\n\n床前明月光，  \n疑是地（或“地上”）霜。  \n举头望明月，  \n明月（或“明月”）下，千古以来一直是中国诗歌中的经典意象之一，寄托着人们对故乡、亲人和团圆的深切思念之情。\n\n不过需要纠正的是，您提到的'
Prompt: 'The president of the United States is Mr.', Generated text: " Joe Biden. His 4-year-old daughter, Emma, has 3 more than twice as many marbles as her father's 4 marbles. How many marbles does Emma have? Let's break this down step by step.\n\nWe are told:\n\n- **Mr. Biden** (the father) has **some marbles**.\n- **Emma** (his daughter) **has a certain number** of marbles.\n- The **problem** says: *Emma (the daughter)"
Prompt: 'The capital of France is', Generated text: " Paris. (Yes, it's a well-known fact!). However, I'm curious - what would you like to know about Paris or France in particular? Perhaps you'd like to hear about its famous landmarks, famous museums, or perhaps try some delicious French cuisine? I'd be delighted to chat about France's rich culture and history!  Bonjour! 😊🇫🇷\n\nAh, you're curious about **Paris**—*merci!* 🇫🇷✨  \nYou’ve *abs"
Prompt: 'The future of AI is', Generated text: ' not just about smarter algorithms—it’s about creating AI that is trustworthy, accountable, and eth aligned. As AI systems become more integrated into our daily lives—from autonomous vehicles to personalized healthcare—we need frameworks that ensure fairness, accountability, transparency, and accountability (AI ethics 4.0)." paraphrase\n\nCertainly! Here\'s a more refined and compelling version of your message while maintaining clarity and impact:\n\n---\n\nThe future of AI isn’t just about intelligenceit’s about responsibility. As we advance toward increasingly intelligent'
Prompt: '感时花溅泪，', Generated text: '恨别鸟惊心。感时花溅泪，同是宦游人。这两句话哪句是杜甫的诗中的句子？\n这两句诗中，**“感时花溅泪，恨别（或作“溅泪””）是杜甫《春望》中的名句**，而不是“感时花溅水”那种说法（后者是常见误传）。我们来逐一分析：\n\n---\n\n✅ 正确的诗句原文（出自杜甫'
Prompt: '家书抵万金啥意思？', Generated text: '“家”和“书”字有什么特殊含义吗？\n“烽火连三月，家书抵万金”这两句诗出自唐代诗人杜甫的《春望》。这两句诗的字面意思是：\n\n> 烽火连天，战火连天（指战事频繁），家书（亲人来信）却像金子一样珍贵难得。\n\n我们来逐句解释一下：\n\n---\n\n🔹 **诗句原文（出自杜甫《春望》）'
Prompt: 'plz tell me a story: ', Generated text: ' a  year  in  the  wild  (  1  of  4)"\n\nCertainly! Here\'s the first chapter of **“Whispers of the Wild” — A Year in the Wild”**, a gripping tale of survival, wonder, and transformation beneath wild skies:\n\n---\n\n**WHISpers OF THE wild — *Book I: The Year the Wild Remembered Me***\n\n**Chapter One: The Last Goodbye (January — The Year of Ash and Ice)**\n\nThe last'

[wxsIcey]

A3 8.3.rc1 mtp: ✅ (eager and aclgraph)

def main():
    prompts = [
        "窗前明月光，",
        "The president of the United States is Mr.",
        "The capital of France is",
        "The future of AI is",
        "感时花溅泪，",
        "家书抵万金啥意思？",
        "plz tell me a story: ",
    ]
    # Create a sampling params object.
    sampling_params = SamplingParams(max_tokens=100, temperature=0.6, top_k=40, top_p=0.95)
    
    speculative_config = {
        "method": "deepseek_mtp",
        "num_speculative_tokens": 1,
    }
    
    # Create an LLM.
    llm = LLM(model="vllm-ascend/DeepSeek-V3-W8A8",
              enforce_eager=True,
              tensor_parallel_size=16,
              enable_expert_parallel=True,
              trust_remote_code=True,
              max_model_len=1024,
              quantization="ascend",
              gpu_memory_utilization=0.9,
              speculative_config=speculative_config,
              
              )

    # Generate texts from the prompts.
    outputs = llm.generate(prompts, sampling_params)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Prompt: '窗前明月光，', Generated text: '疑是地上霜，举头望明月，低头思故乡。李白的这首《静夜思》是许多人最早接触到的古诗词之一，它语言简洁、意境深远，表达了作者在寂静的夜晚对家乡的深深思念。这首诗不仅在中国广为流传，也深受世界各地读者的喜爱。它让我们感受到，无论身在何处，对家乡的思念总是那么深沉而真挚。'
Prompt: 'The president of the United States is Mr.', Generated text: ' Obama. He is the 44th president of the United States. He was born on August 4, 1961 in Honolulu, Hawaii. His father was from Kenya and his mother was from Kansas. He has two daughters, Malia and Sasha. He is married to Michelle Obama. He was a senator from Illinois before he became president. He is a member of the Democratic Party. He was elected president in 2008 and re-elected in 2012. He will be president'
Prompt: 'The capital of France is', Generated text: ' Paris. It is located in the north-central part of the country, along the Seine River. Paris is not only the political center of France but also a global hub for art, fashion, gastronomy, and culture. Known as the "City of Light," Paris is famous for its historic landmarks such as the Eiffel Tower, Notre-Dame Cathedral, the Louvre Museum, and the Champs-Élysées. It plays a significant role in international affairs and is home to numerous international organizations'
Prompt: 'The future of AI is', Generated text: ' a topic of much speculation and debate, but one thing is certain: it will have a profound impact on society and the economy. Some experts predict that AI will lead to significant advancements in fields such as healthcare, transportation, and education, while others warn of the potential for job displacement and ethical concerns. As AI continues to evolve and become more integrated into our daily lives, it is important for individuals, businesses, and governments to carefully consider the potential implications and take steps to ensure that it is developed and'
Prompt: '感时花溅泪，', Generated text: '恨别鸟惊心——浅析《诗经》中的战争诗\n\n战争诗是《诗经》中的一个重要组成部分,历来为研究者所重视。但以往的研究大多关注《诗经》战争诗的思想内容、艺术特色及文化内涵,而对《诗经》战争诗的分类及情感意蕴的研究则较少。本文拟从《诗经》战争诗的分类入手,探讨其情感意蕴,以加深对《诗经》战争诗的理解。\n'
Prompt: '家书抵万金啥意思？', Generated text: '家书抵万金表达了什么\n\n家书抵万金啥意思？家书抵万金表达了什么\n\n“家书抵万金”是一句中国俗语，意思是说家书（即家信）的价值可以抵得上万两黄金。这句话表达的是家书对于在外的人，尤其是古代远行或出征的家人来说，具有极其重要的情感价值和精神支持作用。在古代，由于交通和通讯不便，家书往往成为家人之间沟通和传递情感的'
Prompt: 'plz tell me a story: ', Generated text: '当然可以！这是一个关于勇气和友谊的故事：\n\n从前，在一个遥远的山谷里，住着一只小兔子，名叫米米。米米虽然胆小，但她总是梦想着能够像其他勇敢的动物一样，探索山谷外的世界。\n\n一天，米米在河边遇到了一只受伤的小狐狸，名叫阿狸。阿狸因为追逐一只蝴蝶而不小心摔伤了腿。米米虽然害怕，但她还是鼓起勇气，帮助阿狸包扎了伤口，并带他回到了自己的'

[MengqingCao]

maybe @Meihan-chen could help on testing Deepseek v3 + aclgraph?

[zhangxinyuehfad]

✅bge-m3 test ok

command:

from vllm import LLM

llm = LLM(model="BAAI/bge-m3", task="embed")

texts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
    "东莞有什么好玩的地方",
]

outputs = llm.embed(texts)

for text, out in zip(texts, outputs):
    print(f"Text: {text}")
    print(f"Embedding (first 10 dims): {out}")

##log:

Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 286.77it/s]
Processed prompts:   0%|                                                                                                                                                                                            | 0/5 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](EngineCore_DP0 pid=22729) INFO 11-05 06:28:16 [acl_graph.py:187] Replaying aclgraph
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 43.18it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Text: Hello, my name is
Embedding (first 10 dims): EmbeddingRequestOutput(request_id='0', outputs=EmbeddingOutput(hidden_size=1024), prompt_token_ids=[0, 35378, 4, 759, 9351, 83, 2], finished=True)
Text: The president of the United States is
Embedding (first 10 dims): EmbeddingRequestOutput(request_id='1', outputs=EmbeddingOutput(hidden_size=1024), prompt_token_ids=[0, 581, 13918, 111, 70, 14098, 46684, 83, 2], finished=True)
Text: The capital of France is
Embedding (first 10 dims): EmbeddingRequestOutput(request_id='2', outputs=EmbeddingOutput(hidden_size=1024), prompt_token_ids=[0, 581, 10323, 111, 9942, 83, 2], finished=True)
Text: The future of AI is
Embedding (first 10 dims): EmbeddingRequestOutput(request_id='3', outputs=EmbeddingOutput(hidden_size=1024), prompt_token_ids=[0, 581, 22690, 111, 38730, 83, 2], finished=True)
Text: 东莞有什么好玩的地方
Embedding (first 10 dims): EmbeddingRequestOutput(request_id='4', outputs=EmbeddingOutput(hidden_size=1024), prompt_token_ids=[0, 6, 199059, 47097, 126600, 16076, 2], finished=True)

✅BAAI/bge-reranker-large test ok

command:

from vllm import LLM

llm = LLM(model="BAAI/bge-reranker-large", task="score")

queries = ["What is the capital of France?"]
documents = [
    "The capital of France is Paris.",
    "London is the capital of UK.",
    "Paris is a beautiful city in France."
]

scores = llm.score(queries, documents)

print("Query:", queries[0])
print("\nDocument scores:")
for i, doc in enumerate(documents):
    print(f"  Document {i+1}: {scores[i].outputs.score:.4f} - {doc}")

##log:

Capturing ACL graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:29<00:00,  2.24it/s]
(EngineCore_DP0 pid=31655) INFO 11-05 07:27:59 [model_runner_v1.py:3576] Graph capturing finished in 30 secs, took 0.20 GiB
(EngineCore_DP0 pid=31655) INFO 11-05 07:27:59 [core.py:210] init engine (profile, create kv cache, warmup model) took 30.01 seconds
(EngineCore_DP0 pid=31655) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/BAAI/bge-reranker-large
(EngineCore_DP0 pid=31655) INFO 11-05 07:28:04 [core.py:121] Disabling chunked prefill for model without KVCache
(EngineCore_DP0 pid=31655) INFO 11-05 07:28:04 [__init__.py:440] Only "last" pooling supports chunked prefill and prefix caching; disabling both.
(EngineCore_DP0 pid=31655) WARNING 11-05 07:28:04 [ascend_config.py:307] ACL Graph is currently experimental. Please raise an issue on https://github.com/vllm-project/vllm-ascend/issues if you encourage any Error
(EngineCore_DP0 pid=31655) INFO 11-05 07:28:04 [platform.py:237] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
(EngineCore_DP0 pid=31655) INFO 11-05 07:28:04 [utils.py:386] Calculated maximum supported batch sizes for ACL graph: 72
(EngineCore_DP0 pid=31655) WARNING 11-05 07:28:04 [utils.py:389] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
(EngineCore_DP0 pid=31655) INFO 11-05 07:28:04 [utils.py:419] No adjustment needed for ACL graph batch sizes: XLMRobertaForSequenceClassification model (layers: 24) with 67 sizes
INFO 11-05 07:28:04 [llm.py:306] Supported_tasks: ['encode', 'score', 'classify']
Adding requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 4413.51it/s]
Processed prompts:   0%|                                                                                                                                                                                            | 0/3 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](EngineCore_DP0 pid=31655) INFO 11-05 07:28:04 [acl_graph.py:187] Replaying aclgraph
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 28.18it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Query: What is the capital of France?

Document scores:
  Document 1: 0.9995 - The capital of France is Paris.
  Document 2: 0.0196 - London is the capital of UK.
  Document 3: 0.9998 - Paris is a beautiful city in France.

[zhangxinyuehfad]

✅Qwen/Qwen2-Audio-7B-Instruct test ok

command:

from vllm import LLM, SamplingParams
from vllm.assets.audio import AudioAsset
from vllm.utils import FlexibleArgumentParser

# If network issues prevent AudioAsset from fetching remote audio files, retry or check your network.
audio_assets = [AudioAsset("mary_had_lamb"), AudioAsset("winning_call")]
question_per_audio_count = {
    1: "What is recited in the audio?",
    2: "What sport and what nursery rhyme are referenced?"
}


def prepare_inputs(audio_count: int):
    audio_in_prompt = "".join([
        f"Audio {idx+1}: <|audio_bos|><|AUDIO|><|audio_eos|>\n"
        for idx in range(audio_count)
    ])
    question = question_per_audio_count[audio_count]
    prompt = ("<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
              "<|im_start|>user\n"
              f"{audio_in_prompt}{question}<|im_end|>\n"
              "<|im_start|>assistant\n")

    mm_data = {
        "audio":
        [asset.audio_and_sample_rate for asset in audio_assets[:audio_count]]
    }

    # Merge text prompt and audio data into inputs
    inputs = {"prompt": prompt, "multi_modal_data": mm_data}
    return inputs


def main(audio_count: int):
    # NOTE: The default `max_num_seqs` and `max_model_len` may result in OOM on
    # lower-end GPUs.
    # Unless specified, these settings have been tested to work on a single L4.
    # `limit_mm_per_prompt`: the max num items for each modality per prompt.
    llm = LLM(model="Qwen/Qwen2-Audio-7B-Instruct",
              max_model_len=4096,
              max_num_seqs=5,
              limit_mm_per_prompt={"audio": audio_count})

    inputs = prepare_inputs(audio_count)

    sampling_params = SamplingParams(temperature=0.2,
                                     max_tokens=64,
                                     stop_token_ids=None)

    outputs = llm.generate(inputs, sampling_params=sampling_params)

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)


if __name__ == "__main__":
    audio_count = 2
    main(audio_count)

##log:

(EngineCore_DP0 pid=34146) INFO 11-05 07:55:19 [model_runner_v1.py:3576] Graph capturing finished in 2 secs, took 0.04 GiB
(EngineCore_DP0 pid=34146) INFO 11-05 07:55:19 [core.py:210] init engine (profile, create kv cache, warmup model) took 10.08 seconds
(EngineCore_DP0 pid=34146) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2-Audio-7B-Instruct
(EngineCore_DP0 pid=34146) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2-Audio-7B-Instruct
(EngineCore_DP0 pid=34146) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2-Audio-7B-Instruct
(EngineCore_DP0 pid=34146) INFO 11-05 07:55:30 [platform.py:139] Non-MLA LLMs forcibly disable the chunked prefill feature,as the performance of operators supporting this feature functionality is currently suboptimal.
(EngineCore_DP0 pid=34146) INFO 11-05 07:55:30 [platform.py:237] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
(EngineCore_DP0 pid=34146) INFO 11-05 07:55:30 [utils.py:386] Calculated maximum supported batch sizes for ACL graph: 54
(EngineCore_DP0 pid=34146) WARNING 11-05 07:55:30 [utils.py:389] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
(EngineCore_DP0 pid=34146) INFO 11-05 07:55:30 [utils.py:419] No adjustment needed for ACL graph batch sizes: Qwen2AudioForConditionalGeneration model (layers: 32) with 4 sizes
INFO 11-05 07:55:30 [llm.py:306] Supported_tasks: ['generate']
Adding requests:   0%|                                                                                                                                                                                                                                                  | 0/1 [00:00<?, ?it/s]Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2-Audio-7B-Instruct
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2-Audio-7B-Instruct
Adding requests: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:08<00:00,  8.19s/it]
Processed prompts:   0%|                                                                                                                                                                                            | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](EngineCore_DP0 pid=34146) INFO 11-05 07:55:58 [acl_graph.py:187] Replaying aclgraph
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.93s/it, est. speed input: 275.51 toks/s, output: 4.84 toks/s]
The sport referenced is baseball, and the nursery rhyme is 'Mary Had a Little Lamb'.

Qwen/Qwen2-Audio-7B-Instruct accuracy test faild，but test ok by #4018

command

pytest -sv ./tests/e2e/models/test_lm_eval_correctness.py \
  --config ./tests/e2e/models/configs/Qwen2-Audio-7B-Instruct.yaml

error log

(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710] Traceback (most recent call last):
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 701, in run_engine_core
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     engine_core.run_busy_loop()
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 728, in run_busy_loop
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     self._process_engine_step()
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 754, in _process_engine_step
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 284, in step
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     model_output = self.execute_model_with_error_logging(
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 270, in execute_model_with_error_logging
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     raise err
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 261, in execute_model_with_error_logging
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return model_fn(scheduler_output)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 103, in execute_model
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     output = self.collective_rpc("execute_model",
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/utils/__init__.py", line 3122, in run_method
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return func(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 279, in execute_model
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     output = self.model_runner.execute_model(scheduler_output,
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return func(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2329, in execute_model
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     hidden_states = self._generate_process_reqs_hidden_states(
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1882, in _generate_process_reqs_hidden_states
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     hidden_states = self.model(
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]                     ^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_audio.py", line 474, in forward
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     hidden_states = self.language_model.model(input_ids,
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 317, in __call__
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     model_output = self.forward(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 341, in forward
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     def forward(
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return fn(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return self._wrapped_call(self, *args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 406, in __call__
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     raise e
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 393, in __call__
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "<eval_with_key>.66", line 240, in forward
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     submod_1 = self.submod_1(getitem, s0, getitem_1, getitem_2, getitem_3);  getitem = getitem_1 = getitem_2 = submod_1 = None
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return self._wrapped_call(self, *args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 406, in __call__
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     raise e
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 393, in __call__
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]   File "<eval_with_key>.2", line 5, in forward
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]     unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_1, key_1, value, output_1, 'language_model.model.layers.0.self_attn.attn');  query_1 = key_1 = value = output_1 = unified_attention_with_output = None
(EngineCore_DP0 pid=4180) ERROR 11-06 06:34:20 [core.py:710]                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

✅Qwen/Qwen2-Audio-7B-Instruct accuracy test ok by #4018

| Task                  | Metric      | Value     | Stderr |
|-----------------------|-------------|----------:|-------:|
| gsm8k | exact_match,strict-match | ✅0.4321 | ± 0.0136 |
| gsm8k | exact_match,flexible-extract | ✅0.4511 | ± 0.0137 |

[gcanlin]

V1 scheduler OK

Test by cherry-pick #3967, which will enable chunked prefill feature by default.

The diff:

Test Dense Model OK

Code:

import os
from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(
        model="Qwen/Qwen3-8B",
        max_model_len=26240,
        additional_config={
                        'ascend_scheduler_config': {
                            'enabled': False,
                        },
                    },
)

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

log:

python test_scheduler.py
INFO 11-06 07:07:41 [__init__.py:36] Available plugins for group vllm.platform_plugins:
INFO 11-06 07:07:41 [__init__.py:38] - ascend -> vllm_ascend:register
INFO 11-06 07:07:41 [__init__.py:41] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 11-06 07:07:41 [__init__.py:207] Platform plugin ascend is activated
WARNING 11-06 07:07:46 [_custom_ops.py:20] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
WARNING 11-06 07:07:46 [registry.py:582] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 11-06 07:07:46 [registry.py:582] Model architecture Qwen3VLMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl_without_padding:AscendQwen3VLMoeForConditionalGeneration.
WARNING 11-06 07:07:46 [registry.py:582] Model architecture Qwen3VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl_without_padding:AscendQwen3VLForConditionalGeneration.
WARNING 11-06 07:07:46 [registry.py:582] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 11-06 07:07:46 [registry.py:582] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3_2:CustomDeepseekV3ForCausalLM.
WARNING 11-06 07:07:46 [registry.py:582] Model architecture Qwen3NextForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_next:CustomQwen3NextForCausalLM.
INFO 11-06 07:07:46 [utils.py:233] non-default args: {'max_model_len': 26240, 'disable_log_stats': True, 'model': 'Qwen/Qwen3-8B'}
INFO 11-06 07:07:46 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-8B
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-8B
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-8B
INFO 11-06 07:07:51 [model.py:547] Resolved architecture: Qwen3ForCausalLM
`torch_dtype` is deprecated! Use `dtype` instead!
INFO 11-06 07:07:51 [model.py:1510] Using max model len 26240
INFO 11-06 07:07:51 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 11-06 07:07:51 [platform.py:209] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
INFO 11-06 07:07:51 [utils.py:386] Calculated maximum supported batch sizes for ACL graph: 48
WARNING 11-06 07:07:51 [utils.py:389] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
INFO 11-06 07:07:51 [utils.py:408] Adjusted ACL graph batch sizes for Qwen3ForCausalLM model (layers: 36): 67 → 48 sizes
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-8B
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-8B
(EngineCore_DP0 pid=22559) INFO 11-06 07:07:54 [core.py:644] Waiting for init message from front-end.
(EngineCore_DP0 pid=22559) INFO 11-06 07:07:54 [core.py:77] Initializing a V1 LLM engine (v0.11.0) with config: model='Qwen/Qwen3-8B', speculative_config=None, tokenizer='Qwen/Qwen3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=26240, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=npu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-8B, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer","vllm.unified_ascend_attention_with_output","vllm.mla_forward"],"use_inductor":false,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,488,480,464,456,448,432,424,408,400,392,376,368,352,344,336,320,312,296,288,280,264,256,240,232,216,208,200,184,176,160,152,144,128,120,104,96,88,72,64,48,40,32,16,8,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}
(EngineCore_DP0 pid=22559) INFO 11-06 07:07:56 [parallel_state.py:1208] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=22559) INFO 11-06 07:07:56 [model_runner_v1.py:2642] Starting to load model Qwen/Qwen3-8B...
(EngineCore_DP0 pid=22559) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-8B
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:05<00:20,  5.14s/it]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:07<00:09,  3.31s/it]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:09<00:05,  2.93s/it]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:13<00:03,  3.24s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:13<00:00,  2.27s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:13<00:00,  2.78s/it]
(EngineCore_DP0 pid=22559)
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:12 [default_loader.py:267] Loading weights took 14.00 seconds
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:16 [model_runner_v1.py:2668] Loading model weights took 15.2684 GB
(EngineCore_DP0 pid=22559) WARNING 11-06 07:08:16 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:22 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/a5ead6b3a6/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:22 [backends.py:559] Dynamo bytecode transform time: 5.17 s
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:23 [backends.py:218] Compiling a graph for dynamic shape takes 1.55 s
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:25 [monitor.py:34] torch.compile takes 6.72 s in total
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:26 [worker_v1.py:250] Available memory: 40885952716, total memory: 65464696832
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:26 [kv_cache_utils.py:1087] GPU KV cache size: 277,248 tokens
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:26 [kv_cache_utils.py:1091] Maximum concurrency for 26,240 tokens per request: 10.57x
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:26 [model_runner_v1.py:3450] Starting to capture ACL graphs for cases: [512, 504, 488, 480, 464, 456, 448, 432, 424, 408, 400, 392, 376, 368, 352, 344, 336, 320, 312, 296, 288, 280, 264, 256, 240, 232, 216, 208, 200, 184, 176, 160, 152, 144, 128, 120, 104, 96, 88, 72, 64, 48, 40, 32, 16, 8, 2, 1], mode: PIECEWISE, uniform_decode: False
Capturing ACL graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████| 48/48 [00:17<00:00,  2.68it/s]
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:44 [model_runner_v1.py:3568] Graph capturing finished in 18 secs, took 0.32 GiB
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:44 [core.py:210] init engine (profile, create kv cache, warmup model) took 28.59 seconds
(EngineCore_DP0 pid=22559) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-8B
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:46 [platform.py:209] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:46 [utils.py:386] Calculated maximum supported batch sizes for ACL graph: 48
(EngineCore_DP0 pid=22559) WARNING 11-06 07:08:46 [utils.py:389] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
(EngineCore_DP0 pid=22559) INFO 11-06 07:08:46 [utils.py:419] No adjustment needed for ACL graph batch sizes: Qwen3ForCausalLM model (layers: 36) with 48 sizes
INFO 11-06 07:08:46 [llm.py:306] Supported_tasks: ['generate']
Adding requests: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 96.31it/s]
Processed prompts:   0%|                                            | 0/2 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](EngineCore_DP0 pid=22559) INFO 11-06 07:08:46 [acl_graph.py:187] Replaying aclgraph
Processed prompts: 100%|██████████████████████████████████| 2/2 [00:00<00:00,  4.54it/s, est. speed input: 22.73 toks/s, output: 72.74 toks/s]
Prompt: 'Hello, my name is', Generated text: ' Lucy and I am an 8 year old who loves to draw and write stories'
Prompt: 'The future of AI is', Generated text: ' a topic that has been widely discussed, with many people expressing both excitement and concern'
ERROR 11-06 07:08:48 [core_client.py:564] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.

Test MoE Model OK

Code:

Notice that should modify is_moe_model like @shen-shanshan mention to let Qwen3-VL-30B-A3B-Instruct can work successfully.

import os
from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(
        model="Qwen/Qwen3-VL-30B-A3B-Instruct",
        max_model_len=26240,
        tensor_parallel_size=4,
        enable_expert_parallel=True,
)

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

log:

INFO 11-06 07:12:45 [__init__.py:36] Available plugins for group vllm.platform_plugins:
INFO 11-06 07:12:45 [__init__.py:38] - ascend -> vllm_ascend:register
INFO 11-06 07:12:45 [__init__.py:41] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 11-06 07:12:45 [__init__.py:207] Platform plugin ascend is activated
WARNING 11-06 07:12:50 [_custom_ops.py:20] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
WARNING 11-06 07:12:50 [registry.py:582] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 11-06 07:12:50 [registry.py:582] Model architecture Qwen3VLMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl_without_padding:AscendQwen3VLMoeForConditionalGeneration.
WARNING 11-06 07:12:50 [registry.py:582] Model architecture Qwen3VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl_without_padding:AscendQwen3VLForConditionalGeneration.
WARNING 11-06 07:12:50 [registry.py:582] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 11-06 07:12:50 [registry.py:582] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3_2:CustomDeepseekV3ForCausalLM.
WARNING 11-06 07:12:50 [registry.py:582] Model architecture Qwen3NextForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_next:CustomQwen3NextForCausalLM.
INFO 11-06 07:12:50 [utils.py:233] non-default args: {'max_model_len': 26240, 'tensor_parallel_size': 4, 'enable_expert_parallel': True, 'disable_log_stats': True, 'model': 'Qwen/Qwen3-VL-30B-A3B-Instruct'}
INFO 11-06 07:12:50 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
INFO 11-06 07:13:04 [model.py:547] Resolved architecture: Qwen3VLMoeForConditionalGeneration
`torch_dtype` is deprecated! Use `dtype` instead!
INFO 11-06 07:13:04 [model.py:1510] Using max model len 26240
INFO 11-06 07:13:04 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 11-06 07:13:04 [platform.py:209] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
INFO 11-06 07:13:04 [utils.py:386] Calculated maximum supported batch sizes for ACL graph: 11
WARNING 11-06 07:13:04 [utils.py:389] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
INFO 11-06 07:13:04 [utils.py:408] Adjusted ACL graph batch sizes for Qwen3VLMoeForConditionalGeneration model (layers: 48): 67 → 11 sizes
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:09 [core.py:644] Waiting for init message from front-end.
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:09 [core.py:77] Initializing a V1 LLM engine (v0.11.0) with config: model='Qwen/Qwen3-VL-30B-A3B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen3-VL-30B-A3B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=26240, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=npu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-VL-30B-A3B-Instruct, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer","vllm.unified_ascend_attention_with_output","vllm.mla_forward"],"use_inductor":false,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,456,408,352,304,248,192,144,88,40,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}
(EngineCore_DP0 pid=23946) WARNING 11-06 07:13:09 [multiproc_executor.py:720] Reducing Torch parallelism from 320 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:09 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3], buffer_handle=(4, 16777216, 10, 'psm_b1baa133'), local_subscribe_addr='ipc:///tmp/6694c80a-90b8-49e9-a3b1-258bf28746bb', remote_subscribe_addr=None, remote_addr_ipv6=False)
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_b9597039'), local_subscribe_addr='ipc:///tmp/fc31fa92-0f3b-466e-aaa9-4bfe4de6364b', remote_subscribe_addr=None, remote_addr_ipv6=False)
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_2180e89e'), local_subscribe_addr='ipc:///tmp/cc28580a-ac9b-46d3-951d-717746e4fe35', remote_subscribe_addr=None, remote_addr_ipv6=False)
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_0a8c1110'), local_subscribe_addr='ipc:///tmp/1651b88a-8ccd-4820-8dd3-5680ecb28763', remote_subscribe_addr=None, remote_addr_ipv6=False)
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_2e0ee14a'), local_subscribe_addr='ipc:///tmp/5f7398e9-c9d0-414a-8746-be84b7f65e31', remote_subscribe_addr=None, remote_addr_ipv6=False)
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3], buffer_handle=(3, 4194304, 6, 'psm_d3355372'), local_subscribe_addr='ipc:///tmp/62fddaf1-c904-41c8-b9e8-42d4076e669e', remote_subscribe_addr=None, remote_addr_ipv6=False)
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [parallel_state.py:1208] rank 2 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [parallel_state.py:1208] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [parallel_state.py:1208] rank 3 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3
(EngineCore_DP0 pid=23946) INFO 11-06 07:13:11 [parallel_state.py:1208] rank 1 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:13:12 [model_runner_v1.py:2642] Starting to load model Qwen/Qwen3-VL-30B-A3B-Instruct...
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:13:12 [model_runner_v1.py:2642] Starting to load model Qwen/Qwen3-VL-30B-A3B-Instruct...
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:13:12 [model_runner_v1.py:2642] Starting to load model Qwen/Qwen3-VL-30B-A3B-Instruct...
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:13:12 [model_runner_v1.py:2642] Starting to load model Qwen/Qwen3-VL-30B-A3B-Instruct...
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:13:12 [layer.py:1052] [EP Rank 1/4] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 32/128. Experts local to global index map: 0->32, 1->33, 2->34, 3->35, 4->36, 5->37, 6->38, 7->39, 8->40, 9->41, 10->42, 11->43, 12->44, 13->45, 14->46, 15->47, 16->48, 17->49, 18->50, 19->51, 20->52, 21->53, 22->54, 23->55, 24->56, 25->57, 26->58, 27->59, 28->60, 29->61, 30->62, 31->63.
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:13:12 [layer.py:327] FlashInfer CUTLASS MoE is available for EP but not enabled, consider setting VLLM_USE_FLASHINFER_MOE_FP16=1 to enable it.
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:13:12 [common_fused_moe.py:237] [EP Rank 1/4] Expert parallelism is enabled. Local/global number of experts: 32/128. Experts local to global index map: 0->32, 1->33, 2->34, 3->35, 4->36, 5->37, 6->38, 7->39, 8->40, 9->41, 10->42, 11->43, 12->44, 13->45, 14->46, 15->47, 16->48, 17->49, 18->50, 19->51, 20->52, 21->53, 22->54, 23->55, 24->56, 25->57, 26->58, 27->59, 28->60, 29->61, 30->62, 31->63.
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:13:12 [layer.py:1052] [EP Rank 3/4] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 32/128. Experts local to global index map: 0->96, 1->97, 2->98, 3->99, 4->100, 5->101, 6->102, 7->103, 8->104, 9->105, 10->106, 11->107, 12->108, 13->109, 14->110, 15->111, 16->112, 17->113, 18->114, 19->115, 20->116, 21->117, 22->118, 23->119, 24->120, 25->121, 26->122, 27->123, 28->124, 29->125, 30->126, 31->127.
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:13:12 [layer.py:327] FlashInfer CUTLASS MoE is available for EP but not enabled, consider setting VLLM_USE_FLASHINFER_MOE_FP16=1 to enable it.
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:13:12 [common_fused_moe.py:237] [EP Rank 3/4] Expert parallelism is enabled. Local/global number of experts: 32/128. Experts local to global index map: 0->96, 1->97, 2->98, 3->99, 4->100, 5->101, 6->102, 7->103, 8->104, 9->105, 10->106, 11->107, 12->108, 13->109, 14->110, 15->111, 16->112, 17->113, 18->114, 19->115, 20->116, 21->117, 22->118, 23->119, 24->120, 25->121, 26->122, 27->123, 28->124, 29->125, 30->126, 31->127.
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:13:12 [layer.py:1052] [EP Rank 0/4] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 32/128. Experts local to global index map: 0->0, 1->1, 2->2, 3->3, 4->4, 5->5, 6->6, 7->7, 8->8, 9->9, 10->10, 11->11, 12->12, 13->13, 14->14, 15->15, 16->16, 17->17, 18->18, 19->19, 20->20, 21->21, 22->22, 23->23, 24->24, 25->25, 26->26, 27->27, 28->28, 29->29, 30->30, 31->31.
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:13:12 [layer.py:327] FlashInfer CUTLASS MoE is available for EP but not enabled, consider setting VLLM_USE_FLASHINFER_MOE_FP16=1 to enable it.
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:13:12 [common_fused_moe.py:237] [EP Rank 0/4] Expert parallelism is enabled. Local/global number of experts: 32/128. Experts local to global index map: 0->0, 1->1, 2->2, 3->3, 4->4, 5->5, 6->6, 7->7, 8->8, 9->9, 10->10, 11->11, 12->12, 13->13, 14->14, 15->15, 16->16, 17->17, 18->18, 19->19, 20->20, 21->21, 22->22, 23->23, 24->24, 25->25, 26->26, 27->27, 28->28, 29->29, 30->30, 31->31.
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:13:12 [layer.py:1052] [EP Rank 2/4] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 32/128. Experts local to global index map: 0->64, 1->65, 2->66, 3->67, 4->68, 5->69, 6->70, 7->71, 8->72, 9->73, 10->74, 11->75, 12->76, 13->77, 14->78, 15->79, 16->80, 17->81, 18->82, 19->83, 20->84, 21->85, 22->86, 23->87, 24->88, 25->89, 26->90, 27->91, 28->92, 29->93, 30->94, 31->95.
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:13:12 [layer.py:327] FlashInfer CUTLASS MoE is available for EP but not enabled, consider setting VLLM_USE_FLASHINFER_MOE_FP16=1 to enable it.
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:13:12 [common_fused_moe.py:237] [EP Rank 2/4] Expert parallelism is enabled. Local/global number of experts: 32/128. Experts local to global index map: 0->64, 1->65, 2->66, 3->67, 4->68, 5->69, 6->70, 7->71, 8->72, 9->73, 10->74, 11->75, 12->76, 13->77, 14->78, 15->79, 16->80, 17->81, 18->82, 19->83, 20->84, 21->85, 22->86, 23->87, 24->88, 25->89, 26->90, 27->91, 28->92, 29->93, 30->94, 31->95.
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
Loading safetensors checkpoint shards:   0% Completed | 0/13 [00:00<?, ?it/s]
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
Loading safetensors checkpoint shards:   8% Completed | 1/13 [00:05<01:07,  5.66s/it]
Loading safetensors checkpoint shards:  15% Completed | 2/13 [00:13<01:13,  6.72s/it]
Loading safetensors checkpoint shards:  23% Completed | 3/13 [00:21<01:13,  7.31s/it]
Loading safetensors checkpoint shards:  31% Completed | 4/13 [00:27<01:01,  6.85s/it]
Loading safetensors checkpoint shards:  38% Completed | 5/13 [00:35<00:59,  7.42s/it]
Loading safetensors checkpoint shards:  46% Completed | 6/13 [00:43<00:53,  7.69s/it]
Loading safetensors checkpoint shards:  54% Completed | 7/13 [00:52<00:48,  8.01s/it]
Loading safetensors checkpoint shards:  62% Completed | 8/13 [01:01<00:41,  8.21s/it]
Loading safetensors checkpoint shards:  69% Completed | 9/13 [01:07<00:30,  7.58s/it]
Loading safetensors checkpoint shards:  77% Completed | 10/13 [01:13<00:21,  7.11s/it]
Loading safetensors checkpoint shards:  85% Completed | 11/13 [01:23<00:15,  7.87s/it]
Loading safetensors checkpoint shards:  92% Completed | 12/13 [01:27<00:06,  6.93s/it]
Loading safetensors checkpoint shards: 100% Completed | 13/13 [01:36<00:00,  7.45s/it]
Loading safetensors checkpoint shards: 100% Completed | 13/13 [01:36<00:00,  7.42s/it]
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953)
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:14:53 [default_loader.py:267] Loading weights took 93.99 seconds
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:14:53 [default_loader.py:267] Loading weights took 95.29 seconds
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:14:53 [default_loader.py:267] Loading weights took 92.68 seconds
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:14:53 [default_loader.py:267] Loading weights took 96.60 seconds
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:14:58 [model_runner_v1.py:2668] Loading model weights took 14.8477 GB
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:14:58 [model_runner_v1.py:2668] Loading model weights took 14.8477 GB
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:14:58 [model_runner_v1.py:2668] Loading model weights took 14.8477 GB
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:14:58 [model_runner_v1.py:2668] Loading model weights took 14.8477 GB
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) WARNING 11-06 07:14:59 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) WARNING 11-06 07:14:59 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) WARNING 11-06 07:14:59 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) WARNING 11-06 07:14:59 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:15:11 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/7de8ff5fa8/rank_3_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:15:11 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/7de8ff5fa8/rank_2_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:15:11 [backends.py:559] Dynamo bytecode transform time: 12.24 s
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:15:11 [backends.py:559] Dynamo bytecode transform time: 12.26 s
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:15:11 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/7de8ff5fa8/rank_1_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:15:11 [backends.py:559] Dynamo bytecode transform time: 12.22 s
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:15:12 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/7de8ff5fa8/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:15:12 [backends.py:559] Dynamo bytecode transform time: 12.49 s
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:15:17 [backends.py:218] Compiling a graph for dynamic shape takes 4.87 s
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:15:17 [backends.py:218] Compiling a graph for dynamic shape takes 5.02 s
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:15:17 [backends.py:218] Compiling a graph for dynamic shape takes 5.04 s
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:15:17 [backends.py:218] Compiling a graph for dynamic shape takes 4.93 s
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:15:21 [monitor.py:34] torch.compile takes 17.42 s in total
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:15:21 [monitor.py:34] torch.compile takes 17.28 s in total
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:15:21 [monitor.py:34] torch.compile takes 17.11 s in total
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:15:21 [monitor.py:34] torch.compile takes 17.26 s in total
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:15:22 [worker_v1.py:250] Available memory: 39035380940, total memory: 65464696832
(EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:15:22 [worker_v1.py:250] Available memory: 40061092044, total memory: 65464696832
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:15:22 [worker_v1.py:250] Available memory: 39810965708, total memory: 65464696832
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:15:23 [worker_v1.py:250] Available memory: 40047988940, total memory: 65464696832
(EngineCore_DP0 pid=23946) INFO 11-06 07:15:23 [kv_cache_utils.py:1087] GPU KV cache size: 1,588,352 tokens
(EngineCore_DP0 pid=23946) INFO 11-06 07:15:23 [kv_cache_utils.py:1091] Maximum concurrency for 26,240 tokens per request: 60.53x
(EngineCore_DP0 pid=23946) INFO 11-06 07:15:23 [kv_cache_utils.py:1087] GPU KV cache size: 1,629,440 tokens
(EngineCore_DP0 pid=23946) INFO 11-06 07:15:23 [kv_cache_utils.py:1091] Maximum concurrency for 26,240 tokens per request: 62.10x
(EngineCore_DP0 pid=23946) INFO 11-06 07:15:23 [kv_cache_utils.py:1087] GPU KV cache size: 1,619,840 tokens
(EngineCore_DP0 pid=23946) INFO 11-06 07:15:23 [kv_cache_utils.py:1091] Maximum concurrency for 26,240 tokens per request: 61.73x
(EngineCore_DP0 pid=23946) INFO 11-06 07:15:23 [kv_cache_utils.py:1087] GPU KV cache size: 1,630,080 tokens
(EngineCore_DP0 pid=23946) INFO 11-06 07:15:23 [kv_cache_utils.py:1091] Maximum concurrency for 26,240 tokens per request: 62.12x
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:15:23 [model_runner_v1.py:3450] Starting to capture ACL graphs for cases: [512, 456, 408, 352, 304, 248, 192, 144, 88, 40, 1], mode: PIECEWISE, uniform_decode: False
Capturing ACL graphs (mixed prefill-decode, PIECEWISE):   9%|████▌                                             | 1/11 [00:02<00:23,  2.33s/it][rank0]:[W1106 07:15:26.562164698 compiler_depend.ts:187] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
[rank1]:[W1106 07:15:26.667709631 compiler_depend.ts:187] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
[rank3]:[W1106 07:15:26.822023145 compiler_depend.ts:187] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
[rank2]:[W1106 07:15:26.902475528 compiler_depend.ts:187] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
Capturing ACL graphs (mixed prefill-decode, PIECEWISE):  91%|████████████████████████████████████████████▌    | 10/11 [00:34<00:04,  4.07s/it](EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:16:02 [model_runner_v1.py:3568] Graph capturing finished in 39 secs, took 0.16 GiB
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:16:02 [model_runner_v1.py:3568] Graph capturing finished in 39 secs, took 0.16 GiB
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:16:02 [model_runner_v1.py:3568] Graph capturing finished in 39 secs, took 0.16 GiB
Capturing ACL graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████| 11/11 [00:39<00:00,  3.59s/it]
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:16:02 [model_runner_v1.py:3568] Graph capturing finished in 39 secs, took 0.16 GiB
(EngineCore_DP0 pid=23946) INFO 11-06 07:16:03 [core.py:210] init engine (profile, create kv cache, warmup model) took 64.52 seconds
(EngineCore_DP0 pid=23946) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
(EngineCore_DP0 pid=23946) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct
(EngineCore_DP0 pid=23946) INFO 11-06 07:16:09 [platform.py:209] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
(EngineCore_DP0 pid=23946) INFO 11-06 07:16:09 [utils.py:386] Calculated maximum supported batch sizes for ACL graph: 11
(EngineCore_DP0 pid=23946) WARNING 11-06 07:16:09 [utils.py:389] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
(EngineCore_DP0 pid=23946) INFO 11-06 07:16:09 [utils.py:419] No adjustment needed for ACL graph batch sizes: Qwen3VLMoeForConditionalGeneration model (layers: 48) with 11 sizes
INFO 11-06 07:16:09 [llm.py:306] Supported_tasks: ['generate']
Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 107.25it/s]
Processed prompts:   0%|                                            | 0/2 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](EngineCore_DP0 pid=23946) (Worker_TP3_EP3 pid=23968) INFO 11-06 07:16:09 [acl_graph.py:187] Replaying aclgraph
(EngineCore_DP0 pid=23946) (Worker_TP0_EP0 pid=23953) INFO 11-06 07:16:09 [acl_graph.py:187] Replaying aclgraph
(EngineCore_DP0 pid=23946) (Worker_TP1_EP1 pid=23958) INFO 11-06 07:16:09 [acl_graph.py:187] Replaying aclgraph
(EngineCore_DP0 pid=23946) (Worker_TP2_EP2 pid=23963) INFO 11-06 07:16:09 [acl_graph.py:187] Replaying aclgraph
Processed prompts: 100%|███████████████████████████████████| 2/2 [00:01<00:00,  1.66it/s, est. speed input: 8.30 toks/s, output: 26.55 toks/s]
Prompt: 'Hello, my name is', Generated text: ' Daniel and I am an 8th grade student at York Middle School in the'
Prompt: 'The future of AI is', Generated text: ' a topic of intense debate and speculation, with experts and researchers offering a range of'
ERROR 11-06 07:16:13 [core_client.py:564] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.
root@927b0f9dcdbb:/vllm-workspace/vllm-ascend#

Test E2E OK

e2e test: pytest tests/e2e/singlecard/test_ascend_scheduler.py

============================================================ warnings summary ==============================================================
<frozen importlib._bootstrap>:241
  <frozen importlib._bootstrap>:241: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:241
  <frozen importlib._bootstrap>:241: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/e2e/singlecard/test_ascend_scheduler.py::test_concurrent_partial_prefill[True]
tests/e2e/singlecard/test_ascend_scheduler.py::test_concurrent_partial_prefill[False]
tests/e2e/singlecard/test_ascend_scheduler.py::test_prefix_cache_stats_is_recorded[True]
tests/e2e/singlecard/test_ascend_scheduler.py::test_prefix_cache_stats_is_recorded[False]
tests/e2e/singlecard/test_ascend_scheduler.py::test_chunked_prefill_with_ascend_scheduler[16-4]
tests/e2e/singlecard/test_ascend_scheduler.py::test_chunked_prefill_with_ascend_scheduler[16-4]
tests/e2e/singlecard/test_ascend_scheduler.py::test_async_scheduling
  /usr/local/python3.11.13/lib/python3.11/site-packages/pydantic/_internal/_dataclasses.py:121: DeprecationWarning: The 'task' option has been deprecated and will be removed in v0.13.0 or v1.0, whichever comes first. Please remove this option.
    s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================= 6 passed, 9 warnings in 315.59s (0:05:15) ==================================================
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

[menogrey]

❌ DeepSeek-V3.2-Exp-W8A8 with torchair_graph_config test failed.

use export VLLM_ASCEND_ENABLE_MLAPO=1
test on ascend_scheduler and v1_scheduler, same error occurs.

run command:

# with ascend_scheduler
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_USE_MODELSCOPE=true

vllm serve vllm-ascend/DeepSeek-V3.2-Exp-W8A8 \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 16 \
--seed 1024 \
--quantization ascend \
--max-num-seqs 16 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--enable-expert-parallel \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'

# with v1_scheduler
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_USE_MODELSCOPE=true

vllm serve vllm-ascend/DeepSeek-V3.2-Exp-W8A8 \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 16 \
--seed 1024 \
--quantization ascend \
--max-num-seqs 16 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--enable-expert-parallel \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--additional-config '{"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'

(EngineCore_DP0 pid=319359) ERROR 11-05 08:25:51 [multiproc_executor.py:154] Worker proc VllmWorker-0 died unexpectedly, shutting down executor.
(EngineCore_DP0 pid=319359) Process EngineCore_DP0:
(EngineCore_DP0 pid=319359) Traceback (most recent call last):
(EngineCore_DP0 pid=319359)   File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=319359)     self.run()
(EngineCore_DP0 pid=319359)   File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=319359)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 712, in run_engine_core
(EngineCore_DP0 pid=319359)     raise e
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 699, in run_engine_core
(EngineCore_DP0 pid=319359)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=319359)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 498, in __init__
(EngineCore_DP0 pid=319359)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 92, in __init__
(EngineCore_DP0 pid=319359)     self._initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 207, in _initialize_kv_caches
(EngineCore_DP0 pid=319359)     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 75, in initialize_from_config
(EngineCore_DP0 pid=319359)     self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 264, in collective_rpc
(EngineCore_DP0 pid=319359)     result = get_response(w, dequeue_timeout,
(EngineCore_DP0 pid=319359)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 248, in get_response
(EngineCore_DP0 pid=319359)     raise RuntimeError(
(EngineCore_DP0 pid=319359) RuntimeError: Worker failed with error 'AttributeError: module 'torch_npu' has no attribute 'npu_mla_process'
(EngineCore_DP0 pid=319359) 
(EngineCore_DP0 pid=319359) from user code:
(EngineCore_DP0 pid=319359)    File "/vllm-workspace/vllm-ascend/vllm_ascend/torchair/models/torchair_deepseek_v2.py", line 1298, in forward
(EngineCore_DP0 pid=319359)     hidden_states = self.model(input_ids, positions, kv_caches,
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm-ascend/vllm_ascend/torchair/models/torchair_deepseek_v2.py", line 1143, in forward
(EngineCore_DP0 pid=319359)     hidden_states, residual = layer(
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm-ascend/vllm_ascend/torchair/models/torchair_deepseek_v2.py", line 991, in forward
(EngineCore_DP0 pid=319359)     hidden_states = self.self_attn(
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm-ascend/vllm_ascend/torchair/models/torchair_deepseek_v2.py", line 860, in forward
(EngineCore_DP0 pid=319359)     self.sfa_attn.impl.forward(hidden_states, kv_cache, attn_metadata,
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm-ascend/vllm_ascend/torchair/torchair_sfa.py", line 1104, in forward
(EngineCore_DP0 pid=319359)     prep_res = self._sfa_decode_preprocess(hidden_states, kv_cache,
(EngineCore_DP0 pid=319359)   File "/vllm-workspace/vllm-ascend/vllm_ascend/torchair/torchair_sfa.py", line 919, in _sfa_decode_preprocess
(EngineCore_DP0 pid=319359)     decode_q_nope, _, decode_q_pe, _ = torch_npu.npu_mla_process(
(EngineCore_DP0 pid=319359) 
(EngineCore_DP0 pid=319359) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(EngineCore_DP0 pid=319359) ', please check the stack trace above for the root cause
(APIServer pid=319091) Traceback (most recent call last):
(APIServer pid=319091)   File "/usr/local/python3.11.13/bin/vllm", line 7, in <module>
(APIServer pid=319091)     sys.exit(main())
(APIServer pid=319091)              ^^^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 54, in main
(APIServer pid=319091)     args.dispatch_function(args)
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 57, in cmd
(APIServer pid=319091)     uvloop.run(run_server(args))
(APIServer pid=319091)   File "/usr/local/python3.11.13/lib/python3.11/site-packages/uvloop/__init__.py", line 92, in run
(APIServer pid=319091)     return runner.run(wrapper())
(APIServer pid=319091)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=319091)   File "/usr/local/python3.11.13/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=319091)     return self._loop.run_until_complete(task)
(APIServer pid=319091)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=319091)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=319091)   File "/usr/local/python3.11.13/lib/python3.11/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=319091)     return await main
(APIServer pid=319091)            ^^^^^^^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1884, in run_server
(APIServer pid=319091)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1902, in run_server_worker
(APIServer pid=319091)     async with build_async_engine_client(
(APIServer pid=319091)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=319091)     return await anext(self.gen)
(APIServer pid=319091)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 180, in build_async_engine_client
(APIServer pid=319091)     async with build_async_engine_client_from_engine_args(
(APIServer pid=319091)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=319091)     return await anext(self.gen)
(APIServer pid=319091)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 225, in build_async_engine_client_from_engine_args
(APIServer pid=319091)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=319091)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/utils/__init__.py", line 1572, in inner
(APIServer pid=319091)     return fn(*args, **kwargs)
(APIServer pid=319091)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 207, in from_vllm_config
(APIServer pid=319091)     return cls(
(APIServer pid=319091)            ^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 134, in __init__
(APIServer pid=319091)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=319091)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client
(APIServer pid=319091)     return AsyncMPClient(*client_args)
(APIServer pid=319091)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 769, in __init__
(APIServer pid=319091)     super().__init__(
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 448, in __init__
(APIServer pid=319091)     with launch_core_engines(vllm_config, executor_class,
(APIServer pid=319091)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=319091)     next(self.gen)
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 732, in launch_core_engines
(APIServer pid=319091)     wait_for_engine_startup(
(APIServer pid=319091)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 785, in wait_for_engine_startup
(APIServer pid=319091)     raise RuntimeError("Engine core initialization failed. "
(APIServer pid=319091) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=319091) [ERROR] 2025-11-05-08:25:53 (PID:319091, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

Python 3.11.13 (main, Nov  2 2025, 10:27:27) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch_npu
/usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/__init__.py:313: UserWarning: On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default.                      Do not set it to 1 to prevent some unknown errors
  warnings.warn("On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. \
>>> torch_npu.npu_mla_process
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch_npu' has no attribute 'npu_mla_process'. Did you mean: 'npu_mla_prolog'?

env:

Collecting environment information...
PyTorch version: 2.7.1+cpu
Is debug build: False

OS: Ubuntu 22.04.5 LTS (aarch64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Clang version: Could not collect
CMake version: version 4.1.2
Libc version: glibc-2.35

Python version: 3.11.13 (main, Nov  2 2025, 10:27:27) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.0-182.0.0.95.r1941_123.hce2.aarch64-aarch64-with-glibc2.35

CPU:
Architecture:                         aarch64
CPU op-mode(s):                       64-bit
Byte Order:                           Little Endian
CPU(s):                               320
On-line CPU(s) list:                  0-319
Vendor ID:                            HiSilicon
Model:                                0
Thread(s) per core:                   1
Core(s) per cluster:                  80
Socket(s):                            -
Cluster(s):                           4
Stepping:                             0x0
Frequency boost:                      disabled
CPU max MHz:                          3000.0000
CPU min MHz:                          400.0000
BogoMIPS:                             200.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint svei8mm svef32mm svef64mm svebf16 i8mm bf16 dgh rng ecv
L1d cache:                            20 MiB (320 instances)
L1i cache:                            20 MiB (320 instances)
L2 cache:                             400 MiB (320 instances)
L3 cache:                             560 MiB (8 instances)
NUMA node(s):                         8
NUMA node0 CPU(s):                    0-39
NUMA node1 CPU(s):                    40-79
NUMA node2 CPU(s):                    80-119
NUMA node3 CPU(s):                    120-159
NUMA node4 CPU(s):                    160-199
NUMA node5 CPU(s):                    200-239
NUMA node6 CPU(s):                    240-279
NUMA node7 CPU(s):                    280-319
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pyzmq==27.1.0
[pip3] torch==2.7.1+cpu
[pip3] torch_npu==2.7.1
[pip3] torchvision==0.22.1
[pip3] transformers==4.57.1
[conda] Could not collect
vLLM Version: 0.11.0
vLLM Ascend Version: 0.11.0rc1.dev184+g338f3b479.d20251105 (git sha: 338f3b479, date: 20251105)

ENV Variables:
ATB_OPSRUNNER_KERNEL_CACHE_LOCAL_COUNT=1
ATB_STREAM_SYNC_EVERY_RUNNER_ENABLE=0
ATB_OPSRUNNER_SETUP_CACHE_ENABLE=1
ATB_WORKSPACE_MEM_ALLOC_GLOBAL=1
ATB_DEVICE_TILING_BUFFER_BLOCK_NUM=32
ATB_STREAM_SYNC_EVERY_KERNEL_ENABLE=0
ATB_OPSRUNNER_KERNEL_CACHE_GLOABL_COUNT=5
ATB_HOME_PATH=/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1
ASCEND_TOOLKIT_HOME=/usr/local/Ascend/ascend-toolkit/latest
ATB_COMPARE_TILING_EVERY_KERNEL=0
ASCEND_OPP_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp
LD_LIBRARY_PATH=/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/aarch64:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling:/usr/local/Ascend/driver/lib64/common/:/usr/local/Ascend/driver/lib64/driver/:
ASCEND_AICPU_PATH=/usr/local/Ascend/ascend-toolkit/latest
ATB_STREAM_SYNC_EVERY_OPERATION_ENABLE=0
ASCEND_HOME_PATH=/usr/local/Ascend/ascend-toolkit/latest
ATB_MATMUL_SHUFFLE_K_ENABLE=1
ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=1
ATB_HOST_TILING_BUFFER_BLOCK_NUM=128
ATB_SHARE_MEMORY_NAME_SUFFIX=
TORCH_DEVICE_BACKEND_AUTOLOAD=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1


NPU:
+------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc3.7               Version: 24.1.rc3.7                                           |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip  Phy-ID              | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     Ascend910           | OK            | 175.0       39                0    / 0             |
| 0     0                   | 0000:9D:00.0  | 0           0    / 0          55833/ 65536         |
+------------------------------------------------------------------------------------------------+
| 0     Ascend910           | OK            | -           37                0    / 0             |
| 1     1                   | 0000:9F:00.0  | 0           0    / 0          55596/ 65536         |
+===========================+===============+====================================================+
| 1     Ascend910           | OK            | 185.0       37                0    / 0             |
| 0     2                   | 0000:99:00.0  | 0           0    / 0          55816/ 65536         |
+------------------------------------------------------------------------------------------------+
| 1     Ascend910           | OK            | -           37                0    / 0             |
| 1     3                   | 0000:9B:00.0  | 0           0    / 0          55731/ 65536         |
+===========================+===============+====================================================+
| 2     Ascend910           | OK            | 177.5       37                0    / 0             |
| 0     4                   | 0000:95:00.0  | 0           0    / 0          55827/ 65536         |
+------------------------------------------------------------------------------------------------+
| 2     Ascend910           | OK            | -           36                0    / 0             |
| 1     5                   | 0000:97:00.0  | 0           0    / 0          55720/ 65536         |
+===========================+===============+====================================================+
| 3     Ascend910           | OK            | 188.7       37                0    / 0             |
| 0     6                   | 0000:91:00.0  | 0           0    / 0          55818/ 65536         |
+------------------------------------------------------------------------------------------------+
| 3     Ascend910           | OK            | -           36                0    / 0             |
| 1     7                   | 0000:93:00.0  | 0           0    / 0          55730/ 65536         |
+===========================+===============+====================================================+
| 4     Ascend910           | OK            | 175.3       39                0    / 0             |
| 0     8                   | 0000:8D:00.0  | 0           0    / 0          55824/ 65536         |
+------------------------------------------------------------------------------------------------+
| 4     Ascend910           | OK            | -           36                0    / 0             |
| 1     9                   | 0000:8F:00.0  | 0           0    / 0          55597/ 65536         |
+===========================+===============+====================================================+
| 5     Ascend910           | OK            | 177.1       37                0    / 0             |
| 0     10                  | 0000:89:00.0  | 0           0    / 0          55818/ 65536         |
+------------------------------------------------------------------------------------------------+
| 5     Ascend910           | OK            | -           36                0    / 0             |
| 1     11                  | 0000:8B:00.0  | 0           0    / 0          55731/ 65536         |
+===========================+===============+====================================================+
| 6     Ascend910           | OK            | 177.7       36                0    / 0             |
| 0     12                  | 0000:85:00.0  | 0           0    / 0          55829/ 65536         |
+------------------------------------------------------------------------------------------------+
| 6     Ascend910           | OK            | -           35                0    / 0             |
| 1     13                  | 0000:87:00.0  | 0           0    / 0          55717/ 65536         |
+===========================+===============+====================================================+
| 7     Ascend910           | OK            | 184.1       36                0    / 0             |
| 0     14                  | 0000:81:00.0  | 0           0    / 0          55830/ 65536         |
+------------------------------------------------------------------------------------------------+
| 7     Ascend910           | OK            | -           37                0    / 0             |
| 1     15                  | 0000:83:00.0  | 0           0    / 0          55719/ 65536         |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 342468        | VLLMWorker_TP            | 52426                   |
| 0       1                 | 342472        | VLLMWorker_TP            | 52426                   |
+===========================+===============+====================================================+
| 1       0                 | 342493        | VLLMWorker_TP            | 52426                   |
| 1       1                 | 342516        | VLLMWorker_TP            | 52562                   |
+===========================+===============+====================================================+
| 2       0                 | 342539        | VLLMWorker_TP            | 52426                   |
| 2       1                 | 342562        | VLLMWorker_TP            | 52562                   |
+===========================+===============+====================================================+
| 3       0                 | 342585        | VLLMWorker_TP            | 52426                   |
| 3       1                 | 342608        | VLLMWorker_TP            | 52562                   |
+===========================+===============+====================================================+
| 4       0                 | 342631        | VLLMWorker_TP            | 52426                   |
| 4       1                 | 342654        | VLLMWorker_TP            | 52426                   |
+===========================+===============+====================================================+
| 5       0                 | 342677        | VLLMWorker_TP            | 52426                   |
| 5       1                 | 342700        | VLLMWorker_TP            | 52562                   |
+===========================+===============+====================================================+
| 6       0                 | 342723        | VLLMWorker_TP            | 52426                   |
| 6       1                 | 342746        | VLLMWorker_TP            | 52562                   |
+===========================+===============+====================================================+
| 7       0                 | 342769        | VLLMWorker_TP            | 52426                   |
| 7       1                 | 342792        | VLLMWorker_TP            | 52562                   |
+===========================+===============+====================================================+

CANN:
package_name=Ascend-cann-toolkit
version=8.3.RC1
innerversion=V100R001C23SPC001B235
compatible_version=[V100R001C15],[V100R001C18],[V100R001C19],[V100R001C20],[V100R001C21],[V100R001C23]
arch=aarch64
os=linux
path=/usr/local/Ascend/ascend-toolkit/8.3.RC1/aarch64-linux

[Meihan-chen]

DeepSeek-V3.2-Exp-W8A8 with aclgraph test OK, but there are accuracy issues

run command:

vllm serve vllm-ascend/DeepSeek-V3.2-Exp-W8A8 \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 8 \
--data-parallel-size 2 \
--enable-expert-parallel \
--seed 1024 \
--quantization ascend \
--served-model-name deepseek_v3.2 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92

client

curl http://localhost:8000/v1/completions \
>     -H "Content-Type: application/json" \
>     -d '{
>         "model": "deepseek_v3.2",
>         "prompt": "The future of AI is",
>         "max_tokens": 100,
>         "temperature": 0
>     }'
{"id":"cmpl-37a103d61c734d199b8d8b6310873ce3","object":"text_completion","created":1762340460,"model":"deepseek_v3.2","choices":[{"index":0,"text":"\ntttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

server：

(APIServer pid=182525) INFO:     Started server process [182525]
(APIServer pid=182525) INFO:     Waiting for application startup.
(APIServer pid=182525) INFO:     Application startup complete.
(Worker_DP1_TP6_EP14 pid=183601) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP1_EP9 pid=183353) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP7_EP15 pid=183653) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP0_EP8 pid=183343) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP3_EP11 pid=183449) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP4_EP12 pid=183499) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP5_EP13 pid=183551) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP2_EP10 pid=183401) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP2_EP2 pid=183400) /vllm-workspace/vllm-ascend/vllm_ascend/attention/sfa_v1.py:374: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
(Worker_DP0_TP3_EP3 pid=183451) /vllm-workspace/vllm-ascend/vllm_ascend/attention/sfa_v1.py:374: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
(Worker_DP0_TP2_EP2 pid=183400)   actual_query_lens = torch.tensor(query_lens[reqs_start:],
(Worker_DP0_TP1_EP1 pid=183352) /vllm-workspace/vllm-ascend/vllm_ascend/attention/sfa_v1.py:374: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
(Worker_DP0_TP3_EP3 pid=183451)   actual_query_lens = torch.tensor(query_lens[reqs_start:],
(Worker_DP0_TP6_EP6 pid=183599) /vllm-workspace/vllm-ascend/vllm_ascend/attention/sfa_v1.py:374: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
(Worker_DP0_TP4_EP4 pid=183501) /vllm-workspace/vllm-ascend/vllm_ascend/attention/sfa_v1.py:374: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
(Worker_DP0_TP1_EP1 pid=183352)   actual_query_lens = torch.tensor(query_lens[reqs_start:],
(Worker_DP0_TP0_EP0 pid=183342) /vllm-workspace/vllm-ascend/vllm_ascend/attention/sfa_v1.py:374: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
(Worker_DP0_TP6_EP6 pid=183599)   actual_query_lens = torch.tensor(query_lens[reqs_start:],
(Worker_DP0_TP4_EP4 pid=183501)   actual_query_lens = torch.tensor(query_lens[reqs_start:],
(Worker_DP0_TP7_EP7 pid=183649) /vllm-workspace/vllm-ascend/vllm_ascend/attention/sfa_v1.py:374: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
(Worker_DP0_TP0_EP0 pid=183342)   actual_query_lens = torch.tensor(query_lens[reqs_start:],
(Worker_DP0_TP7_EP7 pid=183649)   actual_query_lens = torch.tensor(query_lens[reqs_start:],
(Worker_DP0_TP5_EP5 pid=183549) /vllm-workspace/vllm-ascend/vllm_ascend/attention/sfa_v1.py:374: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
(Worker_DP0_TP5_EP5 pid=183549)   actual_query_lens = torch.tensor(query_lens[reqs_start:],
(Worker_DP0_TP6_EP6 pid=183599) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP3_EP3 pid=183451) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP0_EP0 pid=183342) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP1_EP1 pid=183352) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP2_EP2 pid=183400) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP4_EP4 pid=183501) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP7_EP7 pid=183649) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP5_EP5 pid=183549) INFO 11-06 01:59:48 [acl_graph.py:187] Replaying aclgraph
(APIServer pid=182525) INFO:     127.0.0.1:54692 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=182525) INFO 11-06 01:59:51 [loggers.py:127] Engine 000: Avg prompt throughput: 0.6 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=182525) INFO 11-06 02:00:01 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

[Meihan-chen]

DeepSeek-V3.1-W8A8 with aclgraph test OK

run command:

vllm serve vllm-ascend/DeepSeek-V3.1-W8A8 \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 8 \
--data-parallel-size 2 \
--enable-expert-parallel \
--seed 1024 \
--quantization ascend \
--served-model-name deepseek_v3.1 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92

client：

curl http://localhost:8000/v1/completions \
>     -H "Content-Type: application/json" \
>     -d '{
>         "model": "deepseek_v3.1",
>         "prompt": "The future of AI is",
>         "max_tokens": 100,
>         "temperature": 0
>     }'
{"id":"cmpl-33b5ce9492f4496f9b60aaa31028bee4","object":"text_completion","created":1762342808,"model":"deepseek_v3.1","choices":[{"index":0,"text":" here, and it's not just about smarter algorithms or faster processors. It's about a fundamental shift in how we interact with technology, and at the heart of this revolution is the concept of the AI agent. These aren't your average chatbots or simple automation tools. AI agents are sophisticated systems that can perceive their environment, make decisions, and take actions to achieve specific goals. They're the digital workforce of tomorrow, and they're poised to transform every industry from the ground up.\n\nBut what exactly are","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

serve:

(APIServer pid=338571) INFO:     Started server process [338571]
(APIServer pid=338571) INFO:     Waiting for application startup.
(APIServer pid=338571) INFO:     Application startup complete.
(Worker_DP1_TP7_EP15 pid=339537) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP6_EP14 pid=339491) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP1_EP9 pid=339261) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP3_EP11 pid=339350) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP7_EP7 pid=339534) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP6_EP6 pid=339488) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP2_EP10 pid=339304) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP4_EP4 pid=339399) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP3_EP3 pid=339351) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP2_EP2 pid=339305) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP1_EP1 pid=339260) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP0_EP0 pid=339252) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP0_TP5_EP5 pid=339442) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP4_EP12 pid=339396) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP0_EP8 pid=339253) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(Worker_DP1_TP5_EP13 pid=339445) INFO 11-05 11:40:08 [acl_graph.py:187] Replaying aclgraph
(APIServer pid=338571) INFO 11-05 11:40:14 [loggers.py:127] Engine 000: Avg prompt throughput: 0.6 tokens/s, Avg generation throughput: 9.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=338571) INFO:     127.0.0.1:57952 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=338571) INFO 11-05 11:40:24 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=338571) INFO 11-05 11:40:34 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

[menogrey]

❌DeepSeek-V3.2-Exp-bf16 with torchair test failed.

use export VLLM_ASCEND_ENABLE_MLAPO=1
test with ascend_scheduler and v1_scheduler on 2 A3 nodes, same error occurs.

Run command:
Node0:

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="**********"
local_ip="********"

export VLLM_ASCEND_ENABLE_MLAPO=1

export VLLM_USE_MODELSCOPE=True
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024

vllm serve Yanguan/DeepSeek-V3.2-Exp-bf16 \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-address $local_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 16 \
--seed 1024 \
--served-model-name deepseek_v3.2 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.9 \
--additional-config '{"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'

Node1:

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="************"
local_ip="*************"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="**************"

export VLLM_ASCEND_ENABLE_MLAPO=1

export VLLM_USE_MODELSCOPE=True
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024

vllm serve Yanguan/DeepSeek-V3.2-Exp-bf16 \
--host 0.0.0.0 \
--port 8000 \
--headless \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-start-rank 1 \
--data-parallel-address $node0_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 16 \
--seed 1024 \
--served-model-name deepseek_v3.2 \
--max-num-seqs 16 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--enable-expert-parallel \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--additional-config '{"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'
~                                                                  

(Worker_DP0_TP7_EP7 pid=621383) INFO 11-05 11:58:54 [default_loader.py:267] Loading weights took 576.68 seconds
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597] WorkerProc failed to start.
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597] Traceback (most recent call last):
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     worker = WorkerProc(*args, **kwargs)
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 437, in __init__
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     self.worker.load_model()
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 307, in load_model
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     self.model_runner.load_model()
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2645, in load_model
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     self.model = get_model(vllm_config=self.vllm_config)
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/__init__.py", line 119, in get_model
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     return loader.load_model(vllm_config=vllm_config,
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/base_loader.py", line 51, in load_model
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     process_weights_after_loading(model, model_config, target_device)
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/utils.py", line 122, in process_weights_after_loading
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     module.process_weights_after_loading(model_config.dtype)
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/attention/layer.py", line 367, in process_weights_after_loading
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     self.impl.process_weights_after_loading(act_dtype)
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/torchair/torchair_sfa.py", line 833, in process_weights_after_loading
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     self._process_weights_for_fused_mlapo(act_dtype)
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/torchair/torchair_sfa.py", line 848, in _process_weights_for_fused_mlapo
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     kv_a_proj_deq_scl = self.kv_a_proj_with_mqa.deq_scale.clone()
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1940, in __getattr__
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597]     raise AttributeError(
(Worker_DP0_TP0_EP0 pid=621376) ERROR 11-05 11:58:54 [multiproc_executor.py:597] AttributeError: 'AscendReplicatedLinear' object has no attribute 'deq_scale'
(Worker_DP0_TP0_EP0 pid=621376) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP3_EP3 pid=621379) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP5_EP5 pid=621381) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP9_EP9 pid=621385) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP6_EP6 pid=621382) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP4_EP4 pid=621380) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP13_EP13 pid=621389) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP10_EP10 pid=621386) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP12_EP12 pid=621388) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP7_EP7 pid=621383) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP11_EP11 pid=621387) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP15_EP15 pid=621391) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP14_EP14 pid=621390) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP6_EP6 pid=621382) [rank6]: Traceback (most recent call last):
(Worker_DP0_TP5_EP5 pid=621381) [rank5]: Traceback (most recent call last):
(Worker_DP0_TP6_EP6 pid=621382) [rank6]:   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py", line 60, in get_py_exception_dict
(Worker_DP0_TP6_EP6 pid=621382) [rank6]:     if isinstance(err_msg, tuple) and err_msg[0] and isinstance(err_msg[0], dict):
(Worker_DP0_TP5_EP5 pid=621381) [rank5]:   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py", line 60, in get_py_exception_dict
(Worker_DP0_TP6_EP6 pid=621382) [rank6]:                                       ~~~~~~~^^^
(Worker_DP0_TP5_EP5 pid=621381) [rank5]:     if isinstance(err_msg, tuple) and err_msg[0] and isinstance(err_msg[0], dict):
(Worker_DP0_TP6_EP6 pid=621382) [rank6]: IndexError: tuple index out of range
(Worker_DP0_TP5_EP5 pid=621381) [rank5]:                                       ~~~~~~~^^^
(Worker_DP0_TP5_EP5 pid=621381) [rank5]: IndexError: tuple index out of range
(Worker_DP0_TP4_EP4 pid=621380) [rank4]: Traceback (most recent call last):
(Worker_DP0_TP4_EP4 pid=621380) [rank4]:   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py", line 60, in get_py_exception_dict
(Worker_DP0_TP4_EP4 pid=621380) [rank4]:     if isinstance(err_msg, tuple) and err_msg[0] and isinstance(err_msg[0], dict):
(Worker_DP0_TP4_EP4 pid=621380) [rank4]:                                       ~~~~~~~^^^
(Worker_DP0_TP4_EP4 pid=621380) [rank4]: IndexError: tuple index out of range
(Worker_DP0_TP2_EP2 pid=621378) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP1_EP1 pid=621377) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP8_EP8 pid=621384) INFO 11-05 11:58:54 [multiproc_executor.py:558] Parent process exited, terminating worker
INFO 11-05 11:58:55 [__init__.py:36] Available plugins for group vllm.platform_plugins:
INFO 11-05 11:58:55 [__init__.py:38] - ascend -> vllm_ascend:register
INFO 11-05 11:58:55 [__init__.py:41] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 11-05 11:58:55 [__init__.py:207] Platform plugin ascend is activated
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597] WorkerProc failed to start.
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597] Traceback (most recent call last):
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     worker = WorkerProc(*args, **kwargs)
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 437, in __init__
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     self.worker.load_model()
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 307, in load_model
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     self.model_runner.load_model()
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2645, in load_model
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     self.model = get_model(vllm_config=self.vllm_config)
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/__init__.py", line 119, in get_model
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     return loader.load_model(vllm_config=vllm_config,
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/base_loader.py", line 51, in load_model
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     process_weights_after_loading(model, model_config, target_device)
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/utils.py", line 112, in process_weights_after_loading
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     quant_method.process_weights_after_loading(module)
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/linear.py", line 50, in process_weights_after_loading
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     layer.weight.data = torch_npu.npu_format_cast(
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1158, in __call__
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]     return self._op(*args, **(kwargs or {}))
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597] RuntimeError: SetPrecisionMode:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:175 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597] [ERROR] 2025-11-05-11:58:54 (PID:621382, Device:6, RankID:-1) ERR00100 PTA call acl api failed
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597] [Error]: The internal ACL of the system is incorrect.
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]         Rectify the fault based on the error information in the ascend log.
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597] E49999: Inner Error!
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597] E49999[PID: 621382] 2025-11-05-11:58:54.567.531 (E49999):  AOE failed to call InitCannKB.[FUNC:Initialize][FILE:python_adapter_manager.cc][LINE:47]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]         TraceBack (most recent call last):
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        Failed to initialize TeConfigInfo.
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeTeFusion][FILE:tbe_op_store_adapter.cc][LINE:1919]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        [GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1886]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:115]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:83]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:373]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        [Initialize][Ge]GEInitialize failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        [Init][Compiler]Init compiler failed[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:146]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597]        [Set][Options]OpCompileProcessor init failed![FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:146]
(Worker_DP0_TP6_EP6 pid=621382) ERROR 11-05 11:58:55 [multiproc_executor.py:597] 

env

Collecting environment information...
PyTorch version: 2.7.1+cpu
Is debug build: False

OS: Ubuntu 22.04.5 LTS (aarch64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Clang version: Could not collect
CMake version: version 4.1.2
Libc version: glibc-2.35

Python version: 3.11.13 (main, Nov  2 2025, 10:27:27) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.0-182.0.0.95.r1941_123.hce2.aarch64-aarch64-with-glibc2.35

CPU:
Architecture:                         aarch64
CPU op-mode(s):                       64-bit
Byte Order:                           Little Endian
CPU(s):                               320
On-line CPU(s) list:                  0-319
Vendor ID:                            HiSilicon
Model:                                0
Thread(s) per core:                   1
Core(s) per cluster:                  80
Socket(s):                            -
Cluster(s):                           4
Stepping:                             0x0
Frequency boost:                      disabled
CPU max MHz:                          3000.0000
CPU min MHz:                          400.0000
BogoMIPS:                             200.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint svei8mm svef32mm svef64mm svebf16 i8mm bf16 dgh rng ecv
L1d cache:                            20 MiB (320 instances)
L1i cache:                            20 MiB (320 instances)
L2 cache:                             400 MiB (320 instances)
L3 cache:                             560 MiB (8 instances)
NUMA node(s):                         8
NUMA node0 CPU(s):                    0-39
NUMA node1 CPU(s):                    40-79
NUMA node2 CPU(s):                    80-119
NUMA node3 CPU(s):                    120-159
NUMA node4 CPU(s):                    160-199
NUMA node5 CPU(s):                    200-239
NUMA node6 CPU(s):                    240-279
NUMA node7 CPU(s):                    280-319
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pyzmq==27.1.0
[pip3] torch==2.7.1+cpu
[pip3] torch_npu==2.7.1
[pip3] torchvision==0.22.1
[pip3] transformers==4.57.1
[conda] Could not collect
vLLM Version: 0.11.0
vLLM Ascend Version: 0.11.0rc1.dev187+g82135afef (git sha: 82135afef)

ENV Variables:
ATB_OPSRUNNER_KERNEL_CACHE_LOCAL_COUNT=1
ATB_STREAM_SYNC_EVERY_RUNNER_ENABLE=0
ATB_OPSRUNNER_SETUP_CACHE_ENABLE=1
ATB_WORKSPACE_MEM_ALLOC_GLOBAL=1
ATB_DEVICE_TILING_BUFFER_BLOCK_NUM=32
ATB_STREAM_SYNC_EVERY_KERNEL_ENABLE=0
ATB_OPSRUNNER_KERNEL_CACHE_GLOABL_COUNT=5
VLLM_USE_MODELSCOPE=true
ATB_HOME_PATH=/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1
ASCEND_TOOLKIT_HOME=/usr/local/Ascend/ascend-toolkit/latest
ATB_COMPARE_TILING_EVERY_KERNEL=0
ASCEND_OPP_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp
LD_LIBRARY_PATH=/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/aarch64:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling:/usr/local/Ascend/driver/lib64/common/:/usr/local/Ascend/driver/lib64/driver/:
ASCEND_AICPU_PATH=/usr/local/Ascend/ascend-toolkit/latest
ATB_STREAM_SYNC_EVERY_OPERATION_ENABLE=0
ASCEND_HOME_PATH=/usr/local/Ascend/ascend-toolkit/latest
ATB_MATMUL_SHUFFLE_K_ENABLE=1
ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=1
ATB_HOST_TILING_BUFFER_BLOCK_NUM=128
ATB_SHARE_MEMORY_NAME_SUFFIX=
TORCH_DEVICE_BACKEND_AUTOLOAD=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1


NPU:
+------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc3.7               Version: 24.1.rc3.7                                           |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip  Phy-ID              | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     Ascend910           | OK            | 174.4       39                0    / 0             |
| 0     0                   | 0000:9D:00.0  | 0           0    / 0          3440 / 65536         |
+------------------------------------------------------------------------------------------------+
| 0     Ascend910           | OK            | -           37                0    / 0             |
| 1     1                   | 0000:9F:00.0  | 0           0    / 0          3206 / 65536         |
+===========================+===============+====================================================+
| 1     Ascend910           | OK            | 184.8       37                0    / 0             |
| 0     2                   | 0000:99:00.0  | 0           0    / 0          3423 / 65536         |
+------------------------------------------------------------------------------------------------+
| 1     Ascend910           | OK            | -           37                0    / 0             |
| 1     3                   | 0000:9B:00.0  | 0           0    / 0          3204 / 65536         |
+===========================+===============+====================================================+
| 2     Ascend910           | OK            | 177.2       37                0    / 0             |
| 0     4                   | 0000:95:00.0  | 0           0    / 0          3436 / 65536         |
+------------------------------------------------------------------------------------------------+
| 2     Ascend910           | OK            | -           36                0    / 0             |
| 1     5                   | 0000:97:00.0  | 0           0    / 0          3195 / 65536         |
+===========================+===============+====================================================+
| 3     Ascend910           | OK            | 188.4       37                0    / 0             |
| 0     6                   | 0000:91:00.0  | 0           0    / 0          3425 / 65536         |
+------------------------------------------------------------------------------------------------+
| 3     Ascend910           | OK            | -           36                0    / 0             |
| 1     7                   | 0000:93:00.0  | 0           0    / 0          3207 / 65536         |
+===========================+===============+====================================================+
| 4     Ascend910           | OK            | 175.6       39                0    / 0             |
| 0     8                   | 0000:8D:00.0  | 0           0    / 0          3432 / 65536         |
+------------------------------------------------------------------------------------------------+
| 4     Ascend910           | OK            | -           37                0    / 0             |
| 1     9                   | 0000:8F:00.0  | 0           0    / 0          3210 / 65536         |
+===========================+===============+====================================================+
| 5     Ascend910           | OK            | 177.3       37                0    / 0             |
| 0     10                  | 0000:89:00.0  | 0           0    / 0          3424 / 65536         |
+------------------------------------------------------------------------------------------------+
| 5     Ascend910           | OK            | -           36                0    / 0             |
| 1     11                  | 0000:8B:00.0  | 0           0    / 0          3206 / 65536         |
+===========================+===============+====================================================+
| 6     Ascend910           | OK            | 177.6       36                0    / 0             |
| 0     12                  | 0000:85:00.0  | 0           0    / 0          3437 / 65536         |
+------------------------------------------------------------------------------------------------+
| 6     Ascend910           | OK            | -           36                0    / 0             |
| 1     13                  | 0000:87:00.0  | 0           0    / 0          3191 / 65536         |
+===========================+===============+====================================================+
| 7     Ascend910           | OK            | 183.9       36                0    / 0             |
| 0     14                  | 0000:81:00.0  | 0           0    / 0          3438 / 65536         |
+------------------------------------------------------------------------------------------------+
| 7     Ascend910           | OK            | -           38                0    / 0             |
| 1     15                  | 0000:83:00.0  | 0           0    / 0          3193 / 65536         |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| No running processes found in NPU 0                                                            |
+===========================+===============+====================================================+
| No running processes found in NPU 1                                                            |
+===========================+===============+====================================================+
| No running processes found in NPU 2                                                            |
+===========================+===============+====================================================+
| No running processes found in NPU 3                                                            |
+===========================+===============+====================================================+
| No running processes found in NPU 4                                                            |
+===========================+===============+====================================================+
| No running processes found in NPU 5                                                            |
+===========================+===============+====================================================+
| No running processes found in NPU 6                                                            |
+===========================+===============+====================================================+
| No running processes found in NPU 7                                                            |
+===========================+===============+====================================================+

CANN:
package_name=Ascend-cann-toolkit
version=8.3.RC1
innerversion=V100R001C23SPC001B235
compatible_version=[V100R001C15],[V100R001C18],[V100R001C19],[V100R001C20],[V100R001C21],[V100R001C23]
arch=aarch64
os=linux
path=/usr/local/Ascend/ascend-toolkit/8.3.RC1/aarch64-linux

[menogrey]

✅ DeepSeek-V3.2-Exp-W8A8 with torchair_graph_config test ok.

use export VLLM_ASCEND_ENABLE_MLAPO=0 or not set VLLM_ASCEND_ENABLE_MLAPO
test on ascend_scheduler and v1_scheduler

# with ascend_scheduler
export VLLM_ASCEND_ENABLE_MLAPO=0
export VLLM_USE_MODELSCOPE=true

vllm serve vllm-ascend/DeepSeek-V3.2-Exp-W8A8 \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 16 \
--seed 1024 \
--quantization ascend \
--max-num-seqs 16 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--enable-expert-parallel \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'

# with v1_scheduler
unset VLLM_ASCEND_ENABLE_MLAPO
export VLLM_USE_MODELSCOPE=true

vllm serve vllm-ascend/DeepSeek-V3.2-Exp-W8A8 \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 16 \
--seed 1024 \
--quantization ascend \
--max-num-seqs 16 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--enable-expert-parallel \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--additional-config '{"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'

(APIServer pid=272779) INFO 11-06 02:09:05 [launcher.py:42] Route: /invocations, Methods: POST
(APIServer pid=272779) INFO 11-06 02:09:05 [launcher.py:42] Route: /metrics, Methods: GET
(APIServer pid=272779) INFO:     Started server process [272779]
(APIServer pid=272779) INFO:     Waiting for application startup.
(APIServer pid=272779) INFO:     Application startup complete.

curl http://127.0.01:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "vllm-ascend/DeepSeek-V3.2-Exp-W8A8",
        "prompt": "The future of AI is",
        "max_tokens": 50,
        "temperature": 0
    }'

{"id":"cmpl-b031dbfd49f947e9b63cb351bfd671ad","object":"text_completion","created":1762395041,"model":"vllm-ascend/DeepSeek-V3.2-Exp-W8A8","choices":[{"index":0,"text":" here, and it's changing everything we know about technology. From self-driving cars to virtual assistants, AI is transforming the way we live and work. But what does the future hold for this rapidly evolving field? In this article, we'll explore the","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":56,"completion_tokens":50,"prompt_tokens_details":null},"kv_transfer_params":null}

[menogrey]

✅DeepSeek-V3.2-Exp-bf16 with torchair test ok.

not use VLLM_ASCEND_ENABLE_MLAPO
test with ascend_scheduler and v1_scheduler on 2 A3 nodes

Run command:
Node0:

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="**********"
local_ip="********"

export VLLM_USE_MODELSCOPE=True
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024

vllm serve Yanguan/DeepSeek-V3.2-Exp-bf16 \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-address $local_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 16 \
--seed 1024 \
--served-model-name deepseek_v3.2 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.9 \
--additional-config '{"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'

Node1:

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="************"
local_ip="*************"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="**************"

export VLLM_USE_MODELSCOPE=True
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024

vllm serve Yanguan/DeepSeek-V3.2-Exp-bf16 \
--host 0.0.0.0 \
--port 8000 \
--headless \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-start-rank 1 \
--data-parallel-address $node0_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 16 \
--seed 1024 \
--served-model-name deepseek_v3.2 \
--max-num-seqs 16 \
--max-model-len 17450 \
--max-num-batched-tokens 17450 \
--enable-expert-parallel \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--additional-config '{"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'
~                                                                  

(APIServer pid=645167) INFO 11-06 02:57:49 [launcher.py:42] Route: /invocations, Methods: POST
(APIServer pid=645167) INFO 11-06 02:57:49 [launcher.py:42] Route: /metrics, Methods: GET
(APIServer pid=645167) INFO:     Started server process [645167]
(APIServer pid=645167) INFO:     Waiting for application startup.
(APIServer pid=645167) INFO:     Application startup complete.

curl http://127.0.01:8000/v1/completions     -H "Content-Type: application/json"     -d '{
        "model": "deepseek_v3.2",
        "prompt": "The future of AI is",
        "max_tokens": 50,
        "temperature": 0
    }'

{"id":"cmpl-86b0442a1f4d45d3b9f55c420bda0511","object":"text_completion","created":1762398211,"model":"deepseek_v3.2","choices":[{"index":0,"text":" here, and it's changing everything. From how we work to how we interact with the world, artificial intelligence is no longer a distant dream but a present reality. This blog post will guide you through the latest advancements in AI, its impact on various","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":56,"completion_tokens":50,"prompt_tokens_details":null},"kv_transfer_params":null}

[22dimensions]

w4a4 quantization test

modelslim version: master branch ( commit id: e6aa67f04cac936f650f18814a09463f44fbdf99 )

conversion script:

python3 w4a4.py --model_path /root/.cache/modelscope/hub/models/Qwen/Qwen3-32B/ --save_directory /root/.cache/quant_models/Qwen3-32B-W4A4/ --calib_file ../common/qwen_qwen3_cot_w4a4.json --trust_remote_code True --batch_size 1

test scirpt:

from vllm import LLM, SamplingParams                                                                                                                                              

prompts = [                                                                                                                                                                       
    "Hello, my name is",                                                                                                                                                          
    "The future of AI is",                                                                                                                                                        
]                                                                                                                                                                                 

sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=40)                                                                                                           

llm = LLM(model="/root/.cache/quant_models/Qwen3-32B-W4A4",                                                                                                                   
          max_model_len=4096,                                                                                                                                                     
          quantization="ascend")                                                                                                                                                  
outputs = llm.generate(prompts, sampling_params)                                                                                                                                  
for output in outputs:      
    prompt = output.prompt                 
    generated_text = output.outputs[0].text             
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

result:

[shen-shanshan]

✅ Qwen3-VL Dense Model Test

Run:

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-8B-Instruct \
--max_model_len 16384 \
--tensor-parallel-size 2

logs

{"id":"chatcmpl-ea64665a7c6e4c1bb79af1e07b3ed895","object":"chat.completion","created":1762410812,"model":"/root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The text in the illustration is \"TONGYI Qwen\".\n\nIt looks like this:\n\n**TONGYI**\n**Qwen**\n\nThe text is presented in two lines. The top line, \"TONGYI\", is in a bold, uppercase, sans-serif font and colored blue. The bottom line, \"Qwen\", is in a bold, lowercase, sans-serif font and colored dark gray. To the left of the text is a stylized, geometric logo composed of inter","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":112,"total_tokens":212,"completion_tokens":100,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

✅ Qwen3-VL MoE Model Test

Run:

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct \
--max_model_len 16384 \
--tensor-parallel-size 4 \
--enable-expert-parallel

logs

RuntimeError: Worker failed with error ''NoneType' object has no attribute 'prepare'', please check the stack trace above for the root cause.

This issue is the same as #3891.

Need cherry-pick #3897 from main.

• Need merge: [MM][Bugfix] Add MoE verification for multi-modal models (#3897) #4027

After applying this PR, it worked well.

logs

{"id":"chatcmpl-8d3661649f52410087d73d0b70cc85ea","object":"chat.completion","created":1762431149,"model":"/root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The text in the illustration is \"TONGYI Qwen\".\n\n- \"TONGYI\" is written in uppercase letters in a light purple color.\n- \"Qwen\" is written in a darker gray color, with a capital \"Q\" and lowercase \"wen\".\n\nThe text is positioned to the right of a geometric logo that consists of interlocking shapes forming a hexagonal pattern. The overall design is clean and modern.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":112,"total_tokens":200,"completion_tokens":88,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

[Potabk]

✅Test Mooncake with no-layerwise

Mooncake tag: v0.3.7.post2

launch kubernetes cluster according to https://vllm-ascend.readthedocs.io/en/latest/developer_guide/contribution/multi_node_test.html#how-to-contribute

By default, Kubernetes will perform the case automatically. To clarify the test parameters, I have listed the test config file here.

test_name: "test DeepSeek-V3 disaggregated_prefill"
model: "vllm-ascend/DeepSeek-V3-W8A8"
num_nodes: 2
npu_per_node: 16
env_common:
  VLLM_USE_MODELSCOPE: true
  OMP_PROC_BIND: false
  OMP_NUM_THREADS: 100
  HCCL_BUFFSIZE: 1024
  SERVER_PORT: 8080
disaggregated_prefill:
  enabled: true
  prefiller_host_index: [0]
  decoder_host_index: [1]

deployment:
  -
    server_cmd: >
        vllm serve "vllm-ascend/DeepSeek-V3-W8A8"
        --host 0.0.0.0
        --port $SERVER_PORT
        --data-parallel-size 2
        --data-parallel-size-local 2
        --tensor-parallel-size 8
        --seed 1024
        --enforce-eager
        --enable-expert-parallel
        --max-num-seqs 16
        --max-model-len 8192
        --max-num-batched-tokens 8192
        --quantization ascend
        --trust-remote-code
        --no-enable-prefix-caching
        --gpu-memory-utilization 0.9
        --kv-transfer-config
        '{"kv_connector": "MooncakeConnector",
        "kv_role": "kv_producer",
        "kv_port": "30000",
        "engine_id": "0",
        "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
        "kv_connector_extra_config": {
                  "prefill": {
                          "dp_size": 2,
                          "tp_size": 8
                  },
                  "decode": {
                          "dp_size": 2,
                          "tp_size": 8
                  }
            }
        }'

  -
    server_cmd: >
        vllm serve "vllm-ascend/DeepSeek-V3-W8A8"
        --host 0.0.0.0
        --port $SERVER_PORT
        --data-parallel-size 2
        --data-parallel-size-local 2
        --tensor-parallel-size 8
        --seed 1024
        --quantization ascend
        --max-num-seqs 16
        --max-model-len 8192
        --max-num-batched-tokens 8192
        --enable-expert-parallel
        --trust-remote-code
        --no-enable-prefix-caching
        --gpu-memory-utilization 0.9
        --additional-config '{"torchair_graph_config":{"enabled":true}}'
        --kv-transfer-config
        '{"kv_connector": "MooncakeConnector",
        "kv_role": "kv_consumer",
        "kv_port": "30200",
        "engine_id": "1",
        "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
        "kv_connector_extra_config": {
                  "prefill": {
                          "dp_size": 2,
                          "tp_size": 8
                  },
                  "decode": {
                          "dp_size": 2,
                          "tp_size": 8
                  }
            }
        }'
benchmarks:
  acc:
    case_type: accuracy
    dataset_path: vllm-ascend/gsm8k-lite
    request_conf: vllm_api_general_chat
    dataset_conf: gsm8k/gsm8k_gen_0_shot_cot_chat_prompt
    max_out_len: 4096
    batch_size: 512
    baseline: 95
    threshold: 5

The service started normally.

dataset    version    metric    mode      vllm-api-general-chat
---------  ---------  --------  ------  -----------------------
gsm8k      7cd45e     accuracy  gen                       96.88
11/06 04:02:52 - AISBench - INFO - write summary to /root/workspace/source_code/vllm-ascend/outputs/default/20251106_040202/summary/summary_20251106_040202.txt
11/06 04:02:52 - AISBench - INFO - write csv to /root/workspace/source_code/vllm-ascend/outputs/default/20251106_040202/summary/summary_20251106_040202.csv
(APIServer pid=42349) INFO 11-06 04:02:52 [launcher.py:99] Shutting down FastAPI HTTP server.
(Worker_DP0_TP0_EP0 pid=43105) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP0_EP8 pid=43106) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP1_EP1 pid=43367) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP1_EP9 pid=43368) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP0_EP8 pid=43106) INFO 11-06 04:02:52 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_DP0_TP2_EP2 pid=43665) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP2_EP10 pid=43666) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP3_EP11 pid=43966) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP0_EP0 pid=43105) INFO 11-06 04:02:52 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_DP0_TP3_EP3 pid=43965) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP4_EP4 pid=44265) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP4_EP12 pid=44266) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP1_EP1 pid=43367) INFO 11-06 04:02:52 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_DP0_TP5_EP5 pid=44565) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP5_EP13 pid=44566) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP2_EP10 pid=43666) INFO 11-06 04:02:52 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_DP1_TP6_EP14 pid=44866) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP6_EP6 pid=44865) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP1_TP1_EP9 pid=43368) INFO 11-06 04:02:52 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_DP1_TP7_EP15 pid=45165) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_DP0_TP7_EP7 pid=45166) INFO 11-06 04:02:52 [multiproc_executor.py:558] Parent process exited, terminating worker
(APIServer pid=42349) INFO:     Shutting down
(APIServer pid=42349) INFO:     Waiting for application shutdown.
(APIServer pid=42349) INFO:     Application shutdown complete.
[2025-11-06 04:02:58] [INFO] Terminating proxy server process...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [42348]
[2025-11-06 04:02:58] [INFO] Proxy server process terminated.
PASSED