[Release]: Release checklist for v0.10.2rc1

[wangxiyuan]

Release Checklist

Release Version: v0.10.2rc1
Release Branch: main
Release Date: 2025/09/10
Release Manager: @wangxiyuan

Prepare Release Note

• Create a new issue for release feedback [v0.10.2rc1] FAQ / Feedback | 问题/反馈 #2874

• Write the release note PR. [Release] Add release note for v0.10.2rc1 #2921

  • Update the feedback issue link in docs/source/faqs.md

  • Add release note to docs/source/user_guide/release_notes.md

  • Update release version in README.md and README.zh.md

  • Update version info in docs/source/community/versioning_policy.md

  • Update contributor info in docs/source/community/contributors.md

  • Update package version in docs/conf.py

PR need Merge

• [Bug]: accuract test failed due to unexpected keyword argument 'prompt_token_ids' #2865

• Quantization accuracy problem [Quantization] register AscendQuantRMSNorm for quantization #2856 @22dimensions

• aclgraph quantization support @yiz-liu [Feat]support dynamic quantization in allgather #2841

• [main] mlp weight prefetch in Qwen Dense Models #2816

• Fix the bugs about operator registration by PyTorch Dispatcher #2786

• [Bug]: accuracy test failed due to Forward context is not set #2876 accuracy test fix @wxsIcey

• Revert "[Feat] Unquantized linear nz support (#2619)" #2896

• [Core] Disable the chunked prefill feature in Non-MLA LLMs #2894

• Bump vLLM version to v0.10.2rc2 #2902

• [Misc] Upgrade pta to 0822 #2875

• [Bug]: Qwen 235B Multi-Node + 1P1D + 2DP + 16TP + EP failed #2900

• GLM accuracy problem @shen-shanshan

  • [BugFix] Fix glm 4.5 moe accuracy bug. #2898
  • [BugFix] Fix world size bug in model_runner. #2897

• Qwen3-next support @MengqingCao @wangxiyuan

Will not be merged:

• Upgrade CANN version to 8.3.rc1.alpha002 #2880

Functional Test

• Qwen3 235B aclgraph test @zhangxinyuehfad
• Qwen 235B PD DP EP TP @zhangxinyuehfad
• deepseek v3 + torchair + all2allv @Potabk
• Async scheduler feature test @Potabk
• Qwen2.5 VL W8A8 @22dimensions
• Qwen3 8B + flash comm @Potabk
• aclgrah w8a8 @Potabk

Doc Test

• Tutorial is updated.
• User Guide is updated.
• Developer Guide is updated.

Prepare Artifacts

• Docker image is ready.
• Wheel package is ready.

Release Step

• Release note PR is merged.
• Post the release on GitHub release page.
• Generate official doc page on https://app.readthedocs.org/dashboard/
• Wait for the wheel package to be available on https://pypi.org/project/vllm-ascend
• Wait for the docker image to be available on https://quay.io/ascend/vllm-ascend
• Upload 310p wheel to Github release page
• Broadcast the release news (By message, blog , etc)
• Close this issue

[zhangxinyuehfad]

✅ Qwen3 235B Multi-Node DP4 + TP8 + EP test ok;

node0:

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="enp23s0f3"
local_ip="172.22.0.188"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=1024

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
--host 0.0.0.0 \
--port 8004 \
--data-parallel-size 4 \
--data-parallel-size-local 2 \
--data-parallel-address $local_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--seed 1024 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 32768 \
--trust-remote-code \
--served-model-name qwen_235 \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 &

node1:

#!/bin/sh

nic_name="enp23s0f3"
local_ip="172.22.0.212"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=1024


vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
--host 0.0.0.0 \
--port 8004 \
--headless \
--data-parallel-size 4 \
--data-parallel-size-local 2 \
--data-parallel-start-rank 2 \
--data-parallel-address "172.22.0.188" \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--seed 1024 \
--served-model-name qwen_235 \
--max-num-seqs 16 \
--max-model-len 32768 \
--enable-expert-parallel \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 &

star serve log:

(VllmWorker TP5 pid=262305) INFO 09-11 08:50:39 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP7 pid=262309) INFO 09-11 08:50:39 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP6 pid=262307) INFO 09-11 08:50:39 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP3 pid=262301) INFO 09-11 08:50:39 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP1 pid=262297) INFO 09-11 08:50:39 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP4 pid=262302) INFO 09-11 08:50:39 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP2 pid=262298) INFO 09-11 08:50:39 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP0 pid=262296) INFO 09-11 08:50:39 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(EngineCore_1 pid=261887) INFO 09-11 08:50:39 [core.py:214] init engine (profile, create kv cache, warmup model) took 76.09 seconds
(VllmWorker TP1 pid=262304) INFO 09-11 08:50:40 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
Capturing ACL graphs (mixed prefill-decode, PIECEWISE): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:36<00:00, 12.16s/it]
(VllmWorker TP5 pid=262311) INFO 09-11 08:50:40 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP0 pid=262303) INFO 09-11 08:50:40 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP4 pid=262310) INFO 09-11 08:50:40 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP6 pid=262312) INFO 09-11 08:50:40 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP7 pid=262313) INFO 09-11 08:50:40 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP3 pid=262308) INFO 09-11 08:50:40 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(VllmWorker TP2 pid=262306) INFO 09-11 08:50:40 [model_runner_v1.py:2590] Graph capturing finished in 36 secs, took -2.77 GiB
(EngineCore_0 pid=261886) INFO 09-11 08:50:40 [core.py:214] init engine (profile, create kv cache, warmup model) took 76.80 seconds
INFO 09-11 08:50:40 [coordinator.py:187] All engine subscriptions received by DP coordinator
(EngineCore_1 pid=261887) INFO 09-11 08:50:40 [platform.py:200] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
(EngineCore_0 pid=261886) INFO 09-11 08:50:40 [platform.py:200] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
(EngineCore_1 pid=261887) INFO 09-11 08:50:40 [utils.py:340] Calculated maximum supported batch sizes for ACL graph: 3
(EngineCore_1 pid=261887) WARNING 09-11 08:50:40 [utils.py:343] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
(EngineCore_1 pid=261887) INFO 09-11 08:50:40 [utils.py:373] No adjustment needed for ACL graph batch sizes: Qwen3MoeForCausalLM model (layers: 94) with 3 sizes
(EngineCore_0 pid=261886) INFO 09-11 08:50:40 [utils.py:340] Calculated maximum supported batch sizes for ACL graph: 3
(EngineCore_0 pid=261886) WARNING 09-11 08:50:40 [utils.py:343] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
(EngineCore_0 pid=261886) INFO 09-11 08:50:40 [utils.py:373] No adjustment needed for ACL graph batch sizes: Qwen3MoeForCausalLM model (layers: 94) with 3 sizes
(APIServer pid=261615) INFO 09-11 08:50:40 [loggers.py:142] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 20454
(APIServer pid=261615) INFO 09-11 08:50:40 [loggers.py:142] Engine 001: vllm cache_config_info with initialization after num_gpu_blocks is: 20454
(APIServer pid=261615) INFO 09-11 08:50:40 [loggers.py:142] Engine 002: vllm cache_config_info with initialization after num_gpu_blocks is: 20454
(APIServer pid=261615) INFO 09-11 08:50:40 [loggers.py:142] Engine 003: vllm cache_config_info with initialization after num_gpu_blocks is: 20454
(APIServer pid=261615) INFO 09-11 08:50:40 [api_server.py:1611] Supported_tasks: ['generate']
(APIServer pid=261615) WARNING 09-11 08:50:40 [__init__.py:1625] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=261615) INFO 09-11 08:50:40 [serving_responses.py:120] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=261615) INFO 09-11 08:50:40 [serving_chat.py:134] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=261615) INFO 09-11 08:50:40 [serving_completion.py:77] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=261615) INFO 09-11 08:50:40 [api_server.py:1880] Starting vLLM API server 0 on http://0.0.0.0:8004
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:36] Available routes are:
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /docs, Methods: HEAD, GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /redoc, Methods: HEAD, GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /health, Methods: GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /load, Methods: GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /ping, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /ping, Methods: GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /tokenize, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /detokenize, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/models, Methods: GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /version, Methods: GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/responses, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/chat/completions, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/completions, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/embeddings, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /pooling, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /classify, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /score, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/score, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/audio/translations, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /rerank, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v1/rerank, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /v2/rerank, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /invocations, Methods: POST
(APIServer pid=261615) INFO 09-11 08:50:40 [launcher.py:44] Route: /metrics, Methods: GET
(APIServer pid=261615) INFO:     Started server process [261615]
(APIServer pid=261615) INFO:     Waiting for application startup.
(APIServer pid=261615) INFO:     Application startup complete.

output:

root@liteserver-for-vllm-ascend-00003:/vllm-workspace/vllm-ascend# curl http://localhost:8004/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen_235",
        "prompt": "东莞有什么好玩的地方",
        "max_tokens": 100,
        "temperature": 0.5
    }'
{"id":"cmpl-afc1d6af2c424bc898ea688a77929e2f","object":"text_completion","created":1757590550,"model":"qwen_235","choices":[{"index":0,"text":"？ 东莞是一个历史悠久、文化底蕴深厚的城市，拥有许多值得一游的景点。以下是一些东莞好玩的地方推荐：\n\n1. 东莞可园：岭南四大名园之一，是清代广东四大名园之一，具有典型的岭南园林特色，是东莞的标志性景点。\n\n2. 鸦片战争博物馆：位于虎门，是全国重点文物保护单位，展示了鸦片战争时期的历史和林则徐的生平事迹。\n\n3. 虎门大桥：","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":4,"total_tokens":104,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}root@liteserver-for-vllm-ascend-00003:/vllm-workspace/vllm-ascend#

[zhangxinyuehfad]

❌ Qwen 235B Multi-Node + 1P1D + 2DP + 16TP + EP

command

bash gen_ranktable.sh --ips 172.22.0.188 172.22.0.212  \
  --npus-per-node 16 --network-card-name enp23s0f3 --prefill-device-cnt 16 --decode-device-cnt 16

node0:

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="enp23s0f3"
local_ip="172.22.0.188"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=1280
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export PYTORCH_NPU_ALLOC_CONF="max_split_size_mb:256"

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
--host 0.0.0.0 \
--port 8004 \
--data-parallel-size 2 \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--seed 1024 \
--enforce-eager \
--no-enable-prefix-caching \
--enable-expert-parallel \
--max-num-seqs 8 \
--max-model-len 8192 \
--trust-remote-code \
--served-model-name qwen_235 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
    "kv_buffer_device": "npu",
    "kv_role": "kv_producer",
    "kv_parallel_size": 1,
    "kv_port": "20001",
    "engine_id": "0",
    "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'  &

node1:

#!/bin/sh

nic_name="enp23s0f3"
local_ip="172.22.0.212"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=1280
export PYTORCH_NPU_MAX_SPLIT_SIZE_MB=128
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export ASCEND_LAUNCH_BLOCKING=1
export PYTORCH_NPU_ALLOC_CONF="max_split_size_mb:256"


vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
--host 0.0.0.0 \
--port 8004 \
--data-parallel-size 2 \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--seed 1024 \
--enforce-eager \
--served-model-name qwen_235 \
--max-num-seqs 8 \
--max-model-len 8192 \
--enable-expert-parallel \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.9 \
--kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_consumer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'  &

proxy:

python load_balance_proxy_server_example.py --host 0.0.0.0  --port 1025 --prefiller-hosts 172.22.0.188 --prefiller-port 8004 --decoder-hosts 172.22.0.212 --decoder-ports 8004\

error log：

node0:

(EngineCore_DP0 pid=851637) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:276] Delaying free of 1 blocks for request cmpl-0ed7171a-19e2-4f99-85b2-3e9fea46deb7-0
(APIServer pid=851368) INFO:     172.22.0.188:46216 - "POST /v1/completions HTTP/1.1" 200 OK
(Worker_DP0_TP3_EP3 pid=852060) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:367] LLMDataDistCMgrConnectorWorker: Receive message from cluster 20
(Worker_DP0_TP3_EP3 pid=852060) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.3.196_192.45.3.196
(Worker_DP0_TP3_EP3 pid=852060) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP3_EP3 pid=852060) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '3', 'device_ip': '192.23.3.196', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '3', 'device_ip': '192.45.3.196', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP3_EP3 pid=852060) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.3.196_192.45.3.196
(Worker_DP0_TP3_EP3 pid=852060) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {4: 0, 20: 1}
(Worker_DP0_TP2_EP2 pid=852052) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:367] LLMDataDistCMgrConnectorWorker: Receive message from cluster 19
(Worker_DP0_TP6_EP6 pid=852063) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:367] LLMDataDistCMgrConnectorWorker: Receive message from cluster 23
(Worker_DP0_TP2_EP2 pid=852052) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.2.197_192.45.2.197
(Worker_DP0_TP2_EP2 pid=852052) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP2_EP2 pid=852052) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '2', 'device_ip': '192.23.2.197', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '2', 'device_ip': '192.45.2.197', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP6_EP6 pid=852063) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.2.193_192.45.2.193
(Worker_DP0_TP2_EP2 pid=852052) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.2.197_192.45.2.197
(Worker_DP0_TP6_EP6 pid=852063) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP6_EP6 pid=852063) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '6', 'device_ip': '192.23.2.193', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '6', 'device_ip': '192.45.2.193', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP2_EP2 pid=852052) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {3: 0, 19: 1}
(Worker_DP0_TP6_EP6 pid=852063) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.2.193_192.45.2.193
(Worker_DP0_TP6_EP6 pid=852063) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {7: 0, 23: 1}
(Worker_DP0_TP4_EP4 pid=852061) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:367] LLMDataDistCMgrConnectorWorker: Receive message from cluster 21
(Worker_DP0_TP5_EP5 pid=852062) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:367] LLMDataDistCMgrConnectorWorker: Receive message from cluster 22
(Worker_DP0_TP7_EP7 pid=852064) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:367] LLMDataDistCMgrConnectorWorker: Receive message from cluster 24
(Worker_DP0_TP2_EP2 pid=852052) Exception in thread metadata_agent_listener:
(Worker_DP0_TP3_EP3 pid=852060) Exception in thread metadata_agent_listener:
(Worker_DP0_TP4_EP4 pid=852061) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.2.195_192.45.2.195
(Worker_DP0_TP2_EP2 pid=852052) Traceback (most recent call last):
(Worker_DP0_TP2_EP2 pid=852052)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
(Worker_DP0_TP5_EP5 pid=852062) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.3.194_192.45.3.194
(Worker_DP0_TP4_EP4 pid=852061) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP4_EP4 pid=852061) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '4', 'device_ip': '192.23.2.195', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '4', 'device_ip': '192.45.2.195', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP3_EP3 pid=852060) Traceback (most recent call last):
(Worker_DP0_TP5_EP5 pid=852062) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP3_EP3 pid=852060)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
(Worker_DP0_TP5_EP5 pid=852062) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '5', 'device_ip': '192.23.3.194', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '5', 'device_ip': '192.45.3.194', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP7_EP7 pid=852064) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.3.192_192.45.3.192
(Worker_DP0_TP4_EP4 pid=852061) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.2.195_192.45.2.195
(Worker_DP0_TP5_EP5 pid=852062) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.3.194_192.45.3.194
(Worker_DP0_TP0_EP0 pid=852049) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:367] LLMDataDistCMgrConnectorWorker: Receive message from cluster 17
(Worker_DP0_TP4_EP4 pid=852061) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {5: 0, 21: 1}
(Worker_DP0_TP5_EP5 pid=852062) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {6: 0, 22: 1}
(Worker_DP0_TP7_EP7 pid=852064) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP7_EP7 pid=852064) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '7', 'device_ip': '192.23.3.192', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '7', 'device_ip': '192.45.3.192', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP7_EP7 pid=852064) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.3.192_192.45.3.192
(Worker_DP0_TP7_EP7 pid=852064) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {8: 0, 24: 1}
(Worker_DP0_TP2_EP2 pid=852052)     self.run()
(Worker_DP0_TP1_EP1 pid=852051) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:367] LLMDataDistCMgrConnectorWorker: Receive message from cluster 18
(Worker_DP0_TP0_EP0 pid=852049) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.2.199_192.45.2.199
(Worker_DP0_TP6_EP6 pid=852063) Exception in thread metadata_agent_listener:
(Worker_DP0_TP2_EP2 pid=852052)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 982, in run
(Worker_DP0_TP6_EP6 pid=852063) Traceback (most recent call last):
(Worker_DP0_TP6_EP6 pid=852063)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
(Worker_DP0_TP0_EP0 pid=852049) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP3_EP3 pid=852060)     self.run()
(Worker_DP0_TP0_EP0 pid=852049) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '0', 'device_ip': '192.23.2.199', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '0', 'device_ip': '192.45.2.199', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP3_EP3 pid=852060)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 982, in run
(Worker_DP0_TP0_EP0 pid=852049) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.2.199_192.45.2.199
(Worker_DP0_TP0_EP0 pid=852049) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {1: 0, 17: 1}
(Worker_DP0_TP1_EP1 pid=852051) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.3.198_192.45.3.198
(Worker_DP0_TP2_EP2 pid=852052)     self._target(*self._args, **self._kwargs)
(Worker_DP0_TP2_EP2 pid=852052)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 371, in listen_for_agent_metadata_req
(Worker_DP0_TP3_EP3 pid=852060)     self._target(*self._args, **self._kwargs)
(Worker_DP0_TP1_EP1 pid=852051) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP3_EP3 pid=852060)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 371, in listen_for_agent_metadata_req
(Worker_DP0_TP2_EP2 pid=852052)     self.add_remote_agent(decode_msg)
(Worker_DP0_TP6_EP6 pid=852063)     self.run()
(Worker_DP0_TP2_EP2 pid=852052)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 691, in add_remote_agent
(Worker_DP0_TP1_EP1 pid=852051) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '1', 'device_ip': '192.23.3.198', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '1', 'device_ip': '192.45.3.198', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP1_EP1 pid=852051) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.3.198_192.45.3.198
(Worker_DP0_TP6_EP6 pid=852063)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 982, in run
(Worker_DP0_TP3_EP3 pid=852060)     self.add_remote_agent(decode_msg)
(Worker_DP0_TP1_EP1 pid=852051) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {2: 0, 18: 1}
(Worker_DP0_TP3_EP3 pid=852060)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 691, in add_remote_agent
(Worker_DP0_TP2_EP2 pid=852052)     comm_id = self.llm_datadist.link(comm_name, cluster_rank_info,
(Worker_DP0_TP2_EP2 pid=852052)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP6_EP6 pid=852063)     self._target(*self._args, **self._kwargs)
(Worker_DP0_TP2_EP2 pid=852052)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/v2/llm_datadist.py", line 334, in link
(Worker_DP0_TP6_EP6 pid=852063)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 371, in listen_for_agent_metadata_req
(Worker_DP0_TP3_EP3 pid=852060)     comm_id = self.llm_datadist.link(comm_name, cluster_rank_info,
(Worker_DP0_TP3_EP3 pid=852060)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=852060)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/v2/llm_datadist.py", line 334, in link
(Worker_DP0_TP2_EP2 pid=852052)     handle_llm_status(ret, '[link]', cluster_rank_info)
(Worker_DP0_TP6_EP6 pid=852063)     self.add_remote_agent(decode_msg)
(Worker_DP0_TP2_EP2 pid=852052)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/status.py", line 125, in handle_llm_status
(Worker_DP0_TP2_EP2 pid=852052)     raise LLMException(f"{func_name} failed, error code is {code_2_status(status)}, {other_info}.",
(Worker_DP0_TP2_EP2 pid=852052) llm_datadist.status.LLMException: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {3: 0, 19: 1}.
(Worker_DP0_TP3_EP3 pid=852060)     handle_llm_status(ret, '[link]', cluster_rank_info)
(Worker_DP0_TP3_EP3 pid=852060)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/status.py", line 125, in handle_llm_status
(Worker_DP0_TP6_EP6 pid=852063)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 691, in add_remote_agent
(Worker_DP0_TP3_EP3 pid=852060)     raise LLMException(f"{func_name} failed, error code is {code_2_status(status)}, {other_info}.",
(Worker_DP0_TP3_EP3 pid=852060) llm_datadist.status.LLMException: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {4: 0, 20: 1}.
(Worker_DP0_TP6_EP6 pid=852063)     comm_id = self.llm_datadist.link(comm_name, cluster_rank_info,
(Worker_DP0_TP6_EP6 pid=852063)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP6_EP6 pid=852063)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/v2/llm_datadist.py", line 334, in link
(Worker_DP0_TP5_EP5 pid=852062) Exception in thread metadata_agent_listener:
(Worker_DP0_TP5_EP5 pid=852062) Traceback (most recent call last):
(Worker_DP0_TP5_EP5 pid=852062)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
(Worker_DP0_TP4_EP4 pid=852061) Exception in thread metadata_agent_listener:
(Worker_DP0_TP4_EP4 pid=852061) Traceback (most recent call last):
(Worker_DP0_TP4_EP4 pid=852061)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
(Worker_DP0_TP6_EP6 pid=852063)     handle_llm_status(ret, '[link]', cluster_rank_info)
(Worker_DP0_TP6_EP6 pid=852063)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/status.py", line 125, in handle_llm_status
(Worker_DP0_TP6_EP6 pid=852063)     raise LLMException(f"{func_name} failed, error code is {code_2_status(status)}, {other_info}.",
(Worker_DP0_TP6_EP6 pid=852063) llm_datadist.status.LLMException: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {7: 0, 23: 1}.
(Worker_DP0_TP7_EP7 pid=852064) Exception in thread metadata_agent_listener:
(Worker_DP0_TP7_EP7 pid=852064) Traceback (most recent call last):
(Worker_DP0_TP4_EP4 pid=852061)     self.run()
(Worker_DP0_TP5_EP5 pid=852062)     self.run()
(Worker_DP0_TP7_EP7 pid=852064)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
(Worker_DP0_TP5_EP5 pid=852062)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 982, in run
(Worker_DP0_TP4_EP4 pid=852061)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 982, in run
(Worker_DP0_TP4_EP4 pid=852061)     self._target(*self._args, **self._kwargs)
(Worker_DP0_TP4_EP4 pid=852061)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 371, in listen_for_agent_metadata_req
(Worker_DP0_TP5_EP5 pid=852062)     self._target(*self._args, **self._kwargs)
(Worker_DP0_TP0_EP0 pid=852049) Exception in thread metadata_agent_listener:
(Worker_DP0_TP7_EP7 pid=852064)     self.run()
(Worker_DP0_TP4_EP4 pid=852061)     self.add_remote_agent(decode_msg)
(Worker_DP0_TP5_EP5 pid=852062)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 371, in listen_for_agent_metadata_req
(Worker_DP0_TP0_EP0 pid=852049) Traceback (most recent call last):
(Worker_DP0_TP4_EP4 pid=852061)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 691, in add_remote_agent
(Worker_DP0_TP0_EP0 pid=852049)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
(Worker_DP0_TP7_EP7 pid=852064)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 982, in run
(Worker_DP0_TP5_EP5 pid=852062)     self.add_remote_agent(decode_msg)
(Worker_DP0_TP4_EP4 pid=852061)     comm_id = self.llm_datadist.link(comm_name, cluster_rank_info,
(Worker_DP0_TP4_EP4 pid=852061)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP4_EP4 pid=852061)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/v2/llm_datadist.py", line 334, in link
(Worker_DP0_TP7_EP7 pid=852064)     self._target(*self._args, **self._kwargs)
(Worker_DP0_TP5_EP5 pid=852062)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 691, in add_remote_agent
(Worker_DP0_TP7_EP7 pid=852064)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 371, in listen_for_agent_metadata_req
(Worker_DP0_TP1_EP1 pid=852051) Exception in thread metadata_agent_listener:
(Worker_DP0_TP1_EP1 pid=852051) Traceback (most recent call last):
(Worker_DP0_TP1_EP1 pid=852051)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
(Worker_DP0_TP7_EP7 pid=852064)     self.add_remote_agent(decode_msg)
(Worker_DP0_TP4_EP4 pid=852061)     handle_llm_status(ret, '[link]', cluster_rank_info)
(Worker_DP0_TP7_EP7 pid=852064)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 691, in add_remote_agent
(Worker_DP0_TP4_EP4 pid=852061)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/status.py", line 125, in handle_llm_status
(Worker_DP0_TP0_EP0 pid=852049)     self.run()
(Worker_DP0_TP5_EP5 pid=852062)     comm_id = self.llm_datadist.link(comm_name, cluster_rank_info,
(Worker_DP0_TP4_EP4 pid=852061)     raise LLMException(f"{func_name} failed, error code is {code_2_status(status)}, {other_info}.",
(Worker_DP0_TP0_EP0 pid=852049)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 982, in run
(Worker_DP0_TP4_EP4 pid=852061) llm_datadist.status.LLMException: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {5: 0, 21: 1}.
(Worker_DP0_TP5_EP5 pid=852062)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP7_EP7 pid=852064)     comm_id = self.llm_datadist.link(comm_name, cluster_rank_info,
(Worker_DP0_TP5_EP5 pid=852062)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/v2/llm_datadist.py", line 334, in link
(Worker_DP0_TP7_EP7 pid=852064)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP7_EP7 pid=852064)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/v2/llm_datadist.py", line 334, in link
(Worker_DP0_TP7_EP7 pid=852064)     handle_llm_status(ret, '[link]', cluster_rank_info)
(Worker_DP0_TP5_EP5 pid=852062)     handle_llm_status(ret, '[link]', cluster_rank_info)
(Worker_DP0_TP7_EP7 pid=852064)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/status.py", line 125, in handle_llm_status
(Worker_DP0_TP5_EP5 pid=852062)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/status.py", line 125, in handle_llm_status
(Worker_DP0_TP0_EP0 pid=852049)     self._target(*self._args, **self._kwargs)
(Worker_DP0_TP0_EP0 pid=852049)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 371, in listen_for_agent_metadata_req
(Worker_DP0_TP1_EP1 pid=852051)     self.run()
(Worker_DP0_TP7_EP7 pid=852064)     raise LLMException(f"{func_name} failed, error code is {code_2_status(status)}, {other_info}.",
(Worker_DP0_TP7_EP7 pid=852064) llm_datadist.status.LLMException: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {8: 0, 24: 1}.
(Worker_DP0_TP1_EP1 pid=852051)   File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 982, in run
(Worker_DP0_TP5_EP5 pid=852062)     raise LLMException(f"{func_name} failed, error code is {code_2_status(status)}, {other_info}.",
(Worker_DP0_TP5_EP5 pid=852062) llm_datadist.status.LLMException: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {6: 0, 22: 1}.
(Worker_DP0_TP0_EP0 pid=852049)     self.add_remote_agent(decode_msg)
(Worker_DP0_TP1_EP1 pid=852051)     self._target(*self._args, **self._kwargs)
(Worker_DP0_TP1_EP1 pid=852051)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 371, in listen_for_agent_metadata_req
(Worker_DP0_TP0_EP0 pid=852049)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 691, in add_remote_agent
(Worker_DP0_TP1_EP1 pid=852051)     self.add_remote_agent(decode_msg)
(Worker_DP0_TP1_EP1 pid=852051)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/llmdatadist_c_mgr_connector.py", line 691, in add_remote_agent
(Worker_DP0_TP0_EP0 pid=852049)     comm_id = self.llm_datadist.link(comm_name, cluster_rank_info,
(Worker_DP0_TP0_EP0 pid=852049)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP1_EP1 pid=852051)     comm_id = self.llm_datadist.link(comm_name, cluster_rank_info,
(Worker_DP0_TP0_EP0 pid=852049)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/v2/llm_datadist.py", line 334, in link
(Worker_DP0_TP1_EP1 pid=852051)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP1_EP1 pid=852051)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/v2/llm_datadist.py", line 334, in link
(Worker_DP0_TP1_EP1 pid=852051)     handle_llm_status(ret, '[link]', cluster_rank_info)
(Worker_DP0_TP1_EP1 pid=852051)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/status.py", line 125, in handle_llm_status
(Worker_DP0_TP0_EP0 pid=852049)     handle_llm_status(ret, '[link]', cluster_rank_info)
(Worker_DP0_TP0_EP0 pid=852049)   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/llm_datadist/status.py", line 125, in handle_llm_status
(Worker_DP0_TP1_EP1 pid=852051)     raise LLMException(f"{func_name} failed, error code is {code_2_status(status)}, {other_info}.",
(Worker_DP0_TP1_EP1 pid=852051) llm_datadist.status.LLMException: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {2: 0, 18: 1}.
(Worker_DP0_TP0_EP0 pid=852049)     raise LLMException(f"{func_name} failed, error code is {code_2_status(status)}, {other_info}.",
(Worker_DP0_TP0_EP0 pid=852049) llm_datadist.status.LLMException: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {1: 0, 17: 1}.
(APIServer pid=851368) INFO 09-12 03:40:07 [loggers.py:123] Engine 000: Avg prompt throughput: 0.4 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=851368) INFO 09-12 03:40:17 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=851368) Task was destroyed but it is pending!
(APIServer pid=851368) task: <Task pending name='Task-17' coro=<AsyncMicrobatchTokenizer._batch_encode_loop() done, defined at /vllm-workspace/vllm/vllm/utils/__init__.py:580> wait_for=<Future cancelled>>

node1:

(APIServer pid=139846) INFO:     Started server process [139846]
(APIServer pid=139846) INFO:     Waiting for application startup.
(APIServer pid=139846) INFO:     Application startup complete.
(Worker_DP0_TP4_EP4 pid=140539) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:737] Try request remote metadata from socket......
(Worker_DP0_TP3_EP3 pid=140538) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:737] Try request remote metadata from socket......
(Worker_DP0_TP1_EP1 pid=140528) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:737] Try request remote metadata from socket......
(Worker_DP0_TP6_EP6 pid=140541) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:737] Try request remote metadata from socket......
(Worker_DP0_TP2_EP2 pid=140529) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:737] Try request remote metadata from socket......
(Worker_DP0_TP7_EP7 pid=140542) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:737] Try request remote metadata from socket......
(Worker_DP0_TP5_EP5 pid=140540) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:737] Try request remote metadata from socket......
(Worker_DP0_TP3_EP3 pid=140538) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:743] recving metadata: LLMDataDistCMgrAgentMetadata(super_pod_id='11', server_id='172.22.0.188', device_id='3', device_ip='192.23.3.196', super_device_id='98369551', cluster_id=4)
(Worker_DP0_TP3_EP3 pid=140538) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.3.196_192.45.3.196
(Worker_DP0_TP0_EP0 pid=140527) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:737] Try request remote metadata from socket......
(Worker_DP0_TP3_EP3 pid=140538) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP3_EP3 pid=140538) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '3', 'device_ip': '192.23.3.196', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '3', 'device_ip': '192.45.3.196', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP3_EP3 pid=140538) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.3.196_192.45.3.196
(Worker_DP0_TP3_EP3 pid=140538) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {4: 0, 20: 1}
(Worker_DP0_TP2_EP2 pid=140529) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:743] recving metadata: LLMDataDistCMgrAgentMetadata(super_pod_id='11', server_id='172.22.0.188', device_id='2', device_ip='192.23.2.197', super_device_id='98369551', cluster_id=3)
(Worker_DP0_TP2_EP2 pid=140529) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.2.197_192.45.2.197
(Worker_DP0_TP2_EP2 pid=140529) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP6_EP6 pid=140541) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:743] recving metadata: LLMDataDistCMgrAgentMetadata(super_pod_id='11', server_id='172.22.0.188', device_id='6', device_ip='192.23.2.193', super_device_id='98369551', cluster_id=7)
(Worker_DP0_TP2_EP2 pid=140529) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '2', 'device_ip': '192.23.2.197', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '2', 'device_ip': '192.45.2.197', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP2_EP2 pid=140529) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.2.197_192.45.2.197
(Worker_DP0_TP2_EP2 pid=140529) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {3: 0, 19: 1}
(Worker_DP0_TP6_EP6 pid=140541) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.2.193_192.45.2.193
(Worker_DP0_TP6_EP6 pid=140541) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP6_EP6 pid=140541) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '6', 'device_ip': '192.23.2.193', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '6', 'device_ip': '192.45.2.193', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP6_EP6 pid=140541) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.2.193_192.45.2.193
(Worker_DP0_TP6_EP6 pid=140541) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {7: 0, 23: 1}
(Worker_DP0_TP4_EP4 pid=140539) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:743] recving metadata: LLMDataDistCMgrAgentMetadata(super_pod_id='11', server_id='172.22.0.188', device_id='4', device_ip='192.23.2.195', super_device_id='98369551', cluster_id=5)
(Worker_DP0_TP5_EP5 pid=140540) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:743] recving metadata: LLMDataDistCMgrAgentMetadata(super_pod_id='11', server_id='172.22.0.188', device_id='5', device_ip='192.23.3.194', super_device_id='98369551', cluster_id=6)
(Worker_DP0_TP4_EP4 pid=140539) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.2.195_192.45.2.195
(Worker_DP0_TP7_EP7 pid=140542) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:743] recving metadata: LLMDataDistCMgrAgentMetadata(super_pod_id='11', server_id='172.22.0.188', device_id='7', device_ip='192.23.3.192', super_device_id='98369551', cluster_id=8)
(Worker_DP0_TP4_EP4 pid=140539) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP4_EP4 pid=140539) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '4', 'device_ip': '192.23.2.195', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '4', 'device_ip': '192.45.2.195', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP5_EP5 pid=140540) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.3.194_192.45.3.194
(Worker_DP0_TP4_EP4 pid=140539) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.2.195_192.45.2.195
(Worker_DP0_TP4_EP4 pid=140539) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {5: 0, 21: 1}
(Worker_DP0_TP7_EP7 pid=140542) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.3.192_192.45.3.192
(Worker_DP0_TP5_EP5 pid=140540) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP5_EP5 pid=140540) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '5', 'device_ip': '192.23.3.194', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '5', 'device_ip': '192.45.3.194', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP5_EP5 pid=140540) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.3.194_192.45.3.194
(Worker_DP0_TP7_EP7 pid=140542) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP7_EP7 pid=140542) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '7', 'device_ip': '192.23.3.192', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '7', 'device_ip': '192.45.3.192', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP5_EP5 pid=140540) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {6: 0, 22: 1}
(Worker_DP0_TP7_EP7 pid=140542) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.3.192_192.45.3.192
(Worker_DP0_TP0_EP0 pid=140527) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:743] recving metadata: LLMDataDistCMgrAgentMetadata(super_pod_id='11', server_id='172.22.0.188', device_id='0', device_ip='192.23.2.199', super_device_id='98369551', cluster_id=1)
(Worker_DP0_TP7_EP7 pid=140542) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {8: 0, 24: 1}
(Worker_DP0_TP0_EP0 pid=140527) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.2.199_192.45.2.199
(Worker_DP0_TP0_EP0 pid=140527) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP0_EP0 pid=140527) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '0', 'device_ip': '192.23.2.199', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '0', 'device_ip': '192.45.2.199', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP0_EP0 pid=140527) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.2.199_192.45.2.199
(Worker_DP0_TP0_EP0 pid=140527) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {1: 0, 17: 1}
(Worker_DP0_TP1_EP1 pid=140528) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:743] recving metadata: LLMDataDistCMgrAgentMetadata(super_pod_id='11', server_id='172.22.0.188', device_id='1', device_ip='192.23.3.198', super_device_id='98369551', cluster_id=2)
(Worker_DP0_TP1_EP1 pid=140528) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:685] LLMDataDistCMgrConnectorWorker: try link with remote, comm id: pd_comm_192.23.3.198_192.45.3.198
(Worker_DP0_TP1_EP1 pid=140528) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] rank table
(Worker_DP0_TP3_EP3 pid=140538) ERROR 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:581] KV transfer task failed: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {4: 0, 20: 1}.
(Worker_DP0_TP1_EP1 pid=140528) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:688] {'version': '1.2', 'server_count': '2', 'status': 'completed', 'server_list': [{'device': [{'device_id': '1', 'device_ip': '192.23.3.198', 'super_device_id': '98369551', 'rank_id': '0'}], 'server_id': '172.22.0.188'}, {'device': [{'device_id': '1', 'device_ip': '192.45.3.198', 'super_device_id': '190644239', 'rank_id': '1'}], 'server_id': '172.22.0.212'}], 'super_pod_list': [{'super_pod_id': '11', 'server_list': [{'server_id': '172.22.0.188'}, {'server_id': '172.22.0.212'}]}]}
(Worker_DP0_TP1_EP1 pid=140528) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:689] comm name: pd_comm_192.23.3.198_192.45.3.198
(Worker_DP0_TP1_EP1 pid=140528) INFO 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:690] cluster rank info: {2: 0, 18: 1}
(Worker_DP0_TP2_EP2 pid=140529) ERROR 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:581] KV transfer task failed: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {3: 0, 19: 1}.
(Worker_DP0_TP4_EP4 pid=140539) ERROR 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:581] KV transfer task failed: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {5: 0, 21: 1}.
(Worker_DP0_TP6_EP6 pid=140541) ERROR 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:581] KV transfer task failed: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {7: 0, 23: 1}.
(Worker_DP0_TP7_EP7 pid=140542) ERROR 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:581] KV transfer task failed: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {8: 0, 24: 1}.
(Worker_DP0_TP5_EP5 pid=140540) ERROR 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:581] KV transfer task failed: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {6: 0, 22: 1}.
(Worker_DP0_TP0_EP0 pid=140527) ERROR 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:581] KV transfer task failed: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {1: 0, 17: 1}.
(Worker_DP0_TP1_EP1 pid=140528) ERROR 09-12 03:40:02 [llmdatadist_c_mgr_connector.py:581] KV transfer task failed: [link] failed, error code is LLMStatusCode.LLM_LINK_FAILED, {2: 0, 18: 1}.

[zhangxinyuehfad]

✅ Async scheduler test ok

command:

vllm serve Qwen/Qwen3-8B --async_scheduling  &

output:

root@liteserver-for-vllm-ascend-00004:/vllm-workspace/vllm-ascend# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-8e57d3690c134b2896106e581c84e39a","object":"chat.completion","created":1757665021,"model":"Qwen/Qwen3-8B","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\nOkay, the user asked about the future of AI. I need to cover different aspects like technological advancements, societal impact, ethical considerations, and maybe some challenges. Let me start by thinking about the current state of AI. It's already making big strides in areas like machine learning, natural language processing, and robotics. So, the future might involve even more integration into daily life.\n\nWait, the user might be interested in both the positive and negative aspects. I should mention potential benefits like solving complex problems, improving healthcare, and enhancing productivity. But also address the challenges, like job displacement, privacy issues, and ethical dilemmas. Oh, and maybe touch on regulation and international cooperation since AI development is a global concern.\n\nI should also consider the different sectors where AI could have a significant impact. For example, in healthcare, AI could lead to personalized treatments. In transportation, autonomous vehicles might change how we move. Education could become more personalized with AI tutors. But there's also the risk of bias in algorithms, security threats, and the need for transparency.\n\nDon't forget to mention the ethical considerations. Issues like AI decision-making fairness, accountability, and the potential for misuse. Also, the importance of human oversight to prevent AI from making harmful decisions. Maybe include some examples, like AI in criminal justice systems or hiring processes, where bias could be a problem.\n\nI need to structure this in a way that's easy to follow. Maybe start with technological advancements, then move to societal impact, followed by ethical and regulatory challenges. Conclude with a balanced view that acknowledges both the opportunities and the risks. Also, highlight the need for responsible development and collaboration between different stakeholders.\n\nWait, the user might also be interested in the timeline. How soon could these developments happen? Should I mention specific areas where AI is already being applied and where it's still in the research phase? For example, quantum computing and AI synergy is a future possibility. Or AI in climate change solutions, like optimizing energy use or predicting environmental changes.\n\nI should also think about the role of governments and organizations in setting guidelines. Maybe touch on the EU's AI Act or other regulatory frameworks. The importance of public awareness and education to ensure that society can adapt to AI changes.\n\nDon't want to make it too technical. Keep the language accessible. Maybe use examples that are relatable, like how AI is already used in recommendation systems or virtual assistants, and where they might go next. Also, address the potential for AI to augment human capabilities rather than replace them, which is a common concern.\n\nCheck if there are any recent trends or studies that should be included. For instance, the rise of generative AI models like GPT-4 or DALL-E, and their implications for creativity and content creation. Also, the potential for AI in scientific research, such as drug discovery or materials science.\n\nMake sure to balance the discussion by not being overly optimistic or pessimistic. Acknowledge that the future of AI depends on how it's developed and governed. Emphasize the importance of interdisciplinary collaboration—engineers, ethicists, policymakers, and the public working together.\n\nAlright, I think that covers the main points. Now, structure them into coherent sections with clear headings. Use examples to illustrate points, and ensure the conclusion ties everything together, reinforcing the need for responsible innovation.\n</think>\n\nThe future of AI is a topic of immense interest and debate, with potential to reshape nearly every aspect of human life. Here's a structured overview of key trends, opportunities, challenges, and considerations:\n\n---\n\n### **1. Technological Advancements**\n- **AI Integration Across Sectors**: AI will become more deeply embedded in industries like healthcare (personalized treatments, diagnostics), education (adaptive learning tools), and transportation (autonomous vehicles). For example, AI-driven drug discovery could accelerate medical breakthroughs.\n- **Specialized AI Models**: Advances in natural language processing (e.g., large language models like GPT-4) and generative AI (e.g., DALL-E, MidJourney) will enhance creativity and productivity, enabling tasks like content creation, design, and scientific research.\n- **Quantum Computing Synergy**: Combining AI with quantum computing could solve complex problems in fields like cryptography, material science, and climate modeling, though this remains in early research stages.\n\n---\n\n### **2. Societal Impact**\n- **Economic Transformation**: AI may automate routine tasks, displacing jobs in sectors like manufacturing and customer service, but could also create new roles in AI development, maintenance, and oversight. Upskilling and reskilling will be critical.\n- **Enhanced Efficiency**: AI could optimize resource management (e.g., energy grids, agriculture) and improve public services (e.g., smart cities, disaster response).\n- **Global Inequality**: Access to AI technologies may widen the gap between developed and developing nations, necessitating global collaboration to ensure equitable benefits.\n\n---\n\n### **3. Ethical and Social Challenges**\n- **Bias and Fairness**: AI systems may inherit biases from training data, leading to discriminatory outcomes in areas like hiring, law enforcement, and lending. Ensuring transparency and fairness in algorithms is essential.\n- **Privacy Concerns**: Widespread data collection for AI training raises risks of surveillance and data misuse. Robust privacy frameworks (e.g., GDPR) and encryption will be vital.\n- **Accountability**: Determining responsibility for AI-driven decisions (e.g., autonomous vehicles causing accidents) requires clear legal and ethical guidelines.\n\n---\n\n### **4. Regulatory and Governance Frameworks**\n- **Global Collaboration**: Nations and organizations (e.g., EU AI Act, OECD guidelines) are developing regulations to address risks like misuse of AI in warfare or deepfakes. Harmonizing standards will be key.\n- **Human Oversight**: Ensuring AI systems operate within ethical boundaries and are subject to human review, especially in high-stakes domains like healthcare and criminal justice.\n- **Public Engagement**: Involving diverse stakeholders (including marginalized communities) in shaping AI policies to reflect societal values and needs.\n\n---\n\n### **5. Risks and Mitigation**\n- **Security Threats**: AI could be weaponized for cyberattacks or deepfake propaganda. Strengthening cybersecurity and developing AI detection tools are priorities.\n- **Existential Risks**: While speculative, concerns about superintelligent AI surpassing human control highlight the need for long-term research into safe AI development (e.g., the \"control problem\").\n- **Environmental Impact**: Training large AI models consumes significant energy. Innovations in energy-efficient algorithms and renewable energy use will be critical.\n\n---\n\n### **6. The Human-AI Relationship**\n- **Augmentation vs. Replacement**: AI is more likely to augment human capabilities (e.g., doctors using AI for diagnostics) than replace humans entirely. Collaboration between humans and AI will define future productivity.\n- **Ethical AI Design**: Prioritizing values like transparency, accountability, and inclusivity in AI development to ensure systems align with human interests.\n\n---\n\n### **Conclusion: A Balanced Future**\nThe future of AI hinges on how we choose to develop and govern it. While the technology holds transformative potential, its success depends on addressing ethical, social, and technical challenges through interdisciplinary collaboration, robust regulation, and public engagement. The goal should be to harness AI as a tool for empowerment, equity, and progress, ensuring it serves humanity's best interests. As the field evolves, continuous dialogue and adaptability will be essential to navigate its complexities.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":13,"total_tokens":1526,"completion_tokens":1513,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}r

[zhangxinyuehfad]

✅ Qwen3-8B flash comm + eager/aclgraph test ok

command:

VLLM_ASCEND_ENABLE_DENSE_OPTIMIZE=1 VLLM_ASCEND_ENABLE_FLASHCOMM=1 vllm serve Qwen/Qwen3-8B  &
VLLM_ASCEND_ENABLE_DENSE_OPTIMIZE=1 VLLM_ASCEND_ENABLE_FLASHCOMM=1 vllm serve Qwen/Qwen3-8B  --enforce_eager &

aclgraph output:

root@liteserver-for-vllm-ascend-00004:/vllm-workspace/vllm-ascend# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-836c3bd3cbbc47ca9f55dbef17770e0a","object":"chat.completion","created":1757667301,"model":"Qwen/Qwen3-8B","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\nOkay, the user asked about the future of AI. I need to cover different aspects like technological advancements, societal impact, ethical considerations, and maybe some challenges. Let me start by thinking about the current state of AI. It's already making big strides in areas like machine learning, natural language processing, and robotics. So, the future might involve even more integration into daily life.\n\nWait, the user might be interested in both the positive and negative aspects. I should mention potential benefits like solving complex problems, improving healthcare, and enhancing productivity. But also address the challenges, like job displacement, privacy issues, and ethical dilemmas. Oh, and maybe touch on regulation and international cooperation since AI development is a global concern.\n\nI should also consider the different sectors where AI could have a significant impact. For example, in healthcare, AI could lead to personalized treatments. In transportation, autonomous vehicles might change how we move. Education could become more personalized with AI tutors. But there's also the risk of bias in algorithms, security threats, and the need for transparency.\n\nDon't forget to mention the ethical considerations. Issues like AI decision-making fairness, accountability, and the potential for misuse. Also, the importance of human oversight to prevent AI from making harmful decisions. Maybe include some examples, like AI in criminal justice systems or hiring processes, where bias could be a problem.\n\nI need to structure this in a way that's easy to follow. Maybe start with technological advancements, then move to societal impact, followed by ethical and regulatory challenges. Conclude with a balanced view that acknowledges both the opportunities and the risks. Also, highlight the need for responsible development and collaboration between different stakeholders.\n\nWait, the user might also be interested in the timeline. How soon could these developments happen? Should I mention specific areas where AI is already being applied and where it's still in the research phase? For example, quantum computing and AI synergy is a future possibility. Or AI in climate change solutions, like optimizing energy use or predicting environmental changes.\n\nI should also think about the role of governments and organizations in setting guidelines. Maybe touch on the EU's AI Act or other regulatory frameworks. The importance of public awareness and education to ensure that society can adapt to AI changes.\n\nDon't want to make it too technical. Keep the language accessible. Maybe use examples that are relatable, like how AI is already used in recommendation systems or virtual assistants, and where they might go next. Also, address the potential for AI to augment human capabilities rather than replace them, which is a common concern.\n\nCheck if there are any recent trends or studies that should be included. For instance, the rise of generative AI models like GPT-4 or DALL-E, and their implications for creativity and content creation. Also, the potential for AI in scientific research, such as drug discovery or materials science.\n\nMake sure to balance the discussion by not being overly optimistic or pessimistic. Acknowledge that the future of AI depends on how it's developed and governed. Emphasize the importance of interdisciplinary collaboration—engineers, ethicists, policymakers, and the public working together.\n\nAlright, I think that covers the main points. Now, structure them into coherent sections with clear headings. Use examples to illustrate points, and ensure the conclusion ties everything together, reinforcing the need for responsible innovation.\n</think>\n\nThe future of AI is a topic of immense interest and debate, with potential to reshape nearly every aspect of human life. Here's a structured overview of key trends, opportunities, challenges, and considerations:\n\n---\n\n### **1. Technological Advancements**\n- **AI Integration Across Sectors**: AI will become more deeply embedded in industries like healthcare (personalized treatments, diagnostics), education (adaptive learning tools), and transportation (autonomous vehicles). For example, AI-driven drug discovery could accelerate medical breakthroughs.\n- **Specialized AI Models**: Advances in natural language processing (e.g., large language models like GPT-4) and generative AI (e.g., DALL-E, MidJourney) will enhance creativity and productivity, enabling tasks like content creation, design, and scientific research.\n- **Quantum Computing Synergy**: Combining AI with quantum computing could solve complex problems in fields like cryptography, material science, and climate modeling, though this remains in early research stages.\n\n---\n\n### **2. Societal Impact**\n- **Economic Transformation**: AI may automate routine tasks, displacing jobs in sectors like manufacturing and customer service, but could also create new roles in AI development, maintenance, and oversight. Upskilling and reskilling will be critical.\n- **Enhanced Efficiency**: AI could optimize resource management (e.g., energy grids, agriculture) and improve public services (e.g., smart cities, disaster response).\n- **Global Inequality**: Access to AI technologies may widen the gap between developed and developing nations, necessitating global collaboration to ensure equitable benefits.\n\n---\n\n### **3. Ethical and Social Challenges**\n- **Bias and Fairness**: AI systems may inherit biases from training data, leading to discriminatory outcomes in areas like hiring, law enforcement, and lending. Ensuring transparency and fairness in algorithms is essential.\n- **Privacy Concerns**: Widespread data collection for AI training raises risks of surveillance and data misuse. Robust privacy frameworks (e.g., GDPR) and encryption will be vital.\n- **Accountability**: Determining responsibility for AI-driven decisions (e.g., autonomous vehicles causing accidents) requires clear legal and ethical guidelines.\n\n---\n\n### **4. Regulatory and Governance Frameworks**\n- **Global Collaboration**: Nations and organizations (e.g., EU AI Act, OECD guidelines) are developing regulations to address risks like misuse of AI in warfare or deepfakes. Harmonizing standards will be key.\n- **Human Oversight**: Ensuring AI systems operate within ethical boundaries and are subject to human review, especially in high-stakes domains like healthcare and criminal justice.\n- **Public Engagement**: Involving diverse stakeholders (including marginalized communities) in shaping AI policies to reflect societal values and needs.\n\n---\n\n### **5. Risks and Mitigation**\n- **Security Threats**: AI could be weaponized for cyberattacks or deepfake propaganda. Strengthening cybersecurity and developing AI detection tools are priorities.\n- **Existential Risks**: While speculative, concerns about superintelligent AI surpassing human control highlight the need for long-term research into safe AI development (e.g., the \"control problem\").\n- **Environmental Impact**: Training large AI models consumes significant energy. Innovations in energy-efficient algorithms and renewable energy use will be critical.\n\n---\n\n### **6. The Human-AI Relationship**\n- **Augmentation vs. Replacement**: AI is more likely to augment human capabilities (e.g., doctors using AI for diagnostics) than replace humans entirely. Collaboration between humans and AI will define future productivity.\n- **Ethical AI Design**: Prioritizing values like transparency, accountability, and inclusivity in AI development to ensure systems align with human interests.\n\n---\n\n### **Conclusion: A Balanced Future**\nThe future of AI hinges on how we choose to develop and govern it. While the technology holds transformative potential, its success depends on addressing ethical, social, and technical challenges through interdisciplinary collaboration, robust regulation, and public engagement. The goal should be to harness AI as a tool for empowerment, equity, and progress, ensuring it serves humanity's best interests. As the field evolves, continuous dialogue and adaptability will be essential to navigate its complexities.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":13,"total_tokens":1526,"completion_tokens":1513,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

earger output:

root@liteserver-for-vllm-ascend-00004:/vllm-workspace/vllm-ascend# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-be7ee8a8874e4a298386fba3989fc9c3","object":"chat.completion","created":1757667641,"model":"Qwen/Qwen3-8B","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\nOkay, the user asked about the future of AI. I need to cover different aspects like technological advancements, societal impact, ethical considerations, and maybe some challenges. Let me start by thinking about the current state of AI. It's already making big strides in areas like machine learning, natural language processing, and robotics. So, the future might involve even more integration into daily life.\n\nWait, the user might be interested in both the positive and negative aspects. I should mention potential benefits like solving complex problems, improving healthcare, and enhancing productivity. But also address the challenges, like job displacement, privacy issues, and ethical dilemmas. Oh, and maybe touch on regulation and international cooperation since AI development is a global concern.\n\nI should also consider the different sectors where AI could have a significant impact. For example, in healthcare, AI could lead to personalized treatments. In transportation, autonomous vehicles might change how we move. Education could become more personalized with AI tutors. But there's also the risk of bias in algorithms, security vulnerabilities, and the need for transparency.\n\nDon't forget to mention the ethical considerations. Issues like AI decision-making fairness, accountability, and the potential for misuse. Also, the role of AI in creating new jobs versus replacing existing ones. Maybe the concept of AI being a tool that enhances human capabilities rather than replacing them.\n\nI need to structure this in a coherent way. Start with technological advancements, then societal impact, ethical issues, challenges, and maybe a conclusion about the balance between innovation and responsibility. Make sure to highlight both opportunities and risks. Also, include examples like AI in climate change solutions or personalized medicine to illustrate points.\n\nWait, the user might be looking for a comprehensive overview, so I should cover key areas without getting too technical. Avoid jargon, keep it accessible. Also, consider the future trends like AI and quantum computing integration, or AI in space exploration. Maybe mention the potential for AI to assist in scientific discovery.\n\nI should also address the importance of interdisciplinary collaboration—how computer science, ethics, law, and social sciences need to work together to guide AI's development. And the need for global standards and policies to ensure responsible use. Maybe touch on the debate around AI's autonomy and the risks of superintelligent systems, though that's more speculative.\n\nMake sure to balance optimism with caution. Emphasize that the future of AI depends on how we develop and govern it. Conclude by highlighting the transformative potential but the necessity of ethical frameworks and public engagement. Alright, that should cover the main points.\n</think>\n\nThe future of AI is a topic of immense interest and debate, shaped by rapid technological advancements, societal needs, and ethical considerations. Here's a structured overview of key aspects that define its trajectory:\n\n---\n\n### **1. Technological Advancements**\n- **General AI (AGI):** While current AI systems are narrow (specialized for specific tasks), the pursuit of AGI—machines capable of human-like reasoning across domains—is a long-term goal. However, this remains speculative and fraught with technical and ethical challenges.\n- **Integration with Emerging Technologies:** AI will increasingly merge with quantum computing, biotechnology, and the Internet of Things (IoT), enabling breakthroughs in fields like drug discovery, climate modeling, and autonomous systems.\n- **AI in Everyday Life:** Enhanced personal assistants, smart homes, and wearable devices will become more intuitive, while AI-driven tools in education, healthcare, and creative industries will redefine workflows.\n\n---\n\n### **2. Societal Impact**\n- **Economic Transformation:** AI could boost productivity and innovation but may also disrupt labor markets. Jobs in repetitive tasks may decline, while demand for skills in AI development, data analysis, and human-AI collaboration will rise.\n- **Healthcare Revolution:** Personalized medicine, predictive diagnostics, and robotic surgery could improve outcomes, though equitable access to AI-driven healthcare remains a challenge.\n- **Education and Accessibility:** AI tutors and adaptive learning platforms may democratize education, but biases in training data could perpetuate inequities if not addressed.\n\n---\n\n### **3. Ethical and Philosophical Challenges**\n- **Bias and Fairness:** AI systems may inherit biases from training data, leading to discriminatory outcomes in hiring, law enforcement, or lending. Ensuring transparency and accountability is critical.\n- **Privacy and Security:** AI's reliance on vast data sets raises concerns about surveillance, data misuse, and vulnerabilities to hacking. Robust encryption and regulatory frameworks will be essential.\n- **Autonomy and Control:** Questions arise about AI's role in decision-making (e.g., in criminal justice or warfare) and the potential for misuse in autonomous weapons or deepfakes.\n\n---\n\n### **4. Global Collaboration and Governance**\n- **Regulation and Standards:** Governments and international bodies are developing frameworks (e.g., the EU’s AI Act) to ensure AI aligns with human rights, safety, and ethical principles. Global cooperation will be vital to address cross-border issues like data privacy and AI arms races.\n- **Public Engagement:** Inclusive dialogue between technologists, policymakers, and citizens will shape AI’s direction, ensuring it serves societal needs rather than corporate or geopolitical interests.\n\n---\n\n### **5. Risks and Mitigation**\n- **Job Displacement:** While AI may create new roles, proactive reskilling and universal basic income (UBI) proposals are being explored to cushion economic transitions.\n- **Existential Risks:** Theoretical concerns about superintelligent AI surpassing human control remain a focus of research, though most experts agree such scenarios are distant and manageable with proper safeguards.\n- **Environmental Impact:** AI’s energy consumption is a growing concern. Efforts to develop energy-efficient algorithms and renewable-powered data centers are critical for sustainability.\n\n---\n\n### **6. Opportunities for Positive Change**\n- **Climate Solutions:** AI can optimize energy grids, monitor deforestation, and accelerate the development of clean technologies.\n- **Scientific Discovery:** AI may unlock new insights in physics, biology, and materials science, accelerating innovation.\n- **Human Augmentation:** AI tools could enhance human capabilities in creativity, problem-solving, and decision-making, rather than replacing humans.\n\n---\n\n### **Conclusion**\nThe future of AI is not predetermined—it hinges on how societies choose to develop, regulate, and ethically deploy the technology. While AI holds transformative potential, its success will depend on balancing innovation with responsibility, ensuring it empowers humanity rather than exacerbates divisions. The path forward requires collaboration across disciplines, cultures, and generations to harness AI’s benefits while mitigating its risks.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":13,"total_tokens":1335,"completion_tokens":1322,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

[zhangxinyuehfad]

✅ W8A8 + aclgraph test ok

command:

vllm serve vllm-ascend/Qwen3-8B-W8A8 \
--quantization ascend \
--max-model-len 4096 \
--trust-remote-code &

output:

root@liteserver-for-vllm-ascend-00004:/vllm-workspace/vllm-ascend# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-39e9baf63d114465b43c9686f19c24ee","object":"chat.completion","created":1757668249,"model":"vllm-ascend/Qwen3-8B-W8A8","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\nOkay, the user asked about the future of AI. I need to cover different aspects. Let me start by thinking about current trends. AI is already in many areas, like healthcare, finance, and transportation. Maybe mention specific examples, like medical diagnostics or self-driving cars.\n\nThen, I should consider the challenges. There's the ethical issues, like bias in AI systems. Also, job displacement is a big concern. People might worry about automation replacing human jobs. Need to address that but also mention potential solutions, like reskilling.\n\nAnother point is the integration of AI with other technologies. Like AI and IoT working together. Maybe talk about smart cities or personalized learning systems. Also, the role of AI in sustainability, such as optimizing energy use or environmental monitoring.\n\nI should mention the ethical and regulatory aspects. Governments are starting to create frameworks for AI, so maybe discuss things like transparency and accountability. Also, the risk of AI being misused, like deepfakes or autonomous weapons.\n\nThen, the future possibilities. AI could lead to breakthroughs in science, like drug discovery or space exploration. But there's also the risk of AI surpassing human intelligence, so need to talk about safety and control mechanisms.\n\nDon't forget the societal impact. How AI changes how we interact with technology, maybe even with each other. The importance of education and public understanding to ensure responsible development.\n\nWait, I should structure this. Maybe start with current state, then challenges, then future possibilities, and wrap up with a balanced view. Make sure to highlight both the opportunities and the risks. Also, check if there's anything I'm missing, like the role of AI in creative industries or personalized services. Yeah, that's a good point. Maybe include how AI can assist in creative fields, like art or music, but also the need for human oversight.\n\nAlso, mention the global competition in AI development. Countries investing heavily, which could lead to both innovation and potential conflicts. Need to balance that with collaboration efforts.\n\nOkay, that's a good outline. Now, structure the answer with these points, making sure it's clear and covers all aspects without being too technical. Keep it engaging and informative.\n</think>\n\nThe future of AI is a dynamic and multifaceted landscape, shaped by rapid technological advancements, ethical considerations, and societal transformations. Here's a structured overview of its potential trajectory:\n\n### **Current State & Emerging Trends**  \nAI is already deeply integrated into daily life, from healthcare diagnostics (e.g., AI-driven imaging tools) to autonomous vehicles and personalized finance. Key trends include:  \n- **Generalization**: AI systems are becoming more versatile, handling complex tasks across domains.  \n- **Human-AI Collaboration**: Tools like AI-powered coding assistants or creative tools (e.g., DALL-E, Midjourney) are augmenting human creativity.  \n- **Ethical AI**: Efforts to embed fairness, transparency, and accountability into AI systems are gaining traction.  \n\n### **Challenges & Risks**  \n1. **Ethical and Social Concerns**:  \n   - **Bias and Fairness**: AI systems can perpetuate historical biases if trained on skewed data.  \n   - **Job Displacement**: Automation may disrupt labor markets, though it could also create new roles in AI development and oversight.  \n   - **Privacy Risks**: Data collection and surveillance raise concerns about individual autonomy.  \n\n2. **Technical Limitations**:  \n   - **Explainability**: \"Black box\" AI models lack transparency, complicating trust and accountability.  \n   - **Security Vulnerabilities**: AI systems are susceptible to adversarial attacks and manipulation.  \n\n3. **Global Power Dynamics**:  \n   - The AI arms race between nations could lead to geopolitical tensions, with implications for military and civilian applications.  \n\n### **Future Possibilities**  \n1. **Revolutionary Applications**:  \n   - **Healthcare**: AI could accelerate drug discovery, personalize treatments, and enable real-time disease monitoring.  \n   - **Climate Action**: AI might optimize energy grids, predict climate patterns, and enhance environmental conservation.  \n   - **Space Exploration**: Autonomous systems could facilitate interplanetary missions and resource management.  \n\n2. **Societal Transformation**:  \n   - **Education**: AI-driven personalized learning could democratize access to education.  \n   - **Cultural Shifts**: AI might redefine art, music, and storytelling, blending human and machine creativity.  \n   - **Governance**: AI could aid in policy-making, disaster response, and resource allocation.  \n\n3. **Existential Risks**:  \n   - **Superintelligent AI**: If AI surpasses human intelligence, its alignment with human values becomes critical.  \n   - **Autonomous Weapons**: Unregulated AI could pose security threats.  \n\n### **Pathways to a Responsible Future**  \n1. **Regulation and Governance**:  \n   - Global frameworks (e.g., EU AI Act) aim to balance innovation with ethical standards.  \n   - Public-private partnerships could foster responsible AI development.  \n\n2. **Education and Inclusivity**:  \n   - Upskilling workers to adapt to AI-driven economies is essential.  \n   - Promoting digital literacy ensures equitable access to AI benefits.  \n\n3. **Collaborative Innovation**:  \n   - Open-source AI research and cross-border collaboration can mitigate risks and accelerate progress.  \n\n### **Conclusion**  \nThe future of AI is not predetermined—it hinges on how humanity chooses to shape it. While the potential for transformative progress is immense, the risks demand vigilance. By prioritizing ethical frameworks, equitable access, and global cooperation, AI can become a tool for collective empowerment, driving a more sustainable, inclusive, and innovative world. The challenge lies in balancing ambition with responsibility, ensuring AI serves humanity’s enduring interests.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":13,"total_tokens":1189,"completion_tokens":1176,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}root@liteserver-for-vllm-ascend-00004:/vllm-workspace/vllm-ascend#

[22dimensions]

Qwen2.5-VL-3B-Instruct-W8A8

How to quantize？
original model：https://modelscope.cn/models/Qwen/Qwen2.5-VL-3B-Instruct

modelslim version：master

steps:

• modify https://gitcode.com/Ascend/msit/blob/master/msmodelslim/example/Qwen2.5-VL/quant_qwen2_5vl.py last line code

# 7.保存权重
calibrator.save(args.save_directory, save_type=["ascendV1"], part_file_size=args.part_file_size)

• execute quantization command: https://gitcode.com/Ascend/msit/tree/master/msmodelslim/example/Qwen2.5-VL#11-qwen25-vl-w8a8量化-异常值抑制算法使用m2
• copy all files from the original model, except the model weights, into the quantized model’s directory

quantized model file: https://www.modelscope.cn/models/vllm-ascend/Qwen2.5-VL-3B-Instruct-W8A8

test script:

from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from qwen_vl_utils import process_vision_info

MODEL_PATH = "/home/cache/quant/Qwen2.5-VL-3B-Instruct-W8A8"

llm = LLM(
    model=MODEL_PATH,
    max_model_len=16384,
    quantization="ascend",
    limit_mm_per_prompt={"image": 10},
)

sampling_params = SamplingParams(
    max_tokens=512
)

image_messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/qwen.png",
                "min_pixels": 224 * 224,
                "max_pixels": 1280 * 28 * 28,
            },
            {"type": "text", "text": "Please provide a detailed description of this image"},
        ],
    },
]

messages = image_messages

processor = AutoProcessor.from_pretrained(MODEL_PATH)
prompt = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

image_inputs, _, _ = process_vision_info(messages, return_video_kwargs=True)

mm_data = {}
if image_inputs is not None:
    mm_data["image"] = image_inputs

llm_inputs = {
    "prompt": prompt,
    "multi_modal_data": mm_data,
}

outputs = llm.generate([llm_inputs], sampling_params=sampling_params)
generated_text = outputs[0].outputs[0].text

print(generated_text)

result:

(EngineCore_DP0 pid=23659) INFO 09-15 02:02:23 [acl_graph.py:184] Replaying aclgraph
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.71s/it, est. speed input: 28.84 toks/s, output: 75.21 toks/s]
The image displays a logo for a company named "TongYi Qwen." The logo consists of two main parts: the text and a graphic symbol. 

1. **Text Part**:
   - The word "TONGYI" is written in uppercase letters. The "Y" in "Tong" is stylized with a vertical line that extends downwards and then curves slightly to the right.
   - Below "TONGYI," the word "Qwen" is written in lowercase letters. The "Q" is stylized with a curved top and a bottom that extends downward and then curves slightly to the right. The rest of the word "Qwen" maintains a simple and clean design.

2. **Graphic Symbol**:
   - The symbol on the left side of the text is a stylized, rectangular shape with diagonal lines: one arm extends from the top left to the bottom right, and the other arm extends from the top right to the bottom left.

The overall color scheme of the logo is white text with a blue-to-purple gradient for the symbol. This gradient creates a sense of depth and adds a modern touch to the logo.

The combination of the text with the stylized graphic symbol helps to convey a sense of technology or futuristic feel, which is appropriate for a company that might be involved in areas such as artificial intelligence or digital solutions.

[zhangxinyuehfad]

✅ deepseek v3 + torchair + all2allv test ok

command ：

VLLM_ASCEND_ENABLE_MOE_ALL2ALL_SEQ=1 vllm serve vllm-ascend/DeepSeek-V3-W8A8 \
--tensor-parallel-size 16 \
--quantization ascend \
--max-model-len 4096 \
--trust-remote-code \
--additional-config '{"torchair_graph_config":{"enabled":true}}' 

output:

APIServer pid=12349) INFO 09-12 11:21:12 [config.py:235] Replacing legacy 'type' key with 'rope_type'
(APIServer pid=12349) INFO 09-12 11:21:12 [api_server.py:1971] Starting vLLM API server 0 on http://0.0.0.0:8000
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:36] Available routes are:
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /docs, Methods: HEAD, GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /redoc, Methods: HEAD, GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /health, Methods: GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /load, Methods: GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /ping, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /ping, Methods: GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /tokenize, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /detokenize, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/models, Methods: GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /version, Methods: GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/responses, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/chat/completions, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/completions, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/embeddings, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /pooling, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /classify, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /score, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/score, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/audio/translations, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /rerank, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v1/rerank, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /v2/rerank, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /invocations, Methods: POST
(APIServer pid=12349) INFO 09-12 11:21:12 [launcher.py:44] Route: /metrics, Methods: GET
(APIServer pid=12349) INFO:     Started server process [12349]
(APIServer pid=12349) INFO:     Waiting for application startup.
(APIServer pid=12349) INFO:     Application startup complete.
(Worker_TP15 pid=16316) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP2 pid=13115) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP13 pid=15825) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP9 pid=14845) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP0 pid=12813) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP7 pid=14355) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP4 pid=13616) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP6 pid=14112) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP11 pid=15335) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP14 pid=16070) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP5 pid=13864) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP8 pid=14600) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP12 pid=15580) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP10 pid=15090) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP3 pid=13371) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP1 pid=12947) WARNING 09-12 11:21:16 [cudagraph_dispatcher.py:102] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(APIServer pid=12349) INFO 09-12 11:21:22 [loggers.py:123] Engine 000: Avg prompt throughput: 0.2 tokens/s, Avg generation throughput: 4.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=12349) INFO:     127.0.0.1:57462 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=12349) INFO 09-12 11:21:32 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=12349) INFO 09-12 11:21:42 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

root@liteserver-for-vllm-ascend-00003:/workspace# curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "vllm-ascend/DeepSeek-V3-W8A8",
        "prompt": "what is AI",
        "max_tokens": 100,
        "temperature": 0
    }'
{"id":"cmpl-edcff76fa1a4423d8386d8e39285fc6e","object":"text_completion","created":1757676076,"model":"vllm-ascend/DeepSeek-V3-W8A8","choices":[{"index":0,"text":"? Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think and learn like humans. These machines can perform tasks that typically require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. AI systems are designed to analyze data, recognize patterns, and make decisions with minimal human intervention.\n\n### Key Concepts in AI:\n\n1. **Machine Learning (ML):** A subset of AI that involves the use of algorithms and statistical models to enable","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":4,"total_tokens":104,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}root@liteserver-for-vllm-ascend-00003:/workspace#