[Bug]: Qwen3.5 does not work with pipeline parallelism

[VirgilG72]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

vllm server bash

python3 -m vllm.entrypoints.openai.api_server --served-model-name Qwen3.5-35B-A3B --model /athena/Qwen3.5-35B-A3B --gpu-memory-utilization 0.9 --tensor-parallel-size 1 --pipeline-parallel-size 4 --moe-backend marlin --max-model-len 160000 --max-num-batched-tokens 4096 --max-num-seqs 32 --distributed-executor-backend ray --enable-log-requests --enable-log-outputs --enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prefix-caching --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

error output

root@xuanwu-text-safety-qwen3-5-1355630-z8p8q:/data# nohup python3 -m vllm.entrypoints.openai.api_server --served-model-name Qwen3.5-35B-A3B --model /athena/Qwen3.5-35B-A3B --gpu-memory-utilization 0.9 --tensor-parallel-size 1 --pipeline-parallel-size 4 --moe-backend marlin --max-model-len 160000 --max-num-batched-tokens 4096 --max-num-seqs 32 --distributed-executor-backend ray --enable-log-requests --enable-log-outputs --enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prefix-caching --reasoning-parser qwen3 --speculative-config  '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' > vllm.log 2>&1 &
[1] 443977
root@xuanwu-text-safety-qwen3-5-1355630-z8p8q:/data# tail -f vllm.log  
nohup: ignoring input
(APIServer pid=443977) INFO 03-10 19:20:41 [utils.py:302] 
(APIServer pid=443977) INFO 03-10 19:20:41 [utils.py:302]        █     █     █▄   ▄█
(APIServer pid=443977) INFO 03-10 19:20:41 [utils.py:302]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.17.0
(APIServer pid=443977) INFO 03-10 19:20:41 [utils.py:302]   █▄█▀ █     █     █     █  model   /athena/Qwen3.5-35B-A3B
(APIServer pid=443977) INFO 03-10 19:20:41 [utils.py:302]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=443977) INFO 03-10 19:20:41 [utils.py:302] 
(APIServer pid=443977) INFO 03-10 19:20:41 [utils.py:238] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'enable_log_outputs': True, 'model': '/athena/Qwen3.5-35B-A3B', 'max_model_len': 160000, 'served_model_name': ['Qwen3.5-35B-A3B'], 'reasoning_parser': 'qwen3', 'distributed_executor_backend': 'ray', 'pipeline_parallel_size': 4, 'enable_prefix_caching': True, 'max_num_batched_tokens': 4096, 'max_num_seqs': 32, 'moe_backend': 'marlin', 'speculative_config': {'method': 'qwen3_next_mtp', 'num_speculative_tokens': 2}, 'enable_log_requests': True}
(APIServer pid=443977) INFO 03-10 19:20:41 [model.py:531] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=443977) INFO 03-10 19:20:41 [model.py:1554] Using max model len 160000
(APIServer pid=443977) WARNING 03-10 19:20:41 [speculative.py:346] method `qwen3_next_mtp` is deprecated and replaced with mtp.
(APIServer pid=443977) INFO 03-10 19:20:41 [model.py:531] Resolved architecture: Qwen3_5MoeMTP
(APIServer pid=443977) INFO 03-10 19:20:41 [model.py:1554] Using max model len 262144
(APIServer pid=443977) WARNING 03-10 19:20:41 [speculative.py:487] Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer,which may result in lower acceptance rate
(APIServer pid=443977) Traceback (most recent call last):
(APIServer pid=443977)   File "<frozen runpy>", line 198, in _run_module_as_main
(APIServer pid=443977)   File "<frozen runpy>", line 88, in _run_code
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 545, in <module>
(APIServer pid=443977)     uvloop.run(run_server(args))
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=443977)     return __asyncio.run(
(APIServer pid=443977)            ^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=443977)     return runner.run(main)
(APIServer pid=443977)            ^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=443977)     return self._loop.run_until_complete(task)
(APIServer pid=443977)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=443977)     return await main
(APIServer pid=443977)            ^^^^^^^^^^
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 471, in run_server
(APIServer pid=443977)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 490, in run_server_worker
(APIServer pid=443977)     async with build_async_engine_client(
(APIServer pid=443977)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=443977)     return await anext(self.gen)
(APIServer pid=443977)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 96, in build_async_engine_client
(APIServer pid=443977)     async with build_async_engine_client_from_engine_args(
(APIServer pid=443977)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=443977)     return await anext(self.gen)
(APIServer pid=443977)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 122, in build_async_engine_client_from_engine_args
(APIServer pid=443977)     vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=443977)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1729, in create_engine_config
(APIServer pid=443977)     speculative_config = self.create_speculative_config(
(APIServer pid=443977)                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1443, in create_speculative_config
(APIServer pid=443977)     return SpeculativeConfig(**self.speculative_config)
(APIServer pid=443977)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__
(APIServer pid=443977)     s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/config/speculative.py", line 764, in _verify_args
(APIServer pid=443977)     self.draft_model_config.verify_with_parallel_config(
(APIServer pid=443977)   File "/usr/local/lib/python3.12/dist-packages/vllm/config/model.py", line 1065, in verify_with_parallel_config
(APIServer pid=443977)     raise NotImplementedError(
(APIServer pid=443977) NotImplementedError: Pipeline parallelism is not supported for this model. Supported models implement the `SupportsPP` interface.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ShanningZhuang]

The problem of the PP

Hi, I think the reason is not from the Qwen3.5 but from the speculative model qwen3_next_mtp.

python3 -m vllm.entrypoints.openai.api_server --served-model-name Qwen3.5-35B-A3B --model Qwen/Qwen3.5-35B-A3B --gpu-memory-utilization 0.9 --tensor-parallel-size 1 --pipeline-parallel-size 4 --moe-backend marlin --max-model-len 160000 --max-num-batched-tokens 4096 --max-num-seqs 32 --distributed-executor-backend ray --enable-log-requests --enable-log-outputs --enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prefix-caching --reasoning-parser qwen3

After remove the speculative settings it has no Pipeline support error but come with this

(EngineCore_DP0 pid=1421088) ValueError: moe_backend='marlin' is not supported for unquantized MoE. Expected one of ['triton', 'flashinfer_trtllm', 'flashinfer_cutlass', 'aiter'].

After I change the moe_backend to triton. However since you are using your custom model I think the model is quantized so it won't be a problem for you.

python3 -m vllm.entrypoints.openai.api_server --served-model-name Qwen3.5-35B-A3B --model Qwen/Qwen3.5-35B-A3B --gpu-memory-utilization 0.9 --tensor-parallel-size 1 --pipeline-parallel-size 4 --moe-backend triton --max-model-len 160000 --max-num-batched-tokens 4096 --max-num-seqs 32 --distributed-executor-backend ray --enable-log-requests --enable-log-outputs --enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prefix-caching --reasoning-parser qwen3

Dive into the code

The problem of PP

Qwen3.5-35B-A3B (Qwen3_5ForCausalLMBase) — Supports PP
It explicitly inherits SupportsPP:
qwen3_5.py
Lines 536-541
class Qwen3_5ForCausalLMBase(
    nn.Module,
    HasInnerState,
    SupportsLoRA,
    SupportsPP,
):
class Qwen3_5ForCausalLMBase(    nn.Module,    HasInnerState,    SupportsLoRA,    SupportsPP,):
And it has make_empty_intermediate_tensors (line 588) and forward accepts intermediate_tensors.
Qwen3NextMTP — Does NOT formally support PP
qwen3_next_mtp.py
Lines 223-223
class Qwen3NextMTP(nn.Module, QwenNextMixtureOfExperts):
class Qwen3NextMTP(nn.Module, QwenNextMixtureOfExperts):
It does not inherit SupportsPP and does not set supports_pp = True.