[Bug]: vllm serve for Qwen3-Reranker-8B, report warning and return no data.

[MaoJianwei]

Your current environment

vLLM API server version 0.10.1.dev73+g7728dd77b

🐛 Describe the bug

Running cmd:
VLLM_CPU_OMP_THREADS_BIND="36-47" VLLM_USE_MODELSCOPE=1 VLLM_USE_V1=1 VLLM_CPU_KVCACHE_SPACE=32 vllm serve ~/Qwen3-Reranker-8B/ --served-model-name Qwen3-Reranker-8B --port 8080 --trust-remote-code

INFO:     Started server process [5675]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     10.112.41.17:10649 - "GET /v1/models HTTP/1.1" 200 OK
WARNING 07-28 04:05:12 [logger.py:71] To indicate that the rerank API is not part of the standard OpenAI API, we have located it at `/rerank`. Please update your client accordingly. (Note: Conforms to JinaAI rerank API)
INFO:     10.112.41.17:10900 - "POST /v1/rerank HTTP/1.1" 200 OK
INFO:     10.112.41.17:10941 - "POST /v1/rerank HTTP/1.1" 200 OK
INFO:     10.112.41.17:10950 - "POST /v1/rerank HTTP/1.1" 200 OK

VLLM reports the warning, and returns no data of reranked result.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

try
vllm serve ~/Qwen3-Reranker-8B/ --served-model-name Qwen3-Reranker-8B --task score

or

vllm serve ~/Qwen3-Reranker-8B/ --served-model-name Qwen3-Reranker-8B --task classify

[MaoJianwei]

try vllm serve ~/Qwen3-Reranker-8B/ --served-model-name Qwen3-Reranker-8B --task score

or

vllm serve ~/Qwen3-Reranker-8B/ --served-model-name Qwen3-Reranker-8B --task classify

the warning is still here, but it can return reranked result now, thanks.

what's the difference of "score" and "classify"?

[MaoJianwei]

try vllm serve ~/Qwen3-Reranker-8B/ --served-model-name Qwen3-Reranker-8B --task score

or

vllm serve ~/Qwen3-Reranker-8B/ --served-model-name Qwen3-Reranker-8B --task classify

If using "classify", EngineCore fails to start:

INFO 07-28 05:21:20 [default_loader.py:262] Loading weights took 2.75 seconds
ERROR 07-28 05:21:20 [core.py:641] EngineCore failed to start.
ERROR 07-28 05:21:20 [core.py:641] Traceback (most recent call last):
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/v1/engine/core.py", line 632, in run_engine_core
ERROR 07-28 05:21:20 [core.py:641]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 07-28 05:21:20 [core.py:641]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/v1/engine/core.py", line 450, in __init__
ERROR 07-28 05:21:20 [core.py:641]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/v1/engine/core.py", line 79, in __init__
ERROR 07-28 05:21:20 [core.py:641]     self.model_executor = executor_class(vllm_config)
ERROR 07-28 05:21:20 [core.py:641]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/executor/executor_base.py", line 53, in __init__
ERROR 07-28 05:21:20 [core.py:641]     self._init_executor()
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/executor/uniproc_executor.py", line 49, in _init_executor
ERROR 07-28 05:21:20 [core.py:641]     self.collective_rpc("load_model")
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
ERROR 07-28 05:21:20 [core.py:641]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 07-28 05:21:20 [core.py:641]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/utils/__init__.py", line 2987, in run_method
ERROR 07-28 05:21:20 [core.py:641]     return func(*args, **kwargs)
ERROR 07-28 05:21:20 [core.py:641]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/v1/worker/gpu_worker.py", line 201, in load_model
ERROR 07-28 05:21:20 [core.py:641]     self.model_runner.load_model(eep_scale_up=eep_scale_up)
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/v1/worker/cpu_model_runner.py", line 55, in load_model
ERROR 07-28 05:21:20 [core.py:641]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 07-28 05:21:20 [core.py:641]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/model_executor/model_loader/__init__.py", line 118, in get_model
ERROR 07-28 05:21:20 [core.py:641]     return loader.load_model(vllm_config=vllm_config,
ERROR 07-28 05:21:20 [core.py:641]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/model_executor/model_loader/base_loader.py", line 49, in load_model
ERROR 07-28 05:21:20 [core.py:641]     self.load_weights(model, model_config)
ERROR 07-28 05:21:20 [core.py:641]   File "/root/miniconda3/envs/vllm2/lib/python3.12/site-packages/vllm-0.10.1.dev73+g7728dd77b.cpu-py3.12-linux-x86_64.egg/vllm/model_executor/model_loader/default_loader.py", line 271, in load_weights
ERROR 07-28 05:21:20 [core.py:641]     raise ValueError("Following weights were not initialized from "
ERROR 07-28 05:21:20 [core.py:641] ValueError: Following weights were not initialized from checkpoint: {'score.weight'}

[noooop]

The official original model needs to use the command below

vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

It may be needed or not needed --task score or --task classify, depending on your vllm version

[MaoJianwei]

The official original model needs to use the command below

vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

It may be needed or not needed --task score or --task classify, depending on your vllm version

I'm using the main branch, git pull today

[noooop]

try

vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

No need to add other parameters

[MaoJianwei]

try

vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

No need to add other parameters

thanks, not adding "score" or "classify" is the target solution for next release?

[noooop]

PTAL #21470

[MaoJianwei]

I see, thanks

[haitwang-cloud]

Just FYI, if anyone is leveraging the K8S to use vLLM ,remember to use this command to host the Qwen3-Reranker-8B

          args: [
              'vllm serve Qwen/Qwen3-Reranker-8B
              --hf_overrides ''{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}''',
            ]

[MaoJianwei]

Just FYI, if anyone is leveraging the K8S to use vLLM ,remember to use this command to host the Qwen3-Reranker-8B

          args: [
              'vllm serve Qwen/Qwen3-Reranker-8B
              --hf_overrides ''{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}''',
            ]

Thanks for your information :)