RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

[ailinuxok]

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Jan_15_19:20:09_PST_2025
Cuda compilation tools, release 12.8, V12.8.61
Build cuda_12.8.r12.8/compiler.35404655_0
NVIDIA GeForce RTX 4090
| NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 12.8

vllm -v
0.11.1rc5.dev34+g48eb8eba5.cu129
### Now I see this kind of error reported

(EngineCore_DP0 pid=51760) raise AttributeError(
(EngineCore_DP0 pid=51760) AttributeError: '_OpNamespace' '_moe_C' object has no attribute 'topk_softmax'
Traceback (most recent call last):
File "/data/api_ocr.py", line 6, in
llm = LLM(
^^^^
File "/data/deepseek/.venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 336, in init
self.llm_engine = LLMEngine.from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/deepseek/.venv/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 188, in from_engine_args
return cls(
^^^^
File "/data/deepseek/.venv/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 122, in init
self.engine_core = EngineCoreClient.make_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/deepseek/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 93, in make_client
return SyncMPClient(vllm_config, executor_class, log_stats)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/deepseek/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 639, in init
super().init(
File "/data/deepseek/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 468, in init
with launch_core_engines(vllm_config, executor_class, log_stats) as (
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pythonenv/python312/lib/python3.12/contextlib.py", line 144, in exit
next(self.gen)
File "/data/deepseek/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 889, in launch_core_engines
wait_for_engine_startup(
File "/data/deepseek/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 946, in wait_for_engine_startup
raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}