Supports the vLLM inference framework

[TracyBin]

Does Alluxio support caching and accelerating model weight files based on NFS to achieve faster vLLM model inference startup?

[YichuanSun]

We believe it’s feasible. May I ask whether your focus is on model distribution or inference? We have solutions for both. If convenient, please leave your email so we can discuss the details further.

[TracyBin]

thanks you yan_5386@163.com

[TracyBin]

We are currently facing an issue. Originally, we planned to use Alluxio to improve the model inference startup speed of the Xinference/vLLM framework. However, we discovered that the Xinference/vLLM framework uses mmap to map the weight files into memory and then copies them to the GPU memory. Through testing, we found that vLLM cannot read the weight files via FUSE/POSIX APIs, which causes the model inference startup to hang.

[TracyBin]

https://www.alluxio.com.cn/inferless-case-study/
Is the solution mentioned in this document supported in the open-source version?

[YichuanSun]

https://www.alluxio.com.cn/inferless-case-study/ Is the solution mentioned in this document supported in the open-source version?

Sorry, this is only supported in the enterprise version.