Important

This is an early developer preview of the vLLM Gaudi Plugin and is not yet intended for general use. For a more stable experience, consider using the HabanaAI/vllm-fork or the in-tree Gaudi implementation available in vllm-project/vllm.

Welcome to vLLM x Intel Gaudi

x

vLLM Gaudi plugin (vllm-gaudi) integrates Intel Gaudi accelerators with vLLM to optimize large language model inference.

This plugin follows the [RFC]: Hardware pluggable and [RFC]: Enhancing vLLM Plugin Architecture principles, providing a modular interface for Intel Gaudi hardware.

Learn more:

📚 Intel Gaudi Documentation
🚀 vLLM Plugin System Overview

Getting Started

1. Install vLLM with pip or from source:

   pip install vllm

   or

   # Build vLLM from source for empty platform, reusing existing torch installation
   git clone https://github.com/vllm-project/vllm
   cd vllm
   pip install -r <(sed '/^[torch]/d' requirements/build.txt)
   VLLM_TARGET_DEVICE=empty pip install --no-build-isolation -e .
   cd ..

2. Install vLLM-Gaudi from source:

   git clone https://github.com/vllm-project/vllm-gaudi
   cd vllm-gaudi
   pip install -e .

Full installation from source (vLLM and vLLM-Gaudi):

# Build vLLM from source for empty platform, reusing existing torch installation
git clone https://github.com/vllm-project/vllm
cd vllm
pip install -r <(sed '/^[torch]/d' requirements/build.txt)
VLLM_TARGET_DEVICE=empty pip install --no-build-isolation -e .
cd ..

# Build vLLM-Gaudi from source
git clone https://github.com/vllm-project/vllm-gaudi
cd vllm-gaudi
pip install -e .