[Frontend] Add chunked processing to handle long inputs in embedding models #22280
Conversation
…scripts, incorporating chunk processing capabilities to handle exceptionally long inputs. The README documentation has been revised to provide comprehensive instructions on usage methods and configuration options. Signed-off-by: x22x22 <[email protected]>
x22x22
requested review from
simon-mo,
WoosukKwon,
youkaichao,
robertgshaw2-redhat,
mgoin,
tlrmchlsmth,
houseroad,
hmellor and
aarnphm
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
mergify
bot
added
documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces chunked processing for long text embeddings, allowing vLLM to handle inputs that exceed the model's maximum context length. The changes include new configuration options in PoolerConfig, core logic for chunking and aggregation in serving_embedding.py, and several robustness improvements in serving_engine.py. Additionally, new examples are provided to demonstrate and test the new feature.
My review found one potential high-severity issue in vllm/config.py where logic for selecting the correct transformers backend for pooling models seems to have been accidentally removed. This could lead to errors when using pooling models with the transformers backend. The rest of the implementation for chunked processing appears solid and well-designed.
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
…iguration options to facilitate long-text input support. Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
… Extensive Texts Signed-off-by: x22x22 <[email protected]>
- Restore vllm/transformers_utils/processor.py to main branch - Restore vllm/inputs/registry.py to main branch - Ensure all file metadata matches main branch exactly Signed-off-by: x22x22 <[email protected]>
…edding generation. Signed-off-by: x22x22 <[email protected]>
…edding generation. Signed-off-by: x22x22 <[email protected]>
…edding generation. Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Signed-off-by: x22x22 <[email protected]>
|
@maxdebayser @DarkLight1337 @hmellor |
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
|
Let's merge this |
DarkLight1337
added
the
ready
python -m mypy vllm/entrypoints/openai/serving_embedding.py --python-version 3.11
vllm/entrypoints/openai/serving_embedding.py:160: error: Incompatible return value type (got "PoolerConfig | bool | None", expected "bool") [return-value]
Found 1 error in 1 file (checked 1 source file)After merging the main, new errors occurred. Let me handle it. |
Signed-off-by: x22x22 <[email protected]>
…models (vllm-project#22280) Signed-off-by: x22x22 <[email protected]> Signed-off-by: Kdump <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Maximilien de Bayser <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>
…models (vllm-project#22280) Signed-off-by: x22x22 <[email protected]> Signed-off-by: Kdump <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Maximilien de Bayser <[email protected]> Co-authored-by: DarkLight1337 <[email protected]> Signed-off-by: Boyuan Feng <[email protected]>
…models (vllm-project#22280) Signed-off-by: x22x22 <[email protected]> Signed-off-by: Kdump <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Maximilien de Bayser <[email protected]> Co-authored-by: DarkLight1337 <[email protected]> Signed-off-by: Diego-Castan <[email protected]>
…models (vllm-project#22280) Signed-off-by: x22x22 <[email protected]> Signed-off-by: Kdump <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Maximilien de Bayser <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>
…models (vllm-project#22280) Signed-off-by: x22x22 <[email protected]> Signed-off-by: Kdump <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Maximilien de Bayser <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>
Original pr
#20837
The previous submission was not merged for too long, resulting in too many conflicts, so I'm resubmitting it.
@noooop
@hmellor
@maxdebayser