Enable internVL #1997
Merged
Enable internVL #1997
+961
−180
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes HPU graph issues for gemma3 vision inputs Text warmup to include attn_mask info, so vision+text data can reuse the graph for language model that's warmed up already. Changing slicing to index_select for multimodal bucketing for HPU. Slicing doesn't produce the same hash for the HPU graph with same input shape. Use buckets for the vision tower as well to reduce GC recompile Accuracy bug fix by clone output data of the multimodal-projector. Validated with Muirbench datasets.
Add missing modelscope package - `VLLM_USE_MODELSCOPE` env doesn't work without it.
## Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results ## Purpose ## Test Plan ## Test Result <!--- pyml disable-next-line no-emphasis-as-heading --> --------- Signed-off-by: Libin Tang <[email protected]> Co-authored-by: Libin Tang <[email protected]> Co-authored-by: Libin Tang <[email protected]>
Introduce VLLM_WARMUP_WITH_PENALTY to call apply penalty code in sampler during warmup also. https://github.com/HabanaAI/vllm-fork/blob/libint/intervl_bucket/vllm/model_executor/layers/sampler.py#L280 is not called during warmup. And it causes extra graph compile during runtime as it sets to True for real run.
in sampling if do penalties, the prompt_tokens regenerates for each decode, that takes time. instead we can use cache it, and reset when requests set changes
Co-authored-by: Libin Tang <[email protected]>
yeonsily
requested review from
PatrykWo,
afierka-intel,
deepvars,
jikunshang,
kzawora-intel,
madamczyk-intel,
mgawarkiewicz-intel,
michalkuligowski,
mswiniarsk,
vivekgoe,
wpyszka and
xuechendi
as code owners
Open
libinta
reviewed
Oct 3, 2025
libinta
reviewed
Oct 3, 2025
libinta
reviewed
Oct 3, 2025
|
/run-gaudi-tests |
|
/run-gaudi-tests |
michalkuligowski
approved these changes
Oct 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.