[Model] support new model ovis2.5 #22187

myselvess · 2025-08-04T09:27:45Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Adding support for the ovis2.5

Test Plan

Test Result

(Optional) Documentation Update

Updated supported_models.md

Signed-off-by: myselvess <[email protected]>

mergify · 2025-08-04T09:28:22Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @myselvess.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

github-actions · 2025-08-04T09:30:14Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Code Review

This pull request adds support for the new ovis2.5 model. The changes are comprehensive, including the model implementation, configuration, processor, and updates to the model registry and documentation. My review has identified several potential issues that should be addressed. These include a copy-paste error in an error message, incorrect logic for handling multimodal inputs, a potential AttributeError during training with gradient checkpointing, a potential AttributeError in the model configuration, and missing keys in the special tokens dictionary which could lead to a KeyError.

vllm/transformers_utils/configs/ovis2_5.py

vllm/transformers_utils/processors/ovis2_5.py

vllm/model_executor/models/ovis2_5.py

vllm/model_executor/models/siglip2navit.py

Signed-off-by: myselvess <[email protected]>

hmellor · 2025-08-04T17:22:11Z

vllm/transformers_utils/configs/ovis2_5.py

Could this be imported from Transformers instead?

Thank you for the suggestion! Unfortunately, this cannot be imported from Transformers because the library does not currently support or recognize the ovis25 architecture.

If this config is copied from model repo's configuration_*.py, we can simply add trust_remote_code=True to use remote implementation.

If this config is copied from model repo's configuration_*.py, we can simply add trust_remote_code=True to use remote implementation.

I will check and try this method. :)

If this config is copied from model repo's configuration_*.py, we can simply add trust_remote_code=True to use remote implementation.

I noticed that vllm/transformers_utils/configs/ovis.py was reintroduced in this PR (#21931). Could you clarify the reasoning behind this change? Should I still need to remove ovis2.5 config？

Ah, it's because Ovis2's remote config file is broken after Transformers v4.54 due to these lines about AimV2 ViT: https://huggingface.co/AIDC-AI/Ovis2-1B/blob/main/configuration_ovis.py#L14-L15.

Seems Ovis2.5 has different model_type and ViT (ovis vs ovis2_5) which won't be affected by re-added config, so I think it should still be fine to remove ovis2.5 config if it can run with trust_remote_code=True.

If remote code is also broken in Ovis2.5, we can keep it here :)

Unmaintained remote code models are the worst 😭

The DCO check shows that there are several commits without signatures. So I tried to revise the signatures, but after force pushing, this PR ended up with too many commits.😭 I will close this PR and resubmit a new one.

docs/models/supported_models.md

vllm/transformers_utils/processors/ovis2_5.py

Isotr0py · 2025-08-05T03:23:35Z

vllm/transformers_utils/configs/ovis2_5.py

If this config is copied from model repo's configuration_*.py, we can simply add trust_remote_code=True to use remote implementation.

Isotr0py · 2025-08-05T03:25:04Z

vllm/model_executor/models/ovis2_5.py

+@MULTIMODAL_REGISTRY.register_processor(Ovis2_5MultiModalProcessor,
+                                        info=Ovis2_5ProcessingInfo,
+                                        dummy_inputs=Ovis2_5DummyInputsBuilder)
+class Ovis2_5(nn.Module, SupportsMultiModal):


Any difference between Ovis2/Ovis1.6 and Ovis2.5? Seems they have similar architecture.

Ovis2.5 introduces several differences compared to Ovis2/Ovis1.6:
1.The LLM and ViT components use different backbone models.
2.There are updates in input preprocessing and multi-modal embedding fusion.
3.Ovis2.5 supports video inference.

1.The LLM and ViT components use different backbone models.

The existing Ovis implementation can initialize arbitrary backbone, so this won't be a blocker.

3.Ovis2.5 supports video inference.

IIRC, Ovis2/Ovis1.6 should also support video inputs though I didn't finish its implementation before. (https://huggingface.co/AIDC-AI/Ovis2-2B#usage)

2.There are updates in input preprocessing and multi-modal embedding fusion.

So the only large issue is the updates for preprocessing and multi-modal embedding fusion. I wonder if we can make update on existing Ovis implementation (both model and processor implementation) for Ovis2.5 support, so that Ovis2/Ovis1.6 can also benefit from video support in this PR.

It will require handling many specific details depending on the model type. For example, some fields in the config have different names or position, and indicator definitions also vary.
Perhaps I could first add video input support to the existing Ovis implementation, and then define a new Ovis2.5 class that inherits from Ovis to handle the specific adjustments for Ovis2.5?

It will require handling many specific details depending on the model type. For example, some fields in the config have different names or position, and indicator definitions also vary.

I see, seems that the difference between Ovis2 and Ovis2.5 is a little bit large, let's focus on Ovis2.5 in this PR and leave video input support for existing Ovis implementation in a following PR :)

vllm/model_executor/models/siglip2navit.py

Isotr0py · 2025-08-05T11:17:22Z

vllm/model_executor/models/ovis2_5.py

+@MULTIMODAL_REGISTRY.register_processor(Ovis2_5MultiModalProcessor,
+                                        info=Ovis2_5ProcessingInfo,
+                                        dummy_inputs=Ovis2_5DummyInputsBuilder)
+class Ovis2_5(nn.Module, SupportsMultiModal):


1.The LLM and ViT components use different backbone models.

The existing Ovis implementation can initialize arbitrary backbone, so this won't be a blocker.

3.Ovis2.5 supports video inference.

IIRC, Ovis2/Ovis1.6 should also support video inputs though I didn't finish its implementation before. (https://huggingface.co/AIDC-AI/Ovis2-2B#usage)

2.There are updates in input preprocessing and multi-modal embedding fusion.

So the only large issue is the updates for preprocessing and multi-modal embedding fusion. I wonder if we can make update on existing Ovis implementation (both model and processor implementation) for Ovis2.5 support, so that Ovis2/Ovis1.6 can also benefit from video support in this PR.

vllm/model_executor/models/siglip2navit.py

mergify · 2025-08-06T11:35:53Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @myselvess.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

vllm/model_executor/models/ovis2_5.py

docs/models/supported_models.md

vllm/model_executor/models/registry.py

vllm/model_executor/models/siglip2navit.py

Isotr0py · 2025-08-06T12:18:59Z

vllm/transformers_utils/configs/ovis2_5.py

Ah, it's because Ovis2's remote config file is broken after Transformers v4.54 due to these lines about AimV2 ViT: https://huggingface.co/AIDC-AI/Ovis2-1B/blob/main/configuration_ovis.py#L14-L15.

Seems Ovis2.5 has different model_type and ViT (ovis vs ovis2_5) which won't be affected by re-added config, so I think it should still be fine to remove ovis2.5 config if it can run with trust_remote_code=True.

If remote code is also broken in Ovis2.5, we can keep it here :)

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: myselvess <[email protected]>

Signed-off-by: myselvess <[email protected]>

Signed-off-by: 22quinn <[email protected]>

Signed-off-by: DarkLight1337 <[email protected]>

…#21725) Signed-off-by: alyosha-swamy <[email protected]>

vllm-project#22185) Signed-off-by: Woosuk Kwon <[email protected]>

mergify · 2025-08-07T12:45:30Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @myselvess.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

DarkLight1337 · 2025-08-16T02:14:27Z

I see Ovis 2.5 is available on HF Hub now. Are you planning to open a new PR for this model?

support new model ovis2_5

1d097a8

Signed-off-by: myselvess <[email protected]>

myselvess requested a review from hmellor as a code owner August 4, 2025 09:27

mergify bot added the documentation Improvements or additions to documentation label Aug 4, 2025

mergify bot added new-model Requests to new models needs-rebase labels Aug 4, 2025

gemini-code-assist bot reviewed Aug 4, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into ovis2_5

a076041

Signed-off-by: myselvess <[email protected]>

mergify bot removed the needs-rebase label Aug 4, 2025

hmellor reviewed Aug 4, 2025

View reviewed changes

Isotr0py reviewed Aug 5, 2025

View reviewed changes

mergify bot added needs-rebase and removed needs-rebase labels Aug 6, 2025

Isotr0py reviewed Aug 6, 2025

View reviewed changes

myselvess requested review from DarkLight1337 and ywang96 as code owners August 7, 2025 08:36

mergify bot added the multi-modality Related to multi-modality (#4194) label Aug 7, 2025

myselvess and others added 10 commits August 7, 2025 20:35

Update vllm/transformers_utils/configs/ovis2_5.py

59e35e7

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: myselvess <[email protected]>

Update vllm/model_executor/models/ovis2_5.py

7339792

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: myselvess <[email protected]>

Update vllm/model_executor/models/ovis2_5.py

86ee33c

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: myselvess <[email protected]>

update ovis25

f5f32fe

Signed-off-by: myselvess <[email protected]>

update ovis25: fix pre-commit error

5fa4392

Signed-off-by: myselvess <[email protected]>

update ovis25: rm useless code & lazy import fa & add sdpa fallback

a4aca4b

Signed-off-by: myselvess <[email protected]>

[Sampler] Support returning all logprobs or logits (vllm-project#21792)

48bb18b

Signed-off-by: 22quinn <[email protected]>

[Doc] Update pooling model docs (vllm-project#22186)

034a6e6

Signed-off-by: DarkLight1337 <[email protected]>

Fix Arcee model weight loading: Add custom load_weights (vllm-project…

2b6b1b0

…#21725) Signed-off-by: alyosha-swamy <[email protected]>

[Responses API] Ignore store=True and process the request by default (

8aca87d

vllm-project#22185) Signed-off-by: Woosuk Kwon <[email protected]>

myselvess requested review from simon-mo, aarnphm, russellb, njhill, comaniac, alexm-redhat, youkaichao, houseroad, zou3519 and ProExpertProg as code owners August 7, 2025 12:44

mergify bot added ci/build frontend performance Performance-related issues qwen Related to Qwen models rocm Related to AMD ROCm structured-output speculative-decoding v1 labels Aug 7, 2025

github-project-automation bot added this to Structured Output Aug 7, 2025

mergify bot added tpu Related to Google TPUs tool-calling labels Aug 7, 2025

github-project-automation bot added this to Tool Calling Aug 7, 2025

mergify bot added the needs-rebase label Aug 7, 2025

myselvess closed this Aug 7, 2025

github-project-automation bot moved this to Done in Structured Output Aug 7, 2025

github-project-automation bot moved this to Done in Tool Calling Aug 7, 2025

Isotr0py mentioned this pull request Aug 16, 2025

[Feature]: Support for AIDC-AI/Ovis2.5-9B multimodal #23011

Closed

1 task

myselvess deleted the ovis2_5 branch August 26, 2025 02:57

Uh oh!

[Model] support new model ovis2.5 #22187

[Model] support new model ovis2.5 #22187

Uh oh!

Conversation

myselvess commented Aug 4, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

mergify bot commented Aug 4, 2025

Uh oh!

github-actions bot commented Aug 4, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Isotr0py Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Isotr0py Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mergify bot commented Aug 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Aug 7, 2025

Uh oh!

DarkLight1337 commented Aug 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

myselvess commented Aug 4, 2025 •

edited by github-actions bot

Loading

Isotr0py Aug 5, 2025 •

edited

Loading

Isotr0py Aug 5, 2025 •

edited

Loading

DarkLight1337 commented Aug 16, 2025 •

edited

Loading