Skip to content

Port the glm4.5v model. #1962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: aice/v1.22.0
Choose a base branch
from
Open

Port the glm4.5v model. #1962

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
9947fa6
Update the glm4_1v model
HeJunyan Aug 20, 2025
bcd7770
[Misc] Remove _maybe_ignore_quant_config from GLM4.1v #20432
HeJunyan Aug 20, 2025
08f55da
[Misc] Add SPDX-FileCopyrightText #20428
HeJunyan Aug 20, 2025
bc4e92f
[Bugfix] Fix GLM-4.1-V video prompt update #20635
HeJunyan Aug 20, 2025
76cdf2b
[Bugfix] Fix OOM tests in initialization test #21921
HeJunyan Aug 20, 2025
9b4bb8c
[Bugfix] Fix glm4.1v video inference issue #22067
HeJunyan Aug 20, 2025
1a8c6a6
for glm-4.1V update #22000
HeJunyan Aug 20, 2025
ab5acb5
[Model] Decouple glm4v #22751
HeJunyan Aug 20, 2025
bd793a0
[Model] Add missing prefix to glm4_1v #22716
HeJunyan Aug 20, 2025
5bf0d61
In uses_mrope, consider text_config
HeJunyan Aug 30, 2025
e8a4e73
Add the padding to glm4_1v, make it work on Gaudi
HeJunyan Sep 22, 2025
d26cf17
add glm4_5v to vision_language.py
HeJunyan Sep 9, 2025
b646699
Disable the graph mode for model.visual temporarily
HeJunyan Sep 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/models/supported_models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,7 +535,7 @@ Specified using `--task generate`.
| `GLM4VForCausalLM`<sup>^</sup> | GLM-4V | T + I | `zai-org/glm-4v-9b`, `zai-org/cogagent-9b-20241220` etc. | ✅︎ | ✅︎ | ✅︎ |
| `Glm4vForConditionalGeneration` | GLM-4.1V-Thinking | T + I<sup>E+</sup> + V<sup>E+</sup> | `zai-org/GLM-4.1V-9B-Thinkg`, etc. | ✅︎ | ✅︎ | ✅︎ |
| `Glm4MoeForCausalLM` | GLM-4.5 | T + I<sup>E+</sup> + V<sup>E+</sup> | `zai-org/GLM-4.5`, etc. | ✅︎ | ✅︎ | ✅︎ |
| `Glm4v_moeForConditionalGeneration` | GLM-4.5V | T + I<sup>E+</sup> + V<sup>E+</sup> | `zai-org/GLM-4.5V`, etc. | ✅︎ | ✅︎ | ✅︎ |
| `Glm4vMoeForConditionalGeneration` | GLM-4.5V | T + I<sup>E+</sup> + V<sup>E+</sup> | `zai-org/GLM-4.5V`, etc. | ✅︎ | ✅︎ | ✅︎ |
| `GraniteSpeechForConditionalGeneration` | Granite Speech | T + A | `ibm-granite/granite-speech-3.3-8b` | ✅︎ | ✅︎ | ✅︎ |
| `H2OVLChatModel` | H2OVL | T + I<sup>E+</sup> | `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc. | | ✅︎ | ✅︎\* |
| `Idefics3ForConditionalGeneration` | Idefics3 | T + I | `HuggingFaceM4/Idefics3-8B-Llama3` etc. | ✅︎ | | ✅︎ |
Expand Down
36 changes: 36 additions & 0 deletions examples/offline_inference/vision_language.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,41 @@ def run_glm4_1v(questions: list[str], modality: str) -> ModelRequestData:
prompts=prompts,
)

# GLM-4.5V
def run_glm4_5v(questions: list[str], modality: str) -> ModelRequestData:
model_name = "zai-org/GLM-4.5V"

engine_args = EngineArgs(
model=model_name,
max_model_len=4096,
max_num_seqs=2,
mm_processor_kwargs={
"size": {"shortest_edge": 12544, "longest_edge": 47040000},
"fps": 1,
},
limit_mm_per_prompt={modality: 1},
tensor_parallel_size=4,
)

if modality == "image":
placeholder = "<|begin_of_image|><|image|><|end_of_image|>"
elif modality == "video":
placeholder = "<|begin_of_video|><|video|><|end_of_video|>"

prompts = [
(
"[gMASK]<sop><|system|>\nYou are a helpful assistant.<|user|>\n"
f"{placeholder}"
f"{question}<|assistant|>assistant\n"
)
for question in questions
]

return ModelRequestData(
engine_args=engine_args,
prompts=prompts,
)


# H2OVL-Mississippi
def run_h2ovl(questions: list[str], modality: str) -> ModelRequestData:
Expand Down Expand Up @@ -1120,6 +1155,7 @@ def run_skyworkr1v(questions: list[str], modality: str) -> ModelRequestData:
"gemma3": run_gemma3,
"glm4v": run_glm4v,
"glm4_1v": run_glm4_1v,
"glm4_5v": run_glm4_5v,
"h2ovl_chat": run_h2ovl,
"idefics3": run_idefics3,
"internvl_chat": run_internvl,
Expand Down
Loading