Skip to content

Add support for CogVLM model #15002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Add support for CogVLM model #15002

wants to merge 17 commits into from
+496 −19

Conversation

Tianyue-Zhao
Copy link

This addresses the requests for CogVLM in #4387 and #4350.
CogVLM is a pretty popular model that now adds in cleanly after the recent additions to libmtmd.
I've converted a GGUF here: Link to GGUF files

Sample command and output:

build/bin/llama-mtmd-cli -m ../cogvlm-chat-hf/cogvlm-13B-chat-v1.1-F16.gguf --mmproj ../cogvlm-chat-hf/mmproj-cogvlm-chat-hf --image ./community.png --chat-template vicuna -p "Describe the picture"

load_hparams: model size:         8448.53 MiB
load_hparams: metadata size:      0.36 MiB
alloc_compute_meta:        CPU compute buffer size =   142.02 MiB
main: loading model: ../cogvlm-chat-hf/cogvlm-13B-chat-v1.1-F16.gguf
encoding image slice...
image slice encoded in 16135 ms
decoding image batch 1/1, n_tokens_batch = 1227
image decoded (batch 1/1) in 54065 ms

1. The image showcases a futuristic urban landscape with a mix of architectural styles. The buildings are multi-storied and have a combination of traditional and modern elements. There's a prominent tree in the foreground, suggesting a blend of nature and urban development. The scene appears to be bustling with activity, with various signs and billboards, indicating commercial or residential zones.


llama_perf_context_print:        load time =  108969.65 ms
llama_perf_context_print: prompt eval time =   85229.27 ms /  1241 tokens (   68.68 ms per token,    14.56 tokens per second)
llama_perf_context_print:        eval time =   19843.15 ms /    83 runs   (  239.07 ms per token,     4.18 tokens per second)
llama_perf_context_print:       total time =  126951.23 ms /  1324 tokens
llama_perf_context_print:    graphs reused =          0

@github-actions github-actions bot added examples python python script changes labels Aug 1, 2025
@Tianyue-Zhao Tianyue-Zhao marked this pull request as ready for review August 1, 2025 02:15
@Tianyue-Zhao
Copy link
Author

I think I've fixed the typecheck and format check workflows that were failing before, can someone approve the workflows to run again?
Also, is there a way to run these Github workflows locally or without needing approval from a reviewer?
It would be good to run these CI/CD checks myself before posting the PR.

@CISC
Copy link
Collaborator

CISC commented Aug 2, 2025

Also, is there a way to run these Github workflows locally or without needing approval from a reviewer? It would be good to run these CI/CD checks myself before posting the PR.

You can run flake8, pyright and editorconfig locally (or via IDE plugins), the build tests can be run manually with ctest.

CISC
CISC reviewed Aug 2, 2025
View reviewed changes
Copy link
Collaborator

@CISC CISC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a complete review as I don't know enough about mtmd, just commenting...

@Tianyue-Zhao
Copy link
Author

Also, is there a way to run these Github workflows locally or without needing approval from a reviewer? It would be good to run these CI/CD checks myself before posting the PR.

You can run flake8, pyright and editorconfig locally (or via IDE plugins), the build tests can be run manually with ctest.

Thanks for the info! That's something I've been wondering about for a while.

CISC
CISC reviewed Aug 8, 2025
View reviewed changes
CISC
CISC reviewed Aug 18, 2025
View reviewed changes
Copy link
Collaborator

@CISC CISC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Further refinement (merge cont+reshape).

@CISC
Copy link
Collaborator

CISC commented Aug 29, 2025

Further refinement (merge cont+reshape).

After #15662 we can avoid these altogether and just create 3D views.

@CISC
Copy link
Collaborator

CISC commented Sep 8, 2025

Further refinement (merge cont+reshape).

After #15662 we can avoid these altogether and just create 3D views.

Merged, rebase and apply updated suggestions.

@Tianyue-Zhao
Copy link
Author

Further refinement (merge cont+reshape).

After #15662 we can avoid these altogether and just create 3D views.

Merged, rebase and apply updated suggestions.

Thanks for the reminder, I've rebased it and removed the extra ggml_cont calls.

@CISC CISC requested a review from ngxson September 9, 2025 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CISC CISC CISC left review comments

@ngxson ngxson Awaiting requested review from ngxson ngxson is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
examples python python script changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Tianyue-Zhao @CISC