Fix ExLlamaV2 kernels on Windows: link cublas.lib and enable PLATFORM.WIN32#2944
Merged
Merged
Conversation
Qubitium
approved these changes
Jul 9, 2026
Collaborator
|
@xXpeira12 LGTM! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2936
Fixes #2937
As invited in both issues - this makes the ExLlamaV2 GPTQ/AWQ kernels usable on Windows.
Changes
gptqmodel/utils/exllamav2.py- add_exllamav2_extra_ldflags()and wire it into bothTorchOpsJitExtensiondefinitions.q_gemm.cu/q_gemm_awq.cucallcublasHgemm; on Linuxthe symbol resolves at load time, but MSVC requires the import library at link time, so the
Windows JIT build always failed with
LNK2019: unresolved external symbol cublasHgemmandsilently fell back to slower kernels. This mirrors
_extra_ldflags()ingptqmodel/exllamav3/ext.py, which already handles this. No behavior change on non-Windows(returns
[]).gptqmodel/nn_modules/qlinear/exllamav2.py,exllamav2_awq.py- addPLATFORM.WIN32toSUPPORTS_PLATFORMfor both kernels.Testing
Windows 11, Python 3.10.6, torch 2.6.0+cu124, transformers 5.12.1, MSVC 14.39 (VS 2022),
CUDA toolkit 12.4.131, RTX 3050 Ti Laptop 4 GB (SM 8.6):
ExLlamaV2 GPTQ: torch.ops JIT extension ready in 48s,ExLlamaV2 AWQ: torch.ops JIT extension ready in 61s.Qwen/Qwen2-VL-2B-Instruct-GPTQ-Int4and-AWQthroughtransformers, greedy decoding over a fixed 3-image document-understanding set with
keyword-recall scoring:
Accuracy and per-case keyword hits/misses are identical to the Triton kernels, so the numerics
match on win32.
I understand CI has no Windows coverage (#2937) - happy to adjust if you'd prefer the platform
enablement gated behind an env flag instead.