Skip to content

Fix ExLlamaV2 kernels on Windows: link cublas.lib and enable PLATFORM.WIN32#2944

Merged
Qubitium merged 1 commit into
ModelCloud:mainfrom
xXpeira12:fix/windows-exllamav2-kernels
Jul 9, 2026
Merged

Fix ExLlamaV2 kernels on Windows: link cublas.lib and enable PLATFORM.WIN32#2944
Qubitium merged 1 commit into
ModelCloud:mainfrom
xXpeira12:fix/windows-exllamav2-kernels

Conversation

@xXpeira12

Copy link
Copy Markdown
Contributor

Fixes #2936
Fixes #2937

As invited in both issues - this makes the ExLlamaV2 GPTQ/AWQ kernels usable on Windows.

Changes

  1. gptqmodel/utils/exllamav2.py - add _exllamav2_extra_ldflags() and wire it into both
    TorchOpsJitExtension definitions. q_gemm.cu / q_gemm_awq.cu call cublasHgemm; on Linux
    the symbol resolves at load time, but MSVC requires the import library at link time, so the
    Windows JIT build always failed with LNK2019: unresolved external symbol cublasHgemm and
    silently fell back to slower kernels. This mirrors _extra_ldflags() in
    gptqmodel/exllamav3/ext.py, which already handles this. No behavior change on non-Windows
    (returns []).

  2. gptqmodel/nn_modules/qlinear/exllamav2.py, exllamav2_awq.py - add
    PLATFORM.WIN32 to SUPPORTS_PLATFORM for both kernels.

Testing

Windows 11, Python 3.10.6, torch 2.6.0+cu124, transformers 5.12.1, MSVC 14.39 (VS 2022),
CUDA toolkit 12.4.131, RTX 3050 Ti Laptop 4 GB (SM 8.6):

  • Both extensions JIT-compile and load: ExLlamaV2 GPTQ: torch.ops JIT extension ready in 48s,
    ExLlamaV2 AWQ: torch.ops JIT extension ready in 61s.
  • End-to-end inference with Qwen/Qwen2-VL-2B-Instruct-GPTQ-Int4 and -AWQ through
    transformers, greedy decoding over a fixed 3-image document-understanding set with
    keyword-recall scoring:
Checkpoint Kernel Decode tok/s Accuracy
GPTQ-Int4 TritonV2 (previous fallback) 5.3 94%
GPTQ-Int4 ExLlamaV2 (this PR) 7.6 94%
AWQ AwqGEMMTriton (previous fallback) 6.1 78%
AWQ ExLlamaV2 (this PR) 6.7 78%

Accuracy and per-case keyword hits/misses are identical to the Triton kernels, so the numerics
match on win32.

I understand CI has no Windows coverage (#2937) - happy to adjust if you'd prefer the platform
enablement gated behind an env flag instead.

@Qubitium Qubitium merged commit f13d578 into ModelCloud:main Jul 9, 2026
1 check passed
@Qubitium

Qubitium commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

@xXpeira12 LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants