Skip to content

fix(gptq): delegate native GPTQModel loads - #2469

Open
ZX-ModelCloud wants to merge 5 commits into
huggingface:mainfrom
Qubitium:fix/gptqmodel-native-load-bridge
Open

fix(gptq): delegate native GPTQModel loads#2469
ZX-ModelCloud wants to merge 5 commits into
huggingface:mainfrom
Qubitium:fix/gptqmodel-native-load-bridge

Conversation

@ZX-ModelCloud

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR allows Optimum to delegate native GPTQModel checkpoint preparation and post-initialization to GPTQModel when its optional loading bridge is available.

The existing GPTQ loading path discovers linear modules from the model structure and replaces them using a globally selected QuantLinear implementation. Native GPTQModel checkpoints may instead contain:

  • A mix of dense and quantized modules.
  • Per-module dynamic quantization settings.
  • Model-specific structural conversions.
  • Quantized modules determined by the serialized weight manifest.

Replacing modules outside that manifest can incorrectly convert an intentionally dense layer or select a kernel that does not support its shape.

This PR:

  • Feature-detects hf_gptqmodel_prepare_model_for_load and hf_gptqmodel_post_init_for_load.
  • Delegates model conversion to GPTQModel when it recognizes a native checkpoint.
  • Preserves the returned quantization configuration and selected kernel.
  • Runs GPTQModel post-initialization after checkpoint tensors are loaded.
  • Keeps the existing Optimum loading path unchanged when the bridge is unavailable or does not handle the checkpoint.

There is no new hard dependency. GPTQModel versions that do not expose the bridge APIs continue using the existing Optimum behavior.

Testing

  • pytest -q tests/gptq/test_quantization.py -k "GPTQNativeLoadBridgeTest or GPTQPostInitTest"
    • all passed
  • ruff check optimum/gptq/quantizer.py tests/gptq/test_quantization.py
  • python -m py_compile optimum/gptq/quantizer.py tests/gptq/test_quantization.py

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes? No user-facing documentation change is required.
  • Did you write any new necessary tests?

Who can review?

GPTQ and quantization: @SunMarc, @IlyasMoutawwakil

Comment thread optimum/gptq/quantizer.py
@ZX-ModelCloud
ZX-ModelCloud marked this pull request as ready for review July 29, 2026 02:18
@ZX-ModelCloud
ZX-ModelCloud requested a review from Qubitium July 30, 2026 09:41

@Qubitium Qubitium left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ZX-ModelCloud Unit test is testing pr change but actually not testing why we want this PR upstreamed to transformers. For example, you need write a specific test using hf/modelcloud hosted small quantied moe model with dynamic bit/groups gptq and test that it only loads correctly with our delegate.

This appears to be the last change I see before we can remove the wip tag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants