AWQ - Compressed tensors LoRa finetuning lead to OOM while GEMM AWQ works normally

[bi1101]

Hi, I'm trying to fine-tune a Qwen3 Moe AWQ model and found a big difference between AWQ compressed-tensor models & AWQ GEMM models

Is this a bug in compressed tensor implementation?

Hardware: 6xA6000 = 288GB OOM with compressed tensor

while 1x A6000= 48Gb works perfectly with AWQ GEMM

Here's my SFT config:

# GEMM AWQ works no OOM
model_name_or_path: ELVISIO/Qwen3-30B-A3B-Instruct-2507-AWQ 

# Compressed-tensor AWQ get OOM
model_name_or_path: cpatonn/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit

# dataset
dataset_name: ...


# Lora
use_peft: True
lora_target_modules: 
  - "q_proj"
  - "k_proj"
  - "v_proj"
  - "o_proj"

# training
learning_rate: 2.0e-05
num_train_epochs: 1
packing: true
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 16
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: true
logging_steps: 1
logging_strategy: "steps"
log_level: "info"
max_length: 8000
warmup_ratio: 0.03
lr_scheduler_type: 'cosine'
bf16: true
bf16_full_eval: true
fp16: false
attn_implementation: 'flash_attention_2'

Note: In serving with vllm, compressed-tensor models seem to have better throughput. So if possible, fine-tuning with compressed-tensor format is better & more future-proof since autoawq is deprecated.

[brian-dellabetta]

Hi @bi1101 , I don't believe we have any support for fine-tuning after quantization like this. Another option would be finetuning the fp16 model and then quantizing in llm-compressor with AWQ. In our latest experiments with Qwen 3 MoE models, we effectively achieve near full recovery to the fp16 model.

[bi1101]

@brian-dellabetta Thanks for the info. That's awesome news!

Will compressed tensor supports finetuning in the future? For our usecase, we need to deploy multiple finetuned models so merging then quantizing will not work.

For now autoawq will still does the job, but unfortunately, may not work for future models. It would be great if compressed tensor can pick up firntuning feature from autoawq

[brian-dellabetta]

Hi @bi1101 , I spoke with some of the team internally. The reason AutoAWQ runs well is they use performant kernels when the model is loaded up into Huggingface. We don't, outside of a very non-performant module that can mimic quantized linear layers for dev purposes, anticipating users to run the quantized models directly in vllm. This will lead to a lot of memory and compute time when our models are loaded up in huggingface instead of vllm.

It looks like there is some QLoRA support in vllm, though we have not tested this for our models. We have discussed adding QLoRA as a feature later this year, but nothing concrete at the moment.

[bi1101]

@brian-dellabetta Thank you for the info!