Skip to content

Conversation

@danisereb
Copy link
Contributor

@danisereb danisereb commented Feb 12, 2026

Purpose

In current vLLM main there is not support for LoRA with NVFP4 MoE models.

This PR enables LoRA support by using the existing MarlinExperts backend.

Test Plan

Test support for LoRA adapters with Nemotron Nano NVFP4:
https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4

LoRA adapters for testing can be found here:
https://huggingface.co/models?other=base_model:adapter:nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

Test Result

Tested on B200 (requires env-var VLLM_LORA_DISABLE_PDL=1).


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for LoRA with NVFP4 Mixture-of-Experts models. The changes achieve this by forcing the selection of the MARLIN backend when LoRA is enabled for NVFP4 MoE, as it provides the necessary moe_sum method for LoRA injection. A warning is correctly logged to inform the user about this backend selection. Additionally, the pull request includes a robustness improvement in vllm/lora/layers/fused_moe.py to handle both positional and keyword arguments when accessing parameters in a decorated function, which prevents potential KeyError exceptions for optional arguments. The changes appear correct and well-implemented.

@danisereb danisereb marked this pull request as ready for review February 12, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant