Skip to content

Conversation

gejq21
Copy link

@gejq21 gejq21 commented Apr 20, 2025

Motivation

Support V2PE in pre-training, fine-tuning, and inference for InternVL.

Modification

V2PE utils

Added the file internvl_chat/internvl/v2pe_utils.py.
It includes the get_rope_pos_id function, which calculates position ids required for V2PE,
and a V2PE module that can replace the conventional RoPE mechanism.

Pre-training and Fine-tuning

Modified the following files to support V2PE in training and fine-tuning:

  • internvl_chat/internvl/train/internvl_chat_pretrain.py
  • internvl_chat/internvl/train/internvl_chat_finetune.py

Specifically:

  • Extended ModelArguments and LazySupervisedDataset to support V2PE.
  • Updated concat_pad_data_collator in internvl_chat/internvl/patch/pad_data_collator.py
    to support passing position ids as lists (to preserve float precision when using V2PE).

Model

InternVLChat

  • Modified internvl_chat/internvl/model/internvl_chat/configuration_internvl_chat.py
    to support passing V2PE-related arguments via config.
  • Updated InternVLChatModel.forward() and InternVLChatModel.chat()
    in internvl_chat/internvl/model/internvl_chat/modeling_internvl_chat.py
    to support V2PE usage during both training and inference.

InternLM2

  • Modified internvl_chat/internvl/model/internlm2/configuration_internlm2.py
    to support V2PE-related config arguments.
  • Updated InternLM2Attention._init_rope(), InternLM2Attention.forward(),
    and InternLM2FlashAttention2.forward() in
    internvl_chat/internvl/model/internlm2/modeling_internlm2.py
    to support replacing RoPE with the V2PE module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants