Question about the freeze_vision_tower setting in training config

[coderpracticer]

Hi,

I noticed that freeze_vision_tower is set to False in the current implementation. In most multimodal training frameworks (like LLaVA), this parameter is typically set to True by default to keep the pre-trained visual weights intact.

Is this an intentional design choice for this project? I'd appreciate it if you could clarify the rationale behind training the vision tower here. Thanks!
@zli12321 @wyu97

[zli12321]

Oh, my mistake. The vision tower should be set to True. It was set to False to test low GPU memory settings but forgot to revert it. You can set it to True.

[coderpracticer]

Thanks for your reply. I encountered another error during the reinforcement learning (RL) stage when trying to use a model that was previously trained with the command:
FORCE_TORCHRUN=1 llamafactory-cli train examples/train_full/Vision-SR1-Cold-Start.yaml
The RL stage works perfectly if I use the SFT model you provided. However, after training with the above command, the resulting model fails when loaded in the RL phase with the following traceback:

File "/home/Vision-SR1/verl/single_controller/ray/base.py", line 432, in func
  return getattr(self.worker_dict[key], name)(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/Vision-SR1/verl/single_controller/base/decorator.py", line 207, in inner
  return func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^
File "/home/Vision-SR1/verl/workers/fsdp_workers.py", line 351, in init_model
  self._build_model_optimizer(
File "/home/Vision-SR1/verl/workers/fsdp_workers.py", line 197, in _build_model_optimizer
  model = auto_class.from_pretrained(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/miniconda3/envs/Vision-SR1/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
  return model_class.from_pretrained(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/miniconda3/envs/Vision-SR1/lib/python3.11/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
  return func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^
File "/home/miniconda3/envs/Vision-SR1/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4342, in from_pretrained
  model = cls(config, *model_args, **model_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/miniconda3/envs/Vision-SR1/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1481, in __init__
  super().__init__(config)
File "/home/miniconda3/envs/Vision-SR1/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1884, in __init__
  self.generation_config = GenerationConfig.from_model_config(config) if self.can_generate() else None
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/miniconda3/envs/Vision-SR1/lib/python3.11/site-packages/transformers/generation/configuration_utils.py", line 1290, in from_model_config
  decoder_config_dict = decoder_config.to_dict()
                        ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'to_dict'

Has you encountered this before? Any workaround or fix would be greatly appreciated.
@zli12321

[zli12321]

I think there is a mismatch between the new Llama-Factory repo and the old easyR1 repo. We may need to change the config file after training llama factory.

The response from Calude to resolve it:

Based on my research, here's what's going on with this issue:

Root Cause: LLaMA-Factory saves config.json with sub-configs as plain dicts instead of proper PretrainedConfig objects

This is a known bug in the HuggingFace transformers library (issue #36281) that affects Qwen2.5-VL models specifically when they've been fine-tuned and saved by tools like LLaMA-Factory or Axolotl. The issue is in transformers/generation/configuration_utils.py — when a composite vision-language model like Qwen2.5-VL is saved after SFT, the text_config field in config.json gets serialized as a plain Python dict rather than a proper PretrainedConfig object. When the RL stage (EasyR1/verl) later tries to load this saved model, the GenerationConfig.from_model_config() method calls decoder_config.to_dict() on what is now a raw dict, causing the AttributeError.

The reason the provided SFT model works but a user-trained one doesn't is likely that the provided model's config.json was either manually fixed or saved by a version/method that properly serializes the nested config objects.

This is a mismatch between LLaMA-Factory's model saving behavior and what EasyR1/transformers expects when loading. LLaMA-Factory doesn't necessarily ensure that composite model configs (with nested text_config, vision_config) are saved as proper config class instances — they get flattened to dicts in the JSON, and on reload, they stay as dicts instead of being re-instantiated as PretrainedConfig objects.

Workarounds for the user:

1. Fix the config after SFT training — After the LLaMA-Factory SFT step finishes, run a small script to reload and re-save the config properly:

from transformers import AutoConfig, PretrainedConfig

config = AutoConfig.from_pretrained("path/to/sft/output", trust_remote_code=True)

# Force sub-configs to be proper PretrainedConfig objects
if isinstance(getattr(config, 'text_config', None), dict):
    config.text_config = PretrainedConfig.from_dict(config.text_config)
if isinstance(getattr(config, 'vision_config', None), dict):
    config.vision_config = PretrainedConfig.from_dict(config.vision_config)

config.save_pretrained("path/to/sft/output")

2. Copy config files from the original Qwen2.5-VL checkpoint — Copy the original config.json and preprocessor_config.json from Qwen/Qwen2.5-VL-7B-Instruct into the SFT output directory, replacing the ones LLaMA-Factory generated. The model weights are compatible; it's just the config metadata that's malformed.

3. Upgrade transformers — There's a suggested one-line fix in the transformers repo (adding an isinstance check before calling .to_dict()), though as of my knowledge this may not have been merged yet. You could monkey-patch the line locally.

This is fundamentally a transformers library bug (not properly handling dict-typed sub-configs) combined with a LLaMA-Factory saving behavior (not ensuring sub-configs are saved/loaded as proper config class instances). It's not specifically an EasyR1 issue, though EasyR1 is where the error surfaces because that's where the model gets reloaded.

[zli12321]

@coderpracticer Just updated the code base to reflect the newest easyR1 version. Now might work with SFT checkpoints. But you should reinstall the environment to run the new code base.

[coderpracticer]

@zli12321 Thanks for the update. After upgrading transformers from the initial version, my issue has been resolved. I also really appreciate the clean and clear new code you provided.