train with train_text_to_image_sdxl.py CUDA out of memory

[MMY1994]

This is my shell script

export MODEL_NAME="./model_base/sd_xl_base"
export VAE_NAME="./model_base/sdxl-vae-fp16-fix"

accelerate launch
--config_file /root/.cache/huggingface/accelerate/default_config.yaml
--num_cpu_threads_per_process=8
./diffusers/examples/text_to_image/train_text_to_image_sdxl_mmy.py
--pretrained_model_name_or_path=$MODEL_NAME
--pretrained_vae_model_name_or_path=$VAE_NAME
--train_data_dir="./fineturn/fineturn.py"
--output_dir="./sdxl-dreamshape-fineturn-model/"
--enable_xformers_memory_efficient_attention
--resolution=1024 --center_crop --random_flip
--proportion_empty_prompts=0.2
--train_batch_size=1
--mixed_precision="fp16"
--gradient_accumulation_steps=4 --gradient_checkpointing
--max_train_steps=5000
--use_8bit_adam
--learning_rate=1e-07
--lr_scheduler="constant" --lr_warmup_steps=0
--validation_prompt="a young man climbing the ladder, high quality" --validation_epochs 1
--checkpointing_steps=200

I Use A800 40GB and --enable_xformers_memory_efficient_attention, --mixed_precision="fp16".

Even when I use 512 resolution and batchsize=1, this error still happen. i don't think it's normal

[tolgacangoz]

Could you share the output of diffusers-cli env command? I see that you added a suffix to the file name and main(args) is at 1333. line, is there any change in the file in this discussion's context? I wonder if --num_cpu_threads_per_process=8 tries to allocate too much data? Also, what happens when you remove --enable_xformers_memory_efficient_attention if your PyTorch>=2.0.

[MMY1994]

Thank you, I will try it

[MMY1994]

After try it, GPU memery problem seems to be related to the torch version(my torch version is 2.0.1)。accelerate config's distributed_type set to "NO" is norml( single GPU about 25GB), but set to "MULTI_GPU"is about 38GB. it reported the Info "torch.nn.parallel.distributed - Reducer buckets have been rebuilt in this iteration".
I will try upgrade them

[MMY1994]

I'm sorry to interrupt, but have you tried single machine multi GPU and distribution, like this

accelerate env:

• Accelerate default config:
  - compute_environment: LOCAL_MACHINE
  - distributed_type: MULTI_GPU
  - mixed_precision: fp16
  - use_cpu: False
  - debug: False
  - num_processes: 8
  - machine_rank: 0
  - num_machines: 1
  - gpu_ids: all
  - rdzv_backend: static
  - same_network: True
  - main_training_function: main
  - enable_cpu_affinity: False
  - downcast_bf16: no
  - tpu_use_cluster: False
  - tpu_use_sudo: False
  - tpu_env: []

According to the results of my experiment, single GPU is normal, but multi-GPUs still takes up too much GPU memory

[tolgacangoz]

AFAIU, PagedAdamW8bit uses as much GPU vRAM as possible. If there seems to be a possibility of OOM, then at that point, it starts to utilize CPU RAM. If multi-GPUs have more vRAM than single GPU then multi-GPUs will use more vRAM according to training needs.
Actually, my knowledge about distributed training is very limited. It might be more proper to open a new discussion about that.

[wzhiyuan2016]

I'm sorry to interrupt, but have you tried single machine multi GPU and distribution, like this

accelerate env:

• Accelerate default config:
  • compute_environment: LOCAL_MACHINE
  • distributed_type: MULTI_GPU
  • mixed_precision: fp16
  • use_cpu: False
  • debug: False
  • num_processes: 8
  • machine_rank: 0
  • num_machines: 1
  • gpu_ids: all
  • rdzv_backend: static
  • same_network: True
  • main_training_function: main
  • enable_cpu_affinity: False
  • downcast_bf16: no
  • tpu_use_cluster: False
  • tpu_use_sudo: False
  • tpu_env: []

According to the results of my experiment, single GPU is normal, but multi-GPUs still takes up too much GPU memory

Hello, did you successfully train in the end? Have you trained your own dataset? How does the trained model generate
I am training my own data and the generation effect is very poor. I don't know why
Some of my discussions are here====>#10195

[DuShunpeng]

This is my shell script

export MODEL_NAME="./model_base/sd_xl_base" export VAE_NAME="./model_base/sdxl-vae-fp16-fix"

accelerate launch --config_file /root/.cache/huggingface/accelerate/default_config.yaml --num_cpu_threads_per_process=8 ./diffusers/examples/text_to_image/train_text_to_image_sdxl_mmy.py --pretrained_model_name_or_path=$MODEL_NAME --pretrained_vae_model_name_or_path=$VAE_NAME --train_data_dir="./fineturn/fineturn.py" --output_dir="./sdxl-dreamshape-fineturn-model/" --enable_xformers_memory_efficient_attention --resolution=1024 --center_crop --random_flip --proportion_empty_prompts=0.2 --train_batch_size=1 --mixed_precision="fp16" --gradient_accumulation_steps=4 --gradient_checkpointing --max_train_steps=5000 --use_8bit_adam --learning_rate=1e-07 --lr_scheduler="constant" --lr_warmup_steps=0 --validation_prompt="a young man climbing the ladder, high quality" --validation_epochs 1 --checkpointing_steps=200

I Use A800 40GB and --enable_xformers_memory_efficient_attention, --mixed_precision="fp16".

Even when I use 512 resolution and batchsize=1, this error still happen. i don't think it's normal

hi, @MMY1994 :
im not sure that "35GB when setting --train_batch_size=1 and --resolution=1024." is true or not,
but pls try as below:
1、make sure that " --use_8bit_adam --enable_xformers_memory_efficient_attention --set_grads_to_none "
2、find "optimizer_class = bnb.optim.AdamW8bit" => "bnb.optim.PagedAdamW8bit " . train_controlnet_sdxl.py or whatever you are using

Good luck!