Some details of ADD code.

[ShaochengShen]

Hi！
I'm currently considering applying the ADD approach to the ControlNet model to accelerate my downstream tasks to a certain extent. While reading your code, I have some questions:

1. If ControlNet undergoes distillation, does it require full parameter fine-tuning?

2. Does the training script differ from the original paper? In train.py, at line 595, why is the loss calculated as the difference between Pred_x0 and GT? Does this imply that the model can only generate in a single step rather than 1-4 steps as mentioned in the paper?

3. Can I interpret the distillation loss in the original paper as recon_loss in the script?

4. During inference, is it possible to perform multi-step inference, such as 2 or 4 steps?

I'm currently working with a very small dataset and have fine-tuned only the ControlNet through distillation according to your code. With a batch size of 4, I trained for 20,000 steps. However, when inferring with just one step, the model's final output is still noise. I would greatly appreciate your response. Thank you!

[akasharidas]

1. In some of our earlier experiments we found that distillation of the ControlNet is usually not required. For example, the ControlNet module trained for the vanilla SDXL model can be applied directly to the 4-step SDXL Turbo model with good results. You may consider trying this for your use-case. We also found good results with distilling the base model first and then training a new ControlNet for the distilled model. I'm not sure about applying distillation to the ControlNet directly.
2. In the line you mention, pred_x0 is the estimate of the noise-free image based on the noise estimation of the model from the input timestep. It does not prevent you from training a model that can generate in 1-4 steps - you just need to sample the timesteps accordingly (see line 534).
3. Yes
4. We tried performing multi-step inference using our 1-step model, but we found that it produced undesirable outputs that look over-sharpened. For multi-step inference it is best to distill a model in a multi-step setting, which is supported in our training script by passing the num_ts argument.

[ShaochengShen]

Thank you very much for your response. I feel that your reply has resolved all my doubts. This repository is excellent and has been very helpful to me—much appreciated !