The denoise function output X0, XT ? why using XT as F_theta to calculate grad

[phyllispeng123]

Thank you for your great work!!! I have a few questions on the training code in rcm/models/t2v_model_distill_rcm.py

1. As the consistency function in rCM paper stated fθ(x, t) = cskip(t)x + cout(t) * Fθ(cin(t)x, cnoise(t)), I suppose dx/dt can be replaced by the teacher output Fteacher only when Fθ(cin(t)x, cnoise(t)) represent the velocity. However, the code in the self.denoise function output F_pred_B_C_T_H_W = (torch.cos(time_B_1_T_1_1) * xt_B_C_T_H_W - x0_pred_B_C_T_H_W) / torch.sin(time_B_1_T_1_1) which is the prediction of the noise by using the trigflow function? I wonder why not output net_output_B_C_T_H_W?
2. Have you try using global velocity instead of teacher output to calculate g_B_C_T_H_W ?

Maybe I misunderstood, looking forward to your reply, many thanks!

[zhengkw18]

Thanks for your interest in our work, and for carefully reading the paper and code!

1. F_pred_B_C_T_H_W = (torch.cos(time_B_1_T_1_1) * xt_B_C_T_H_W - x0_pred_B_C_T_H_W) / torch.sin(time_B_1_T_1_1) is not the noise prediction (I have fixed the annotation for the denoise() function), but the v prediction. Specifically, in TrigFlow, xt=cost*x0 + sint*ε, and v=cost*ε-sint*x0. Therefore, cost*xt-x0=sint*(cost*ε-sint*x0), and we have (cost*xt-x0)/sint=v. The reason why not directly using net_output_B_C_T_H_W is that we want to set up a wrapper so that any noise schedule can adapt to the TrigFlow schedule, as stated in Section 3.1 of the paper.

2. No. If using the global velocity instead of the teacher signal, it is no longer distillation. Intuitively, this will make the training more sensitive to the data quality (in distillation we can use pure synthetic data), and the supervision signal is less clean than using a teacher. I know that works like MeanFlow wants to train from scratch, but according to our knowledge, stable teacher signals are essential for large-scale tasks like 14B video model.

[phyllispeng123]

Thanks for your interest in our work, and for carefully reading the paper and code!

1. F_pred_B_C_T_H_W = (torch.cos(time_B_1_T_1_1) * xt_B_C_T_H_W - x0_pred_B_C_T_H_W) / torch.sin(time_B_1_T_1_1) is not the noise prediction (I have fixed the annotation for the denoise() function), but the v prediction. Specifically, in TrigFlow, xt=cost*x0 + sint*ε, and v=cost*ε-sint*x0. Therefore, cost*xt-x0=sint*(cost*ε-sint*x0), and we have (cost*xt-x0)/sint=v. The reason why not directly using net_output_B_C_T_H_W is that we want to set up a wrapper so that any noise schedule can adapt to the TrigFlow schedule, as stated in Section 3.1 of the paper.
2. No. If using the global velocity instead of the teacher signal, it is no longer distillation. Intuitively, this will make the training more sensitive to the data quality (in distillation we can use pure synthetic data), and the supervision signal is less clean than using a teacher. I know that works like MeanFlow wants to train from scratch, but according to our knowledge, stable teacher signals are essential for large-scale tasks like 14B video model.

ohhh，thank you for your reply !! noise_pred = ( xt - cos(t)*x0 ) / sin(t), and the velocity is velocity_pred = ( cos(t) * xt - x0 ) / sin(t), I misread the function orz.....