Discrepancy Between Scaling Factors in Paper and Code, and Typical Loss Magnitudes

[Peterande]

@zhengkw18 @qsh-zh Excellent work! Could you clarify why the scaling factor reported in the paper is 1.0 and 0.01, whereas in the code the scales for SCM and DMD are set to 100.0 and 1.0, respectively? Additionally, what are the typical magnitudes of the various loss terms during training for text-to-image (T2I) and text-to-video (T2V) models?

[zhengkw18]

Great observation!

It is known that when using the Adam(W) optimizer, the absolute loss scales do not matter because Adam conducts self-normalization to the gradient. Only the relative scales matter. Using (100, 1) instead of (1, 0.01) is inherited from previous works that operate on FP16, which may better prevent numerical underflow. In our BF16 setting, (100, 1) should behave exactly the same as (1, 0.01).

The sCM loss is always near 1*100 due to tangent normalization. DMD loss could be large in scale, for example, 1M. But both loss terms contain normalization/stop-gradient operators, so the absolute value does not mean much.

[Peterande]

@zhengkw18 Thank you for your fast reply! During the generator update, the loss reduction uses sum instead of mean, which is also why the DMD loss appears so large. Since the DMD loss is backpropagated together with the sCM loss, wouldn't this overwhelm or dilute the contribution of the sCM loss?

[Peterande]

Thank you for your fast reply! During the generator update, the loss reduction uses sum instead of mean, which is also why the DMD loss appears so large. Since the DMD loss is backpropagated together with the sCM loss, wouldn't this overwhelm or dilute the contribution of the sCM loss?

I noticed that the sCM loss also appears to use sum reduction—does this mean it becomes extremely large as well, roughly on the order of C × T × H × W?

[zhengkw18]

In our experiments it will not. We find sum+relative weight 0.01 just generalizes. If using mean for DMD, the quality will be bad on video. Intuitively, DMD provides a large correction force to sCM. But as long as the sCM loss exists (which is mode-covering), it will enforce large penalties if the student does not cover any training data and help maintain diversity. As we said in the paper, the two loss terms operate on different generation paths and do not conflict.

For sCM, using sum will result in a final loss value of 1 because of tangent normalization.

These settings may not fit people's intuition, but are empirically verified by us with over 100k GPU hours of experiments. All we can tell is the final conclusion that sum+relative weight 0.01 generalizes for T2I and T2V. The concrete analyses of training dynamics and explanations are left for future work.

[Peterande]

In our experiments it will not. We find sum+relative weight 0.01 just generalizes. If using mean for DMD, the quality will be bad on video. Intuitively, DMD provides a large correction force to sCM. But as long as the sCM loss exists (which is mode-covering), it will enforce large penalties if the student does not cover any training data and help maintain diversity. As we said in the paper, the two loss terms operate on different generation paths and do not conflict.

For sCM, using sum will result in a final loss value of 1 because of tangent normalization.

These settings may not fit people's intuition, but are empirically verified by us with over 100k GPU hours of experiments. All we can tell is the final conclusion that sum+relative weight 0.01 generalizes for T2I and T2V. The concrete analyses of training dynamics and explanations are left for future work.

This aligns well with the gradient direction of DMD—particularly, its magnitude needs to be weighted according to the number of frames, so using sum is indeed reasonable. Thank you very much for your clarification!

[qsh-zh]

Using (100, 1) instead of (1, 0.01) is inherited from previous works that operate on FP16, which may better prevent numerical underflow. In our BF16 setting, (100, 1) should behave exactly the same as (1, 0.01).

Ideally with perfect numerical accuracy, (100, 1) should behave exactly the same as (1, 0.01). In practice. manually loss scaling may still matters thanks to finite bit float number representation and eps in adam.

[Peterande]

Using (100, 1) instead of (1, 0.01) is inherited from previous works that operate on FP16, which may better prevent numerical underflow. In our BF16 setting, (100, 1) should behave exactly the same as (1, 0.01).

Ideally with perfect numerical accuracy, (100, 1) should behave exactly the same as (1, 0.01). In practice. manually loss scaling may still matters thanks to finite bit float number representation and eps in adam.

That sounds very reasonable—thank you! I’ll run some related validations on my end. 😄

[Peterande]

@zhengkw18 @qsh-zh Thank you for your previous reply. I have a new question: Has there been an ablation study on the scaling coefficient of the CM loss? Specifically, on T2I (text-to-image) and T2V (text-to-video) tasks, how would the results be affected if we gradually increased the SCM loss weight—from completely disabling it to scaling it up until it reaches the same magnitude as the DMD loss?

Currently, the DMD loss uses sum reduction, effectively amplifying it by a factor of C×T×H×W compared to using mean reduction. Meanwhile, the SCM loss sums to approximately 1; even if multiplied by 100.0, its magnitude remains tens of thousands of times smaller than that of the DMD loss. Although their gradient directions differ significantly, since both losses are optimized jointly with the same optimizer, such a massive scale discrepancy likely causes the gradients from the CM loss to be completely overwhelmed (or "diluted") by those from the DMD loss.

[zhengkw18]

We need to clarify the following things:

(1) There is a clear effect of the sCM loss. Please refer to our comparison with sCM and DMD on Wan 1.3B. rCM exhibits exactly the same motion patterns as sCM. Your claim of such a massive scale discrepancy likely causes the gradients from the CM loss to be completely overwhelmed (or "diluted") by those from the DMD loss. is irresponsible. You cannot simply judge the effects of losses simply by their scale, given that both loss terms contain normalization/stop-gradient operators. You have no theoretical justifications or experimental verifications.

(2) We experimented with the relative loss weight 0.001 (10x smaller DMD) on Wan 1.3B, and found significant quality degradation. Even in this case, the DMD loss value is still significantly larger than sCM, but it is empirically proven that we still require larger DMD weight for better quality. Therefore, your judgments by simply looking at scales of losses do not hold.

(3) We can never conduct very fine-grained grid search for experiments of this scale. We use around 100 H100 GPUs for each run, which is expensive. Still, after dozens of coarse ablation runs, we manage to figure out a set of well-performed hyperparameters.

(4) It is normal that simply increasing the data dimension or task complexity can change the behavior of algorithms. For example, on ImageNet, sCM/MeanFlow performs reasonably well, and SiD is notably better than DMD, none of which are observed on T2I or T2V. In many cases, intuition is not reliable.

[Peterande]

We need to clarify the following things:

(1) There is a clear effect of the sCM loss. Please refer to our comparison with sCM and DMD on Wan 1.3B. rCM exhibits exactly the same motion patterns as sCM. Your claim of such a massive scale discrepancy likely causes the gradients from the CM loss to be completely overwhelmed (or "diluted") by those from the DMD loss. is irresponsible. You cannot simply judge the effects of losses simply by their scale, given that both loss terms contain normalization/stop-gradient operators. You have no theoretical justifications or experimental verifications.

(2) We experimented with the relative loss weight 0.001 (10x smaller DMD) on Wan 1.3B, and found significant quality degradation. Even in this case, the DMD loss value is still significantly larger than sCM, but it is empirically proven that we still require larger DMD weight for better quality. Therefore, your judgments by simply looking at scales of losses do not hold.

(3) We can never conduct very fine-grained grid search for experiments of this scale. We use around 100 H100 GPUs for each run, which is expensive. Still, after dozens of coarse ablation runs, we manage to figure out a set of well-performed hyperparameters.

(4) It is normal that simply increasing the data dimension or task complexity can change the behavior of algorithms. For example, on ImageNet, sCM/MeanFlow performs reasonably well, and SiD is notably better than DMD, none of which are observed on T2I or T2V. In many cases, intuition is not reliable.

@zhengkw18 I'm glad to hear that the sCM loss still has a significant impact on performance. From some gradient quantification analyses I've done, there is indeed still a noticeable gap between the gradients contributed by the sCM loss and the DMD loss. However, compared to the huge difference in their loss magnitudes, the discrepancy in their gradients is much smaller. This suggests that, despite its much smaller magnitude relative to the DMD loss, the sCM loss still provides non-negligible gradient contributions. Your response has likely clarified many concerns that first-time readers might have had. Thank you very much!

Additionally, I’d like to ask you another question: suppose we design a third loss that is entirely unrelated to the current visual generation task—for example, in a multimodal setting where our model can also generate audio. In the context of CM acceleration, this new loss is combined with the CM loss at a 1:1 ratio; similarly, in DMD acceleration, it’s combined with the DMD loss at a 1:1 ratio as well. However, given the current large magnitude discrepancy between rCM’s CM loss and DMD loss, how should I scale this third loss appropriately?

[zhengkw18]

@zhengkw18 I'm glad to hear that the sCM loss still has a significant impact on performance. From some gradient quantification analyses I've done, there is indeed still a noticeable gap between the gradients contributed by the sCM loss and the DMD loss. However, compared to the huge difference in their loss magnitudes, the discrepancy in their gradients is much smaller. This suggests that, despite its much smaller magnitude relative to the DMD loss, the sCM loss still provides non-negligible gradient contributions. Your response has likely clarified many concerns that first-time readers might have had. Thank you very much!

Additionally, I’d like to ask you another question: suppose we design a third loss that is entirely unrelated to the current visual generation task—for example, in a multimodal setting where our model can also generate audio. In the context of CM acceleration, this new loss is combined with the CM loss at a 1:1 ratio; similarly, in DMD acceleration, it’s combined with the DMD loss at a 1:1 ratio as well. However, given the current large magnitude discrepancy between rCM’s CM loss and DMD loss, how should I scale this third loss appropriately?

Yes, I remember regarding the gradient norm, DMD may be around 10x larger than sCM on video. But if we decrease the DMD weight by 10x, the quality will be bad. A large DMD correction force seems necessary for good quality.

Our finding of the relative loss weight is only verified on single modality and joint training. If you use them separately or incorporate other loss terms, I cannot say exactly how the weight should be set. The best way is to conduct some hyperparameter search and draw conclusions from experiments. If you use sCM and DMD jointly, their relative weight can be determined by observing the quality and diversity.

[Peterande]

@zhengkw18 I'm glad to hear that the sCM loss still has a significant impact on performance. From some gradient quantification analyses I've done, there is indeed still a noticeable gap between the gradients contributed by the sCM loss and the DMD loss. However, compared to the huge difference in their loss magnitudes, the discrepancy in their gradients is much smaller. This suggests that, despite its much smaller magnitude relative to the DMD loss, the sCM loss still provides non-negligible gradient contributions. Your response has likely clarified many concerns that first-time readers might have had. Thank you very much!
Additionally, I’d like to ask you another question: suppose we design a third loss that is entirely unrelated to the current visual generation task—for example, in a multimodal setting where our model can also generate audio. In the context of CM acceleration, this new loss is combined with the CM loss at a 1:1 ratio; similarly, in DMD acceleration, it’s combined with the DMD loss at a 1:1 ratio as well. However, given the current large magnitude discrepancy between rCM’s CM loss and DMD loss, how should I scale this third loss appropriately?

Yes, I remember regarding the gradient norm, DMD may be around 10x larger than sCM on video. But if we decrease the DMD weight by 10x, the quality will be bad. A large DMD correction force seems necessary for good quality.

Our finding of the relative loss weight is only verified on single modality and joint training. If you use them separately or incorporate other loss terms, I cannot say exactly how the weight should be set. The best way is to conduct some hyperparameter search and draw conclusions from experiments. If you use sCM and DMD jointly, their relative weight can be determined by observing the quality and diversity.

Understood. Thank you very much for your patient reply!

[zhengkw18]

@Peterande We conduct an ablation study on $\lambda$ recently, for your reference