Garbled Outputs from vLLM Sampling Causing Instability in Preference Learning

[pspdada]

Hello,

I’ve encountered an issue while building my training dataset. When using vLLM to sample responses for certain math-related questions, the outputs are garbled (e.g., nonsensical or corrupted text). If these samples are used as negative examples during preference learning, the model assigns extremely low probabilities to generating these samples. This results in a very large log ratios (i.e., chosen_logps - rejected_logps), which makes the training process highly unstable.

I’m wondering if anyone else has encountered this issue and how it was resolved. Any insights or suggestions on mitigating this problem would be greatly appreciated.

[WeiXiongUST]

adding some fiterling process or using a stronger initial model?