A question on the Response-level length bias of GRPO

[ZhaoqiZachYang]

In your paper, you mentioned that GRPO suffers from the response-level length bias because of the factor 1/|o|. However, I found that factor 1/|o| is multiplied on a summation of t=1,2,...,|o| items. As a result, for positive advantages, shorter responses should have a small absolute value of summation, together with a large factor 1/|o|. For negative advantages, longer responses should have a large absolute value of summation, together with a small factor 1/|o|. The normalization with 1/|o| actually mitigates the response-level length bias rather than causes it. Where am I wrong in this argument?

[lkevinzc]

@ZhaoqiZachYang , please have a look on the discussion in these two issues:

• Question about Response-level length bias #10
• confused about Response-level length bias #11

If you still have further questions, feel free to add comments.