Further information on policy iteration step and batch size

[conorfhayes]

Hi ,

Thanks for sharing this awesome paper. I have one question on your work.

In each graph, you have measured performance with respect to a policy iteration step. How is this defined?

I am confused because this seems like each iteration step corresponds to a backpropagation step, is this correct? It is further confusing as you do not list the batch size used in the paper. I have followed your shared config files, which state a batch size of 128, is this correct? But then the config files use num_prompt_epoch = 200. Are you training the models for 200 epochs on a batch size of 128?

I am trying to reproduce your training setup. I am getting similar results, however I am finding it hard to understand how to measure performance over time. Any help on this would be greatly appreciated!

[lkevinzc]

Hi @conorfhayes , thanks for your interest!

"policy iteration step" means the number of times we sync the learner's weights to the actors, which counts as 1 "policy iteration".

"num_prompt_epoch" should just be a maximum number of epochs we iterate over the prompt set, but we may early stop the experiments.

"batch_size=128" is exactly our gradient batch size.

Overall, the flow is like this:

1. we sample 128 questions
2. we sample 8 responses per question, hence we get 1024 (x,y) samples
3. we use batch_size=128 to learn these 1024 samples; this takes 8 gradient steps on the learner side
4. after 8 gradient steps we sync the learner's weights to actors; this amounts to one policy iteration step, which is the x-label of our figures

Hope this clarifies and welcome any further questions!

[conorfhayes]

Thank you for the quick response.

I see, so technically 1 policy iteration step is 8 gradient steps. This is very helpful.

Would it be possible to share the reward curves for your training of the largest Qwen model? I understand if this isn't possible.

Given your reward function is either a 1 or 0, what reward value did the models typically converge to when early stopping?

Thanks again!

[lkevinzc]

Yea sure, so here is the reward curve of the largest Qwen model I have for this work: Dr. GRPO on Qwen-2.5-Math-7b model with math-8k dataset:

Hope this helps!

[conorfhayes]

OK, this is great. Thank you!

[conorfhayes]

I have one more follow up question. What is the reward function?

Are you giving a reward for correct formatting? or is it just 0/1 for incorrect/correct? From here: https://github.com/sail-sg/understand-r1-zero/blob/main/understand_r1_zero/math_grader.py it appears you at least tried thought about formatting rewards. Did you try any experiments with formatting rewards?

[lkevinzc]

I tried format reward at the very early stage, and chose to just 0/1 reward for correctness until now. All experiments in the paper used the simple 0/1 reward.

[conorfhayes]

Thank you. For figure 5, do you fine tune Qwen2.5-1.5B-Instruct or Qwen2.5-MATH-1.5B? I cannot find the config for this experiment in the repo. Thank you again!

[lkevinzc]

Hi @conorfhayes , in all the experiments we use the base models. So it should be "Qwen2.5-MATH-1.5B". https://github.com/sail-sg/understand-r1-zero/blob/main/examples/qwen2.5-math-1.5b.sh this should serve as a reference for the training config.

[conorfhayes]

OK thank you. For fig.5 In the paper you state "we adopt the Qwen2.5-1.5B base model and the R1 template". So here you actually mean Qwen2.5-MATH-1.5B? Sorry for the specifics, I am trying to reproduce the results and currently I am only getting a training reward of 0.4, and want to ensure I am using the correct model here.

Do you suspect versioning makes a difference for the grader?

[lkevinzc]

I actually mean Qwen2.5-1.5B. My previous run is like this:

Perhaps you could paste your screenshot here and your command as well if possible. I can help with your reproduction.

[conorfhayes]

Thank you. Interesting, when I add Qwen/Qwen2.5-1.5B to the run script, it crashes when training. It will do an eval step, then crash saying the following:

[learner_0_0/0] [rank0]: Traceback (most recent call last):
[learner_0_0/0] [rank0]: File "/home/conor/oat-lm/oat-rl/lib/python3.10/site-packages/launchpad/nodes/python/process_entry.py", line 129, in
[learner_0_0/0] [rank0]: app.run(main)
[learner_0_0/0] [rank0]: File "/home/conor/oat-lm/oat-rl/lib/python3.10/site-packages/absl/app.py", line 316, in run
[learner_0_0/0] [rank0]: _run_main(main, args)
[learner_0_0/0] [rank0]: File "/home/conor/oat-lm/oat-rl/lib/python3.10/site-packages/absl/app.py", line 261, in _run_main
[learner_0_0/0] [rank0]: sys.exit(main(argv))
[learner_0_0/0] [rank0]: File "/home/conor/oat-lm/oat-rl/lib/python3.10/site-packages/launchpad/nodes/python/process_entry.py", line 124, in main
[learner_0_0/0] [rank0]: functionstask_id
[learner_0_0/0] [rank0]: File "/home/conor/oat-lm/oat-rl/lib/python3.10/site-packages/launchpad/nodes/python/node.py", line 75, in _construct_function
[learner_0_0/0] [rank0]: return functools.partial(self._function, *args, **kwargs)()
[learner_0_0/0] [rank0]: File "/home/conor/oat-lm/oat-rl/lib/python3.10/site-packages/launchpad/nodes/python/node.py", line 207, in run
[learner_0_0/0] [rank0]: instance.run()
[learner_0_0/0] [rank0]: File "/home/conor/oat-lm/oat-rl/lib/python3.10/site-packages/oat/learners/base.py", line 409, in run
[learner_0_0/0] [rank0]: self.eval_and_log(train_info, eval=True, save=True)
[learner_0_0/0] [rank0]: UnboundLocalError: local variable 'train_info' referenced before assignment

This is an oat issue, but I am following the exact install instructions on this repo. Also, the code runs for the Qwen Math models. Is there any reason why Qwen2.5-1.5B doesnt run with this code?

I was able to reproduce the learning curves for the Math 1.5B math model, getting ~0.6, but the code will not run for 1.5B Qwen.

Again, I am sorry for the specifics. To be totally clear, I have an algorithm I am finetuning Qwen2.5-1.5B and Qwen2.5-Math-1.5B with and I am trying to use your models as strong baselines. Hence I am executing your scripts to generate the results for comparison with my method. I do not see Qwen2.5-1.5B's training script in your repo, and the Dr.GRPO fine-tuned version of the the model is not on huggingface either: https://huggingface.co/collections/sail/oat-zero-understanding-r1-zero-like-training. The only model of that size that has a training script and fine-tuned model available is Qwen2.5-Math-1.5B.

Thanks again for your engagement here!

[conorfhayes]

Also, how do you sample during evaluation for Figure 5? Is it pass@8 or something else?

[lkevinzc]

Hi @conorfhayes , I am sorry for missing your previous comments here. For Fig.5 we used pass@1 and used greedy decoding for evaluation (because at that time we followed previously works). Have you solved the training bug? I would love to help if not :)

[conorfhayes]

No worries! Yes we solved our training issue. Currently benchmarking our trained model against the Oat RL models. Thank you for your help!

[lkevinzc]

Cool! Looking forward to your results!