Online Training #1341
Draft
Online Training #1341
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…els like llm reward model
* Add Skywork Reward Model * reorder * working checkpoint with skywork * unify VllmReward class * cleanup wrap text * actually not quite working. needs dp gather and scatter * add gather scatter for vllm rm * working checkpoint * working checkpoint * updates * Instantiate metric recorders on rank 0 only (#1072) * fix gather bug * cleanup * comment * add grpo (runnable, but not increasing rewards yet * log outputs * cleannup * rename * merge * fixes * testing online dpo after merge * bug fix * fixing merge errors * fix bugs * merged with online_training * update grpo * cleanup * fix grpo bug * cleanup * cleanup * isort/black * move vllm generate_rewards to its own function * refactor how reward models use prompt_batch * remove breakpoint * working chkpt * remove irrelevant skywork stuff --------- Co-authored-by: jacklanchantin user <[email protected]> Co-authored-by: Can Balioglu <[email protected]>
* Pairwise J1 prompt * Adding Pairwise-J1 support * Minor changes * Simplifying * Add logging back in * Add generation prompt. * removing debug statements * More logging for scores out of range * Adding reward name as a reward class attribute * Cleaning up generative judges with extractor classes * Typing * logger label --------- Co-authored-by: swarna <[email protected]> Co-authored-by: Ilia Kulikov <[email protected]>
* add gener_verifier for hf backend * working? with HF background * fix bug * comments * working with vllm now * add generative_judge * merge * remove GeneralVerifier * move wrap_text to extractor * rename wrap_text --> format_prompt * string * remove comment * comment * comments * comment
* bug fix * new parser * fix * fix parsing * dont parse ref answer * remove unused
* add new args for reward_handler.create * add qwen25_3b * increase max seq len * revert yaml
* add new args for reward_handler.create * add qwen25_3b * increase max seq len * revert online_dpo
Co-authored-by: Ilia Kulikov <[email protected]>
Co-authored-by: Ilia Kulikov <[email protected]>
…ard is binary (#1262) * parallel vllm worker init, online dpo ref score fix when bs>1 and reward is binary * comments clean * addressing feedback --------- Co-authored-by: Ilia Kulikov <[email protected]>
meta-cla
bot
added
the
CLA Signed
* drgrpo * get vllm logps * Update _wandb.py * remove beta check * format * revert * add importance sampling correction * dont run ref model forward if beta==0 * add tis ratio clamp = 2 * clean up * configs * clean up * default * var name * var name * only use tis_imp_ratio_cap * revert unrelated files * clean up * fix type hint * black/isort * Allow batched inputs for get_vllm_logprobs * allow batch_sz > 1 * Modify condition for reference log probabilities * fix batch>1, microbatching --------- Co-authored-by: Jack Lanchantin <[email protected]>
…dd a comment for calling newer vllm generate function (#1396) * set VLLM_ALLOW_INSECURE_SERIALIZATION=1 for newer vllm versions * update generate function to align with newer vllm version
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do? Please describe:
A summary of the change or the issue that is fixed.
Fixes #{issue number}
Does your PR introduce any breaking changes? If yes, please list them:
List of all backwards-incompatible changes.
Check list: