Add batch invariant RL post #113
Merged
+89
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Bram Wasti <[email protected]>
Signed-off-by: Bram Wasti <[email protected]>
Downloaded GitHub-hosted images to local assets directory and updated all image references to use local paths. Converted standalone images to markdown syntax while keeping centered images as HTML img tags for proper rendering. Signed-off-by: Bram Wasti <[email protected]>
Added Jekyll frontmatter with layout, title, and author metadata to properly render the blog post. Signed-off-by: Bram Wasti <[email protected]>
Restored the original width and height attributes (340x130 and 480x355) for the two centered images to maintain their fixed sizing. Signed-off-by: Bram Wasti <[email protected]>
Added horizontal rule and italic formatting to the acknowledgements section for better visual separation and styling. Signed-off-by: Bram Wasti <[email protected]>
zhuohan123
reviewed
Nov 10, 2025
WoosukKwon
reviewed
Nov 10, 2025
|
|
||
| In the septillions of flops used to pre-train models, this mismatch between values has largely been avoidable. Pre-training typically runs at a fixed batch size which induces the same reduction kernels to be run - often side-stepping the issue entirely. | ||
|
|
||
| Reinforcement learning, on the other hand, seems to almost exclusively run different reduction algorithms due to its inference-heavy (and thus largely latency and memory-bound) nature. Kernels optimized for low-batch size inference typically run reductions all at once, whereas kernels for training models parallelize heavily to reuse data and amp up compute utilization. That means the generators and the trainers are typically running completely different kernels! |
Collaborator
There was a problem hiding this comment.
Kernels optimized for low-batch size inference typically run reductions all at once
I don't understand this part. Are you talking about reductions like in RMS norm?
Contributor
Author
There was a problem hiding this comment.
in the kernels, they don't tile. let me use the word "tile" for clarity
youkaichao
reviewed
Nov 11, 2025
youkaichao
reviewed
Nov 11, 2025
youkaichao
reviewed
Nov 11, 2025
Signed-off-by: Bram Wasti <[email protected]>
Signed-off-by: Bram Wasti <[email protected]>
Signed-off-by: Bram Wasti <[email protected]>
Signed-off-by: Bram Wasti <[email protected]>
Signed-off-by: Bram Wasti <[email protected]>
Added links to #sig-post-training and #sig-batch-invariant Slack channels in the blog post to invite readers to contribute to future developments. Signed-off-by: Bram Wasti <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As in title, text can be found in PR content