Skip to content

Conversation

tmm1
Copy link

@tmm1 tmm1 commented Oct 17, 2024

uses unsafe + std::mem::transmute to re-use encoder keys as decoder values and in the sorted_token_bytes list

this should be safe because all these objects share a lifetime inside CoreBPE

results in memory savings, and performance improvements in some scenarios

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant