Skip to content

Magi_attention experiments#1

Open
hanwen-sun wants to merge 15 commits intov0.11.0from
magi_attention
Open

Magi_attention experiments#1
hanwen-sun wants to merge 15 commits intov0.11.0from
magi_attention

Conversation

@hanwen-sun
Copy link
Copy Markdown
Collaborator

What this pr do:

  • Intergrate MagiAttention with Megatron.
  • Experiments for training llama-1b from scratch with MagiAttention.
    • code are available in ./magiattention_example, you can refer to the ./magiattention_example/README.md for more information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants