Implementing a Transformer from scratch

[d-kleine]

Hi,

I would like to ask what do you think about adding a showcase for how to build, train and test a Transformer "from scratch" with pytorch, for instance for a translation task like english to french?

I have already conducted some research and have found following resources to that:
Attention is all you need: A Pytorch Implementation
Build your own Transformer from scratch using Pytorch
Transformers from Scratch in PyTorch
Transformers from scratch

From my point of view, it would be nice to see

• read in the real text datasets (english, french) + train-test-split
• text wrangling + preprocessing
• implementation of each step of the transformer in an ordered structure (positional enc, MHA/masked MHA, Encoder, Decoder)
• and finally training the transformer on real text data, and evaluating it on real text data

I believe this would be also be great to demonstrate post-LN (as in the original paper) versus pre-LN implementation (performance boost).

In my option, the first resource would be a good reference (but not properly maintained) for an end-to-end workflow implementation working with real text data, but in the simplicity of the 2nd and 3rd resource (those just use some artificially sampled random data points), all in one single notebook.

Kind regards,
Daniel

[rasbt]

Thanks for the suggestion! Coincidentally, I am already working on something like that -- although a bit more comprehensive and focused on decoder models. Thanks for suggesting, though!

[d-kleine]

Alright, got it - thank you though! Looking forward for the decoder models notebook then! 😃

[rasbt]

This is going to be a longer, separate project 😅
https://github.com/rasbt/LLMs-from-scratch

[d-kleine]

Great! 👍 If you need feedback at any stage of this project, I would be available to serve as a reviewer

[rasbt]

Thanks a lot for offering @d-kleine! Would be happy to have you on board. Will follow up with you via LinkedIn!