# How to train a transformer from scratch

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-a-transformer-from-scratch%2F84901%3E%5C
**Category:** Beginners\
**Created:** [May 2, 2024, 9:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-a-transformer-from-scratch%2F84901 "2024-05-02T21:41:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![IParraMartin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iparramartin/32/26521_2.png) [@IParraMartin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIParraMartin)%5C
**Post date:** [May 2, 2024, 9:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-a-transformer-from-scratch%2F84901%2F1 "2024-05-02T21:41:11Z")

</div>

Hi all!

I am trying to train a transformer from scratch using either HuggingFace or the NanoGPT repository. The task will be to generate questions from declarative sentences (e.g., _the dog does bark → Does the dog bark?_).

My problems are:

1. Where can I find a good and straightforward way to train a transformer from scratch?
2. How do I structure the dataset? I was thinking of something like this:  
_The dog does bark \n Does the dog bark?[EOS]_

Thank you all!🤗

---

<div class="post-metadata">

**Author:** ![nimatov](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimatov/32/18720_2.png) [@nimatov](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnimatov)%5C
**Post date:** [May 2, 2024, 9:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-a-transformer-from-scratch%2F84901%2F2 "2024-05-02T21:59:14Z")

</div>

I found this one useful when I tried to understand how transformers work.  
[https://nlp.seas.harvard.edu/annotated-transformer/#encoder-and-decoder-stacks](https://nlp.seas.harvard.edu/annotated-transformer/#encoder-and-decoder-stacks)