# Train GPT2 on wikitext from scratch
**URL:**
**Author:**  [@Hamidreza](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHamidreza)%5C
**Post date:** [April 2, 2021, 5:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt2-on-wikitext-from-scratch%2F5276%2F1 "2021-04-02T17:45:39Z")
Hello everyone,
I would like to train GPT2 on wikitext from scratch (not fine-tune pre-trained model). I launched the following script in [this](https://github.com/huggingface/transformers/tree/master/examples/language-modeling) folder.
python run\_clm.py
–model\_type gpt2
–tokenizer\_name gpt2
–block\_size 256
–dataset\_name wikitext
–dataset\_config\_name wikitext-2-raw-v1
–do\_train
–do\_eval
–overwrite\_output\_dir
–num\_train\_epochs 1
–output\_dir /tmp/test-clm
Now I have two questions:
1- I was wondering if what I did is indeed a correct approach to train GPT2 from scratch?
2- I would like to know what hyperparameters I shoud use for this task? ( as far as I can tell, the suggested hyperparameters in existing examples in huggingface repo are for fine-tuning pre-trainned model)
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [April 5, 2021, 1:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt2-on-wikitext-from-scratch%2F5276%2F2 "2021-04-05T13:17:55Z")
I can confirm the command is correct if you want to train from scratch. As for hyperparameters, you will need to tune them a bit, but the defaults should not be too bad.
---
**Author:**  [@Hamidreza](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHamidreza)%5C
**Post date:** [April 5, 2021, 3:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt2-on-wikitext-from-scratch%2F5276%2F3 "2021-04-05T15:31:45Z")
Thanks for your time and reply. I was also wondering how many epochs you suggest for training from scratch?
---
**Author:**  [@Hamidreza](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHamidreza)%5C
**Post date:** [April 13, 2021, 6:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt2-on-wikitext-from-scratch%2F5276%2F4 "2021-04-13T18:48:26Z")
I was wondering if there is any update in this thread? I like to know how many epochs is suggested for training GPT-2 on wikitext from scratch?
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [September 12, 2021, 1:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt2-on-wikitext-from-scratch%2F5276%2F5 "2021-09-12T01:57:45Z")
I have a custom vocab and want to use the BertWordpieceTokenizer for this reason. How would I do that with this new run\_clm.py script?
---
**Author:**  [@RylanSchaeffer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRylanSchaeffer)%5C
**Post date:** [October 25, 2021, 4:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt2-on-wikitext-from-scratch%2F5276%2F6 "2021-10-25T04:08:32Z")
@Hamidreza did you find out how long GPT-2 takes to train from scratch?