# Training GPT-2 from scratch
**URL:**
**Author:**  [@miguelvictor](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmiguelvictor)%5C
**Post date:** [August 2, 2020, 7:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-gpt-2-from-scratch%2F562%2F1 "2020-08-02T07:22:58Z")
Hello!
I’m currently working on a toy project that uses GPT-2 (smallest variant but only 6 layers, from scratch) to predict next tokens in the context of programming languages. So my dataset are all source codes and I am using a custom tokenizer and i have the following questions:
1. If my sample is longer than 1024 tokens (supposing the model’s max length is 1024), is the past tokens automatically fed back to the model during training? or should I do it myself?
2. My custom tokenizer works well (in my opinion) but i want to use the huggingface API to take advantage of the “fast” tokenizers. How do I go about subclassing the Tokenizer class so that my tokenizer is compatible with huggingface’s tokenizer api ?
Thank you very much!!!
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [August 2, 2020, 2:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-gpt-2-from-scratch%2F562%2F2 "2020-08-02T14:31:33Z")
Hi @miguelvictor,
1. You can train you tokenizer using the`[tokenizers](https://github.com/huggingface/tokenizers) library. These are fast rust tokenizers with python API.
---
**Author:**  [@miguelvictor](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmiguelvictor)%5C
**Post date:** [August 3, 2020, 3:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-gpt-2-from-scratch%2F562%2F3 "2020-08-03T15:41:04Z")
Hello. Thank you for replying.
Do you have any ideas about my question#1? I tried looking at the source code and it seems no “past” is given to the model during training for longer sequences (or maybe I’m wrong)