# GPT2 long text approach

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-long-text-approach%2F28165%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [December 20, 2022, 10:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-long-text-approach%2F28165 "2022-12-20T10:54:30Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![misterkilgore](https://avatars.discourse-cdn.com/v4/letter/m/ac91a4/32.png) [@misterkilgore](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmisterkilgore)%5C
**Post date:** [December 20, 2022, 10:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-long-text-approach%2F28165%2F1 "2022-12-20T10:54:30Z")

</div>

Hi everyone. I was wondering, what’s the best way to approach a long text for gpt2 training?

Suppose the text is 10000 tokens long and the max length for the tokenizer is 2048.

It’s better to split the corpus in 5 sequential samples (first sample from token 0 to 2047, second sample from token 2048 to 4095…) or in 7952 overlapping samples (first sample form token 0 to token 2047, second sample from token 1 to 2048… last sample from token 7952 to token 10000)?

I’ve seen people using both approaches.