# Building a GPT2 dataset from long sequences

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuilding-a-gpt2-dataset-from-long-sequences%2F23241%3E%5C
**Category:** 🤗Datasets\
**Created:** [September 16, 2022, 10:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuilding-a-gpt2-dataset-from-long-sequences%2F23241 "2022-09-16T22:08:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdelas](https://avatars.discourse-cdn.com/v4/letter/m/b5ac83/32.png) [@mdelas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmdelas)%5C
**Post date:** [September 16, 2022, 10:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuilding-a-gpt2-dataset-from-long-sequences%2F23241%2F1 "2022-09-16T22:08:25Z")

</div>

Dear community,

I am trying to build a dataset for posterior GPT2 pretraining for a language model generator. I have very long examples and I was thinking to use the map method to cut them into 1024 sequence tokens using this function:

```auto
def chunk_examples(examples):
    chunks = []
    for sentence in examples["sentence1"]:
        chunks += [sentence[i:i + 50] for i in range(0, len(sentence), 50)]
    return {"chunks": chunks}

```

However, I am not sure which should be the best practice. Should I tokenize the texts before and then, map the function to the dataset to do this kind of “data augmentation” and have more examples of the suitable size (1024 tokens)?

I can find any examples of this kind of practice, neither not using the Dataset Hugging face’s package.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [September 19, 2022, 10:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuilding-a-gpt2-dataset-from-long-sequences%2F23241%2F2 "2022-09-19T10:54:20Z")

</div>

Hi ! You should tokenize before chunking the examples, otherwise you can’t control how many tokens are present in each chunked example.