# Map tokenization and posterior to smaller substrings

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-tokenization-and-posterior-to-smaller-substrings%2F23792%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [September 29, 2022, 5:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-tokenization-and-posterior-to-smaller-substrings%2F23792 "2022-09-29T17:38:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdelas](https://avatars.discourse-cdn.com/v4/letter/m/b5ac83/32.png) [@mdelas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmdelas)%5C
**Post date:** [September 29, 2022, 5:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-tokenization-and-posterior-to-smaller-substrings%2F23792%2F1 "2022-09-29T17:38:14Z")

</div>

Dear community,

My objective is to have sequences of 1024 tokens ready to GPT-2. As input data, I have DNA sequences of 2E5 length, and I have built a BPE tokenizer using this data.

I am trying to build a function which concatenates:

1. Tokenize the full sequences (2E5 max\_length)
2. Chunk data into 1024 sequence length and enlarge the number of rows of my Huggingface Dataset

For now I am trying to use this function, and map it posterior to the Dataset:

```auto
def tokenize_and_chunk(examples):
    chunks = []
    tokenized_inputs = fast_tokenizer(
        examples['text'],
        max_length=200000,
        truncation=True
    )

    for sentence in examples['text']:
        chunks += [sentence[i:i + 1024] for i in range(0, len(sentence), 1024)]

    return {"chunks": chunks} 

chunked_data = dna_data.map(tokenize_and_chunk, remove_columns=dna_data.column_names)

```

The problem is that I receive a list of lists just like this:  
[[‘C’,  
‘C’,  
‘A’,  
‘G’,  
…,  
],…,]

where this list is a list where for each example I have a inner-list with length of 91170 chars.

I can’t figure out how should I correctly map this function. Does someone have an idea of which should be the best practice for this?