# Why does tokenization take so long?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-does-tokenization-take-so-long%2F102098%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [August 13, 2024, 4:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-does-tokenization-take-so-long%2F102098 "2024-08-13T04:13:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jason-Hwang1](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@Jason-Hwang1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJason-Hwang1)%5C
**Post date:** [August 13, 2024, 4:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-does-tokenization-take-so-long%2F102098%2F1 "2024-08-13T04:13:42Z")

</div>

I found that tokenization steps before training takes longer time than training itself.

Yes, Training involves GPU, but I thought tokenization is not that compute-intensive (just splitting the sentence into words and mapping words to ID and other substeps…) so I thought it should be bounded to the IO time for loading the raw dataset. But it takes much time to tokenize subset of Wikipedia for more than 2 hours.

Can someone give me the reason why tokenization steps takes so long?

The code that I used for tokenization is as below. I also tried multiprocessing but it doesn’t make meaningful difference.

```auto
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

    def encode_example(example):
        return tokenizer(example['text'], truncation=True, padding='max_length', max_length=64) # Reduce max_length to save memory

    # Tokenize dataset
    dataset = dataset.map(encode_example, batched=True)

```

---

<div class="post-metadata">

**Author:** ![gweidart](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gweidart/32/43593_2.png) [@gweidart](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgweidart)%5C
**Post date:** [March 19, 2025, 9:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-does-tokenization-take-so-long%2F102098%2F2 "2025-03-19T09:12:31Z")

</div>

lol. That depends on the tokenizer you’re using.

Check out ‘rs-bpe’ on PyPI / GitHub. It currently outperforms both tiktoken and tokenizers.