# Speeding up Tokenization on large text corpus

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspeeding-up-tokenization-on-large-text-corpus%2F23610%3E%5C
**Category:** 🤗Transformers\
**Created:** [September 26, 2022, 5:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspeeding-up-tokenization-on-large-text-corpus%2F23610 "2022-09-26T05:39:15Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jaidev](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jaidev](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJaidev)%5C
**Post date:** [September 26, 2022, 5:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspeeding-up-tokenization-on-large-text-corpus%2F23610%2F1 "2022-09-26T05:39:15Z")

</div>

I have a similar issue, using a pretrained WordPiece tokenizer on a large corpus of text takes several hours. I’m doing:

tokenizer = AutoTokenizer.from\_pretrained(“distilbert-base-uncased”)  
train\_tokenized\_encodings = tokenizer(df[df.split==‘train’].text.tolist(), truncation=True, padding=True, max\_length=MAX\_LENGTH)

Any suggestions for speed up?

Is there a way to parallelize this? (Or does the above automatically use multiple workers?)