# Transformers Tokenizer on GPU?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-on-gpu%2F2850%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 16, 2020, 2:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-on-gpu%2F2850 "2020-12-16T14:39:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaecha](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jaecha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjaecha)%5C
**Post date:** [December 16, 2020, 2:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-on-gpu%2F2850%2F1 "2020-12-16T14:39:53Z")

</div>

Hi, I am finding the tokenizing takes long time when I have large text data. There may be some documentation about this somewhere, but I could not find any that address how to use multiple GPUs to process the tokenization. Any help will be much appreciated. Thanks!

---

<div class="post-metadata">

**Author:** ![clem](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/clem/32/2838_2.png) [@clem](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclem)%5C
**Post date:** [December 17, 2020, 3:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-on-gpu%2F2850%2F2 "2020-12-17T03:29:49Z")

</div>

@Narsil might help be able to help here

---

<div class="post-metadata">

**Author:** ![Narsil](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/narsil/32/40029_2.png) [@Narsil](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNarsil)%5C
**Post date:** [December 17, 2020, 8:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-on-gpu%2F2850%2F3 "2020-12-17T08:39:49Z")

</div>

Hi @jaecha.

Tokenization does not happen on GPU (and won’t anytime soon). If you can show your tokenizer config that could help understand why it takes a long time ? tokenizers version, what kind of model and maybe how large your data ?  
If you see a warning about TOKENIZERS\_PARALLELISM in your console:

- You use multiple threads (like with `DataLoader`) then it’s better to create a tokenizer instance on each thread rather than before the fork otherwise we can’t use multiple cores (because of GIL)

Having a good pre\_tokenizer is important (usually `Whitespace` splitting for languages that allow it) at least.

You can find more information about your options here: [https://huggingface.co/docs/tokenizers/python/latest/pipeline.html#pre-tokenization](https://huggingface.co/docs/tokenizers/python/latest/pipeline.html#pre-tokenization)

Does that help?

Cheers  
Nicolas

---

<div class="post-metadata">

**Author:** ![jaecha](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jaecha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjaecha)%5C
**Post date:** [December 17, 2020, 2:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-on-gpu%2F2850%2F4 "2020-12-17T14:28:16Z")

</div>

Thank you for your replies! I think I know what to do next. Thanks!