# Tokenize a large corpus

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenize-a-large-corpus%2F99348%3E%5C
**Category:** Intermediate\
**Created:** [July 25, 2024, 9:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenize-a-large-corpus%2F99348 "2024-07-25T09:55:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![fabiancpl](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fabiancpl/32/29707_2.png) [@fabiancpl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffabiancpl)%5C
**Post date:** [July 25, 2024, 9:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenize-a-large-corpus%2F99348%2F1 "2024-07-25T09:55:00Z")

</div>

Hi guys,

How can I efficiently tokenize a corpus of 340M of sentences for pre-training models like BERT or GPT-2? Following the standar documentation, the process will take around 20 hours to finish.

Thanks.