# Tokenizer progress bar

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-progress-bar%2F1147%3E%5C
**Category:** 🤗Transformers\
**Created:** [September 15, 2020, 3:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-progress-bar%2F1147 "2020-09-15T15:51:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![congcongwang](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/congcongwang/32/3282_2.png) [@congcongwang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcongcongwang)%5C
**Post date:** [September 15, 2020, 3:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-progress-bar%2F1147%2F1 "2020-09-15T15:51:29Z")

</div>

Hi Great Community,

Is it possible to have a progress bar to track the tokenisation process when calling the following method?

tokenizer(large\_batch, padding=True, truncation=True,max\_length=512)

---

<div class="post-metadata">

**Author:** ![lhallee](https://avatars.discourse-cdn.com/v4/letter/l/dfb087/32.png) [@lhallee](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhallee)%5C
**Post date:** [April 6, 2023, 7:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-progress-bar%2F1147%2F2 "2023-04-06T19:18:01Z")

</div>

I would also like a progress bar for tokenizing! Maybe a verbose setting? Did you ever hear back about this?

---

<div class="post-metadata">

**Author:** ![hughiephan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hughiephan/32/28723_2.png) [@hughiephan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhughiephan)%5C
**Post date:** [August 6, 2023, 1:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-progress-bar%2F1147%2F3 "2023-08-06T13:52:08Z")

</div>

tqdm library allows you to output a progress bar for the tokenizer.

```auto
from tqdm import tqdm
def tokenizer_with_progress(large_batch):
    tokenized_texts = []
    for text in tqdm(large_batch, desc="Tokenizing", unit="text"):
        tokenized_texts.append(tokenizer(text, padding=True, truncation=True, max_length=512))
    return tokenized_texts

train_encodings = tokenizer_with_progress(training_sentences)
test_encodings = tokenizer_with_progress(testing_sentences)

```