# DataCollatorWithPaddings without Tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollatorwithpaddings-without-tokenizer%2F11068%3E%5C
**Category:** Beginners\
**Created:** [October 25, 2021, 8:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollatorwithpaddings-without-tokenizer%2F11068 "2021-10-25T20:39:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![fredguth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fredguth/32/2843_2.png) [@fredguth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffredguth)%5C
**Post date:** [October 25, 2021, 8:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollatorwithpaddings-without-tokenizer%2F11068%2F1 "2021-10-25T20:39:31Z")

</div>

I want to fine-tune a model…  
`model = BertForTokenClassification.from_pretrained('monilouise/ner_pt_br'`  
with this dataset:  
`raw_datasets = load_dataset('lener_br')`

The r`aw_datasets` loaded are already tokenized and encoded. And I don’t know how it was tokenized. Now, I want to pad the inputs, but I don’t know how to use `DataCollatorWithPaddings` in this case.

I noticed that this dataset is similar to `wnut` dataset from the docs. Still, I can’t figure out what should I do.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 25, 2021, 8:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollatorwithpaddings-without-tokenizer%2F11068%2F2 "2021-10-25T20:41:41Z")

</div>

You can use the base BERT tokenizer I would say (since it’s a BERT model). Just make sure the pad token is compatible with what the model expects.

---

<div class="post-metadata">

**Author:** ![fredguth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fredguth/32/2843_2.png) [@fredguth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffredguth)%5C
**Post date:** [October 25, 2021, 8:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollatorwithpaddings-without-tokenizer%2F11068%2F3 "2021-10-25T20:44:43Z")

</div>

Is there a way to check this from the downloaded model? Or this is something I will find in the model card?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 25, 2021, 8:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollatorwithpaddings-without-tokenizer%2F11068%2F4 "2021-10-25T20:47:43Z")

</div>

Check the model config `pad_token_id` field.
