# DataCollator vs. Tokenizers

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollator-vs-tokenizers%2F5897%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 1, 2021, 12:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollator-vs-tokenizers%2F5897 "2021-05-01T00:40:01Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [May 1, 2021, 11:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollator-vs-tokenizers%2F5897%2F2 "2021-05-01T11:15:10Z")

</div>

hey @hamel, welcome to the 🤗 forum!

you’re spot on about using data collators to do padding on-the-fly. to understand why this helps, consider the following scenarios:

1. use the tokenizer to pad each example in the dataset to the length of the longest example in the dataset
2. use the tokenizer and `DataCollatorWithPadding` ([docs](https://huggingface.co/transformers/master/main_classes/data_collator.html?highlight=datacollator#datacollatorwithpadding)) to pad each example in a batch to the length of the longest example in the batch

clearly, scenario 2 is more efficient, especially in cases where a few examples happen to be much longer than the median length and scenario 1 would introduce a lot of unnecessary padding.

btw under the hood, the data collators are fed to the `collate_fn` argument of pytorch’s `DataLoader`, see e.g. here: [transformers/trainer.py at 4e7bf94e7280d2b725ac4644dbe9808560afa5d8 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/4e7bf94e7280d2b725ac4644dbe9808560afa5d8/src/transformers/trainer.py#L617)

the pytorch [docs](https://pytorch.org/docs/stable/data.html#working-with-collate-fn) are not super informative on `collate_fn` itself, but you can find various discussion in their forums (e.g. [here](https://discuss.pytorch.org/t/how-to-create-batches-of-a-list-of-varying-dimension-tensors/50773/14))

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollator-vs-tokenizers%2F5897)._