# DataCollator uses Tokenizer while having BatchEncodings?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollator-uses-tokenizer-while-having-batchencodings%2F75350%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 29, 2024, 2:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollator-uses-tokenizer-while-having-batchencodings%2F75350 "2024-02-29T14:29:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![bengisucam](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bengisucam/32/21900_2.png) [@bengisucam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbengisucam)%5C
**Post date:** [February 29, 2024, 2:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatacollator-uses-tokenizer-while-having-batchencodings%2F75350%2F1 "2024-02-29T14:29:39Z")

</div>

I did not understand the intuition behind this: The DataCollatorForLanguageModeling expects tokenizer while it forces to operate on tokenized data (BatchEncodings).

I tried to provide text dataset into trainer while having datacollator, it gave me an error:  
transformers/tokenization\_utils\_base.py", line 3245, in pad  
raise ValueError(  
ValueError: You should supply an encoding or a list of encodings to this method that includes input\_ids, but you provided []