# Tokenizer to dataset to datacollator

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-to-dataset-to-datacollator%2F17097%3E%5C
**Category:** Beginners\
**Created:** [April 22, 2022, 7:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-to-dataset-to-datacollator%2F17097 "2022-04-22T19:57:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dashapyly](https://avatars.discourse-cdn.com/v4/letter/d/87869e/32.png) [@dashapyly](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdashapyly)%5C
**Post date:** [April 22, 2022, 7:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-to-dataset-to-datacollator%2F17097%2F1 "2022-04-22T19:57:15Z")

</div>

Hi, I’m having trouble finding how does one from the output of  
inputs = tokenizer([text], return\_tensors = ‘pt’)  
to a huggingface dataset. Every time a dataset is mentioned it’s being loaded with “load\_dataset”. I want to tokenize with line 2 and then create the dataset instead of using .map once I “load\_dataset” from parquets. Can I do that? Alternatively, if you could point me to how prepare the dataset for a MLM if it’s to be used by a DataCollator - that would help too. (ie how to properly make labels from input\_ids, am I right to leave input\_ids unmasked if it’s going into DataCollator, do I need to remove any columns, etc).  
I’m also not clear how to use DataCollator with MLM. I don’t want to pad to a max\_length, I want for padding to be handled separately for each batch to the “longest”. How do I do that? When I set pad\_to\_multiple\_of to 8, I still can an error from trying to run trainer.  
**ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.**  
Thanks a lot!  
PS Is there an English version of this notebook? [notebooks/training.ipynb at main · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/main/transformers_doc/training.ipynb)

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [April 28, 2022, 2:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-to-dataset-to-datacollator%2F17097%2F2 "2022-04-28T14:06:07Z")

</div>

Hi!

[This tutorial](https://huggingface.co/course/chapter7/3?fw=pt) explains how to prepare input data for MLM. Additionally, we have this code packaged in scripts [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) (`run_mlm.py` and `run_mlm_no_trainer.py`)

> PS Is there an English version of this notebook? [notebooks/training.ipynb at main · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/main/transformers_doc/training.ipynb)

This should be fixed now.