# Preparing a nlp dataset for MLM
**URL:**
**Author:**  [@tillfurger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftillfurger)%5C
**Post date:** [February 15, 2021, 7:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreparing-a-nlp-dataset-for-mlm%2F3652%2F1 "2021-02-15T19:25:45Z")
Hi I’am trying to use nlp datasets to train a RoBERTa Model from scratch and I am not sure how to perpare the dataset to put it in the Trainer:
```
!pip install datasets
from datasets import load_dataset
dataset = load_dataset('wikicorpus', 'raw_en')
from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer,
mlm=True,
mlm_probability=0.15)
from transformers import Trainer, TrainingArguments
training_args = TrainingArguments(
output_dir="./",
overwrite_output_dir=True,
num_train_epochs=1,
per_gpu_train_batch_size=16,
save_steps=10_000,
save_total_limit=2)
trainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset=dataset)
```
How do I have to `dataset.set_format()` such that it only takes the `text` of the dataset, line-by-line?
Or what’s the proper way to prepare the dataset for MLM?
In the past I have been doing it with:
```
from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
tokenizer=tokenizer,
file_path="/dataset.txt"
)
```
which will be removed soon and does not support multiple txt files.
Thanks
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [February 16, 2021, 2:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreparing-a-nlp-dataset-for-mlm%2F3652%2F2 "2021-02-16T02:58:51Z")
You should have a look at the preprocessing done in the [run\_mlm example](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). There is also the [corresponding notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) that can help.
---
**Author:**  [@Lucas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLucas)%5C
**Post date:** [June 12, 2021, 7:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreparing-a-nlp-dataset-for-mlm%2F3652%2F3 "2021-06-12T07:25:22Z")
Dear there
The first link does not work. And the notebook you’ve mentioned does not even work in colab.
---
**Author:**  [@rashmi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frashmi)%5C
**Post date:** [June 12, 2021, 9:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreparing-a-nlp-dataset-for-mlm%2F3652%2F4 "2021-06-12T09:53:13Z")
This is the link [run\_mlm.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py)
I’m not sure about the notebook though.
---
**Author:**  [@viharshah85](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fviharshah85)%5C
**Post date:** [November 8, 2024, 6:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreparing-a-nlp-dataset-for-mlm%2F3652%2F5 "2024-11-08T06:48:08Z")
If you’re preparing an NLP dataset for a Masked Language Model (MLM), it’s important to have high-quality, diverse data to ensure the model can effectively understand and predict contextual language. For a comprehensive list of NLP datasets to help you get started, check out this blog: - [Top NLP Datasets to Supercharge Your Machine Learning Models](https://www.shaip.com/blog/15-nlp-dataset-for-nlp/) . These datasets offer a variety of text sources that can support a range of NLP tasks, including MLM training.