# BERT pre-training run\_mlm\_flax.py questions

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-pre-training-run-mlm-flax-py-questions%2F11330%3E%5C
**Category:** Beginners\
**Created:** [November 3, 2021, 5:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-pre-training-run-mlm-flax-py-questions%2F11330 "2021-11-03T17:12:50Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcelb9](https://avatars.discourse-cdn.com/v4/letter/m/7ea924/32.png) [@marcelb9](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarcelb9)%5C
**Post date:** [November 3, 2021, 5:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-pre-training-run-mlm-flax-py-questions%2F11330%2F1 "2021-11-03T17:12:50Z")

</div>

Hey, I am setting up a pre-training enviroment for RoBERTa… I have a few questions.

1. I will use Wikipedia to pre-train RoBERTa. I noticed though the wiki dataset is cleaned, there are h2 headings, links, as well as other non-sentence things. Do I remove them beforehand to get a better result? I would say yes, but I can’t see anything like that in the [`run_mlm_flax.py`](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py) script.
2. I am currently at `line 437` in the aforementioned script. The next step would be tokenizing each wiki article. The script in `line 452` suggests that the `group_texts` function is the `"Main data processing function that will concatenate all texts from our dataset and generate chunks of max_seq_length."` My question is, am I not losing lots of the text here? How is RoBERTa trained, do I just continuously feed sequences of length max\_seq\_length? I thought the batches will be sentences. And if I really just feed these long sequences, doesn’t it make sense to split the wiki articles beforehand into chunks \< max\_seq\_length such that I don’t lose sentences - and just pad the rest?
3. Also, please, could you answer this question of mine: [python - NonMatchingSplitsSizesError when loading huggingface dataset - Stack Overflow](https://bit.ly/3CHX6qJ).
4. I may be encountering other issues soon, should I open a new thread or use this one in case I can’t resolve it myself / there is too much uncertainty?

Thanks.