# RoBERTa MLM fine-tuning

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Froberta-mlm-fine-tuning%2F1330%3E%5C
**Category:** Beginners\
**Created:** [October 1, 2020, 10:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Froberta-mlm-fine-tuning%2F1330 "2020-10-01T10:16:29Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![stefan-jo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stefan-jo/32/64_2.png) [@stefan-jo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstefan-jo)%5C
**Post date:** [October 1, 2020, 10:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Froberta-mlm-fine-tuning%2F1330%2F1 "2020-10-01T10:16:29Z")

</div>

Hello,

I want to fine-tune RoBERTa for MLM on a dataset of about 200k texts. The texts are reviews from online forums ranging from basic conversations to technical descriptions with a very specific vocabulary.

I have two questions regarding data preparation:

- Can I simply use `RobertaTokenizer.from_pretrained("roberta-base")` even if the vocabulary of my fine-tuning corpus might differ significantly from the pre-training corpus? Or is there a way to “adjust” the tokenizer to the new data?
- Each review comes with the title of the thread it has been posted in. From earlier experiments I know that concatenating titles and texts (and adding a special separator token between them) improves model performance for classification. However, I am wondering how this should be handled during language model fine-tuning? Since some threads contain hundreds of reviews, it seems wasteful for the language model to predict on the same title over and over again.

---

<div class="post-metadata">

**Author:** ![Pataleros](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pataleros/32/3303_2.png) [@Pataleros](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPataleros)%5C
**Post date:** [November 24, 2021, 4:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Froberta-mlm-fine-tuning%2F1330%2F2 "2021-11-24T16:18:15Z")

</div>

Hello there,

I am currently trying to do the same : fine-tune Roberta on a very specific vocabulary of mine (let’s say : biology stuff).

About your first question, you should at least add some new words, specific to your vocabulary, in the Tokenizer vocabulary. See this discussion : [how can i finetune BertTokenizer? · Issue #2691 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/2691#issuecomment-907633931)

Considering the MLM training, what class did you use exactly ? I am looking for more info online, found this ([NLP-with-Deep-Learning/fine\_tuning\_bert\_with\_MLM.ipynb at master · yash-007/NLP-with-Deep-Learning · GitHub](https://github.com/yash-007/NLP-with-Deep-Learning/blob/master/BERT/Fine%20Tune%20BERT/fine_tuning_bert_with_MLM.ipynb)) but wonder how this would work for Roberta.

Thanks