# Fine-tune BERT for Masked Language Modeling

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-bert-for-masked-language-modeling%2F1275%3E%5C
**Category:** 🤗Transformers\
**Created:** [September 28, 2020, 11:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-bert-for-masked-language-modeling%2F1275 "2020-09-28T11:53:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nes](https://avatars.discourse-cdn.com/v4/letter/n/3be4f8/32.png) [@nes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnes)%5C
**Post date:** [September 28, 2020, 11:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-bert-for-masked-language-modeling%2F1275%2F1 "2020-09-28T11:53:40Z")

</div>

Hello,

I have used a pre-trained BERT model using Hugging Transformers for a project. I would like to know how to “fine-tune” the BERT for Masked Language Modeling for a task like spelling correction. The links “[https://github.com/huggingface/transformers/tree/master/examples/lm\_finetuning](https://github.com/huggingface/transformers/tree/master/examples/lm_finetuning)” and “[https://github.com/huggingface/transformers/blob/master/examples/lm\_finetuning/pregenerate\_training\_data.py](https://github.com/huggingface/transformers/blob/master/examples/lm_finetuning/pregenerate_training_data.py)” are not found which seemed to be of great resource. As well as I would also like to know the dataset (like what kind of inputs and labels are to be given to the model) format that BERTForMaskedLM requires to be trained on. I would be grateful if anyone could help me in this regard.

Thanks,  
Nes

---

<div class="post-metadata">

**Author:** ![tillfurger](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@tillfurger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftillfurger)%5C
**Post date:** [January 5, 2021, 6:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-bert-for-masked-language-modeling%2F1275%2F2 "2021-01-05T18:20:46Z")

</div>

Interested in this too…

---

<div class="post-metadata">

**Author:** ![VP1](https://avatars.discourse-cdn.com/v4/letter/v/ecd19e/32.png) [@VP1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVP1)%5C
**Post date:** [January 22, 2021, 10:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-bert-for-masked-language-modeling%2F1275%2F3 "2021-01-22T10:20:21Z")

</div>

it seems “lm\_finetunin” script is not active.  
there is this:

> <https://github.com/huggingface/transformers/blob/master/examples/legacy/run_language_modeling.py>

---

<div class="post-metadata">

**Author:** ![ningmoufubi](https://avatars.discourse-cdn.com/v4/letter/n/f14d63/32.png) [@ningmoufubi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fningmoufubi)%5C
**Post date:** [January 25, 2021, 7:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-bert-for-masked-language-modeling%2F1275%2F4 "2021-01-25T07:37:26Z")

</div>

data\_text like this:  
traindata = {  
‘text’: [  
‘我们有一个愉快的星期天’,  
‘我们明天去吃饭？’,  
‘我们有英语课程’,  
‘我们明天出去玩吧’,  
‘哈哈哈，今天是星期二’  
]}  
when make MLM model train data，mask traindata as the model input, origin traindata as the label.  
for example:  
input=‘我们[MASK]天出去玩吧’, //mask position is random.  
label=‘我们明天出去玩吧’