# Tutorial on Pretraining BERT

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftutorial-on-pretraining-bert%2F2828%3E%5C
**Category:** Beginners\
**Created:** [December 15, 2020, 8:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftutorial-on-pretraining-bert%2F2828 "2020-12-15T08:58:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![npan1990](https://avatars.discourse-cdn.com/v4/letter/n/3d9bf3/32.png) [@npan1990](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnpan1990)%5C
**Post date:** [December 15, 2020, 8:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftutorial-on-pretraining-bert%2F2828%2F1 "2020-12-15T08:58:10Z")

</div>

I would like to pretrain a BERT model with custom data. For example, data in my native language. Is there any example script about pretraining a BERT model? I found the tutorial with the EsperBERTo model but this is not a BERT model as far as I understand.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [December 15, 2020, 9:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftutorial-on-pretraining-bert%2F2828%2F2 "2020-12-15T09:01:11Z")

</div>

You can have a look at the example scripts here: [https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling)