# Which model can use to pre-train a BERT model?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-model-can-use-to-pre-train-a-bert-model%2F13027%3E%5C
**Category:** Beginners\
**Created:** [December 22, 2021, 2:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-model-can-use-to-pre-train-a-bert-model%2F13027 "2021-12-22T02:29:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![MahdiA](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahdia/32/2700_2.png) [@MahdiA](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMahdiA)%5C
**Post date:** [December 22, 2021, 2:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-model-can-use-to-pre-train-a-bert-model%2F13027%2F1 "2021-12-22T02:29:42Z")

</div>

I am going to do pre-train a BERT model on specific dataset aiming for Sentiment Analysis.  
To self-train the model, which method will be better to use: Masked Language Modeling or Next Sentence Prediction? Or maybe there is not specific answer.

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fosanseviero)%5C
**Post date:** [December 22, 2021, 1:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-model-can-use-to-pre-train-a-bert-model%2F13027%2F2 "2021-12-22T13:44:52Z")

</div>

Choosing depends on what you want to do.

- Using masked language modeling is good when you want good representations of the data with which it was trained.
- Next sentence prediction, or rather causal language modeling (such as GPT), are better when you want to focus in generation.

The course has a section in how to fine-tune a masked language model that could be interesting to you: [Main NLP tasks - Hugging Face Course](https://huggingface.co/course/chapter7/3?fw=pt).