# How to do unsupervised fine-tuning?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-do-unsupervised-fine-tuning%2F3429%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 29, 2021, 2:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-do-unsupervised-fine-tuning%2F3429 "2021-01-29T02:11:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![msamogh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/msamogh/32/967_2.png) [@msamogh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmsamogh)%5C
**Post date:** [January 29, 2021, 2:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-do-unsupervised-fine-tuning%2F3429%2F1 "2021-01-29T02:11:54Z")

</div>

I have a custom text dataset, which I want BERT to get acquainted with. My final goal is not to run any supervised task (it is actually to act as a starting point to get sentence embeddings from [S-BERT](https://github.com/UKPLab/sentence-transformers).

I just want to _continue_ doing the unsupervised training on my dataset. How do I do this?

So far, I have come across two possible candidates in the documentation for this:

1. `BertForPreTraining` (the self-explanatory name led me to this)
2. `BERTForMaskedLM` (as used in [this blog post](https://huggingface.co/blog/how-to-train)).

Can both of them be used for this purpose? Is one more attuned to my purpose? Have you previously tried to do something like this? Any additional suggestions would also be very helpful.

Thank you 🙂

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [January 29, 2021, 9:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-do-unsupervised-fine-tuning%2F3429%2F2 "2021-01-29T09:16:43Z")

</div>

`BertForPreTraining` has two heads, one for masked language modeling and one for next sentence prediction task. This class should be used when you want to pre-train the bert as described in the paper i.e MLM + NSP

`BERTForMaskedLM`, is for MLM training which can be used for pre-training.