# MLM vs CLM, can be exchanged?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmlm-vs-clm-can-be-exchanged%2F21886%3E%5C
**Category:** Models\
**Created:** [August 21, 2022, 6:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmlm-vs-clm-can-be-exchanged%2F21886 "2022-08-21T18:02:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdelas](https://avatars.discourse-cdn.com/v4/letter/m/b5ac83/32.png) [@mdelas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmdelas)%5C
**Post date:** [August 21, 2022, 6:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmlm-vs-clm-can-be-exchanged%2F21886%2F1 "2022-08-21T18:02:18Z")

</div>

Hello everyone, I have been building a BERT model with a Masked Language Model approach as my pretraining goal. All of a sudden I need to, instead of using a Masked Language Model, need the outputs of a Causal Language Model (predict the n+1 token of the sequence).

I was thinking if it would be correct to use this model and just the last token in the preprocessing and re-train my model. Do you think that’s a good approach? Or I should build again a model specific to an autoregressive behaviour…

Any references?

Thank you very much