# Fill-mask and classification at the same time
**URL:**
**Author:**  [@jmenke](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjmenke)%5C
**Post date:** [March 18, 2022, 10:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffill-mask-and-classification-at-the-same-time%2F15846%2F1 "2022-03-18T10:14:40Z")
Hi I was wondering whether one can easily train both masking and sentence classification at the same time.
So the goal is to use a pre-trained model and keep masking input for my domain-specific dataset, but at the same time use the Start Token to make prediction. I would combine the two losses and perform backpropagation.
---
**Author:**  [@lucadini](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flucadini)%5C
**Post date:** [March 18, 2022, 10:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffill-mask-and-classification-at-the-same-time%2F15846%2F2 "2022-03-18T10:36:39Z")
The original BERT architecture was trained both for Masked Language Modeling and Next Sentence Prediction tasks. The latter is basically sentence classification, so you can simply use the origina architecture. You can find it at [modeling\_bert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py), it’s called `BertForNextSentencePrediction`. You can change the classification task simply deciding to give the model different labels!
---
**Author:**  [@jmenke](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjmenke)%5C
**Post date:** [March 18, 2022, 2:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffill-mask-and-classification-at-the-same-time%2F15846%2F3 "2022-03-18T14:00:49Z")
Thanks you very much for the answer.
Is there also an (easy) way to train pretrained models which were originally only trained on MLM to also include an additional prediction task.
---
**Author:**  [@lucadini](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flucadini)%5C
**Post date:** [March 18, 2022, 2:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffill-mask-and-classification-at-the-same-time%2F15846%2F4 "2022-03-18T14:46:57Z")
I think the procedure is the same but you use pre-trained weights in model creation by calling
> model = BertForNextSentencePrediction.from\_pretrained(
> model\_args.model\_name\_or\_path, […])
You can use the weights of the same model trained only for MLM, you simply get a warning that some weights are initialized from scratch (the weights of the classification layer), and that’s what you want!
EDIT: I’m assuming you’re using BERT, I don’t know if there is the same model structure for other models. You should look in the models folder [here](https://github.com/huggingface/transformers/tree/master/src/transformers/models). Probabily there is a class named [Name of the model]ForSequenceClassification or something similar!
It’s possible that you have to mix the MLM and sequence classification heads tho!
---
**Author:**  [@jmenke](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjmenke)%5C
**Post date:** [March 18, 2022, 3:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffill-mask-and-classification-at-the-same-time%2F15846%2F5 "2022-03-18T15:29:10Z")
Ah okay thank is what I was looking for. I am using Roberta which seems to not offer this option, if i looked it up correctly