# Questions on the \`BertModelLMHeadModel\`
**URL:**
**Author:**  [@h56cho](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fh56cho)%5C
**Post date:** [September 30, 2020, 5:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-the-bertmodellmheadmodel%2F1317%2F1 "2020-09-30T17:36:41Z")
Hello,
I have a few questions about the `BertModelLMHeadModel`:
1. Is `BertModelLMHeadModel` used to conduct the regular language modeling (next token prediction), as it is the case for the `GPT2LMHeadModel`?
2. For `GPT2LMHeadModel`, I can just specify `labels = input_ids` for convenience. I just specify the `labels` in this way for the `BertModelLMHeadModel` as well?
Thanks,
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 2, 2020, 2:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-the-bertmodellmheadmodel%2F1317%2F2 "2020-10-02T14:55:36Z")
Hi @h56cho
do you mean the `BertLMHeadModel` ?
If yes then, it’s intended to be used with the `EncoderDecoder` model which allows you to use pre-trained encoder for as both encoder and decoder for seq2seq tasks. It’s not intended for language modeling.
While you can use that class as a standalone decoder by passing `is_decoder=True` to `config` it might not give you good results as it’s trained as an encoder.
---
**Author:**  [@h56cho](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fh56cho)%5C
**Post date:** [October 2, 2020, 4:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-the-bertmodellmheadmodel%2F1317%2F3 "2020-10-02T16:42:00Z")
Hello,
HuggingFace Transformer documentation seem to point out that `BertLMHeadModel` can be used for causal language modeling([https://huggingface.co/transformers/model\_doc/bert.html#bertmodellmheadmodel](https://huggingface.co/transformers/model_doc/bert.html#bertmodellmheadmodel)). If you look at the returned values from this model, it includes `causalLMoutput`. doesn’t the term “causal language modeling” refer to regular language modeling, as in the case for GPT-2? I am not so interested in the accuracy of the results, my intention is to examine the distribution of the attention weights.
Also, when providing “labels” for the causal language modeling with the `BertLMHeadModel`, can I just use `labels = input_ids` as in the case for GPT-2, for convinence?
Thank you,
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 2, 2020, 5:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-the-bertmodellmheadmodel%2F1317%2F4 "2020-10-02T17:02:33Z")
That’s what I said in the last comment,
It can be used as a standalone decoder (standalone decoder = causal LM).
Yes, you can pass `labels = input_ids`
---
**Author:**  [@h56cho](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fh56cho)%5C
**Post date:** [October 2, 2020, 5:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-the-bertmodellmheadmodel%2F1317%2F5 "2020-10-02T17:03:23Z")
Thank you!
---
**Author:**  [@h56cho](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fh56cho)%5C
**Post date:** [October 2, 2020, 8:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-the-bertmodellmheadmodel%2F1317%2F6 "2020-10-02T20:27:00Z")
Hello,
Sorry I have some additional question. This question is about the `BertForMaskedLM` model.
The documentation for `BertForMaskedLM` provides the following example to illustrate the model’s usage:
```auto
>>> from transformers import BertTokenizer, BertForMaskedLM
>>> import torch
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = BertForMaskedLM.from_pretrained('bert-base-uncased', return_dict=True)
>>> input_ids = tokenizer("Hello, my dog is cute", return_tensors="pt")["input_ids"]
>>> outputs = model(input_ids, labels=input_ids)
>>> loss = outputs.loss
>>> prediction_logits = outputs.logits
```
In the example above, I don’t see any `[MASK]` token in the input; can the `BertForMaskedLM` model really be used with an input string that does not include `[MASK]` token? If I provide `BertForMaskedLM` model an input string that does not include the `[MASK]` token, from which token will the output of the model be produced from? In this case, would `BertForMaskedLM` automatically insert `[MASK]` token in the beginning of the input sequence?
Thank you again,
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 5, 2020, 10:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-the-bertmodellmheadmodel%2F1317%2F7 "2020-10-05T10:41:30Z")
That’s probably a mistake,
This [might](https://huggingface.co/transformers/task_summary.html#masked-language-modeling) help.
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 5, 2020, 1:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-the-bertmodellmheadmodel%2F1317%2F8 "2020-10-05T13:36:03Z")
Yes it’s definitely a mistake. Will fix this morning.