# Difference BertModel, AutoModel and AutoModelForMaskedLM

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 12, 2022, 2:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340 "2022-10-12T14:16:45Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Indramal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/indramal/32/61744_2.png) [@Indramal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIndramal)%5C
**Post date:** [October 12, 2022, 2:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F1 "2022-10-12T14:16:45Z")

</div>

What are the different following codes,

Code 1:

```auto
from transformers import BertModel
model = BertModel.from_pretrained("bert-base-uncased")

```

Code 2:

```auto
from transformers import AutoModel
model = AutoModel.from_pretrained("bert-base-uncased")

```

Code 3:

```auto
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")

```

Considering the following image, please explain. It will help for me.

When I search `AutoModelForMaskedLM` method loading models have head but not sure. I have check layers but all are same. All have 12 layers.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/acfb84e676784b195eab88b2e8b88fc38ea85e17.png)

---

<div class="post-metadata">

**Author:** ![NimaBoscarino](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimaboscarino/32/4358_2.png) [@NimaBoscarino](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNimaBoscarino)%5C
**Post date:** [October 12, 2022, 6:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F2 "2022-10-12T18:06:00Z")

</div>

The `AutoModel` will look at the `bert-base-uncased` model’s configuration and choose the appropriate base model architecture to use, which in this case is `BertModel`. So Code 1 and Code 2 will essentially do the same thing, and when you run an inference on either of those models you’ll get the same output, which is the last hidden states from the `bert-base-uncased` model body.

However! `bert-base-uncased` was trained using a masked language modelling objective, so the model also has an associated classifier for turning those hidden states into logits that you can use for masked token prediction. In order to do that, you need to instantiate the model with a MaskedLM head, which you can do in a couple ways, including:

1. You can use [BertForMaskedLM](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForMaskedLM)
2. You can use AutoModelForMaskedLM, like you did in Code 3 (which will go find the appropriate `<MODEL>ForMaskedLM` class).

So then when you run an inference on the resulting model, the outputs have the logits.

[Here’s a notebook illustrating it.](https://colab.research.google.com/drive/1URaUupHwU4DCmqpB9z16aOmm48L7NC2E?usp=sharing)

**In short:** With Code 1 and Code 2 you’re instantiating the model _without the Head_, and Code 3 is instantiating it _with the head_.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![Indramal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/indramal/32/61744_2.png) [@Indramal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIndramal)%5C
**Post date:** [October 13, 2022, 3:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F3 "2022-10-13T03:32:58Z")

</div>

Thank you for your information.

I have checked using config but all models in codes 1, 2 and 3 give a same number of hidden layers 12.

> BertConfig {  
> “\_name\_or\_path”: “bert-base-uncased”,  
> “architectures”: [  
> “BertForMaskedLM”  
> ],  
> “attention\_probs\_dropout\_prob”: 0.1,  
> “classifier\_dropout”: null,  
> “gradient\_checkpointing”: false,  
> “hidden\_act”: “gelu”,  
> “hidden\_dropout\_prob”: 0.1,  
> “hidden\_size”: 768,  
> “initializer\_range”: 0.02,  
> “intermediate\_size”: 3072,  
> “layer\_norm\_eps”: 1e-12,  
> “max\_position\_embeddings”: 512,  
> “model\_type”: “bert”,  
> “num\_attention\_heads”: 12,  
> “num\_hidden\_layers”: 12,  
> “pad\_token\_id”: 0,  
> “position\_embedding\_type”: “absolute”,  
> “transformers\_version”: “4.23.1”,  
> “type\_vocab\_size”: 2,  
> “use\_cache”: true,  
> “vocab\_size”: 30522  
> }

Code example:

```auto
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
print(model.config)

```

Why it shows the same model architecture? Is there any other method to check? (Without checking the output)

I need to check the architecture.

---

<div class="post-metadata">

**Author:** ![Indramal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/indramal/32/61744_2.png) [@Indramal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIndramal)%5C
**Post date:** [October 13, 2022, 5:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F4 "2022-10-13T05:23:23Z")

</div>

Also need to know `AutoModel` and `BertModel` give **Embeddings + Layers + Hidden States** or **Embeddings + Layers**??

---

<div class="post-metadata">

**Author:** ![mineshj1291](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mineshj1291/32/37149_2.png) [@mineshj1291](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmineshj1291)%5C
**Post date:** [October 13, 2022, 5:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F5 "2022-10-13T05:38:55Z")

</div>

you can try this way:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/4/4facf9e846606d4b02490845d90e88d411399e1a.png)

---

<div class="post-metadata">

**Author:** ![Indramal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/indramal/32/61744_2.png) [@Indramal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIndramal)%5C
**Post date:** [October 13, 2022, 8:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F6 "2022-10-13T08:40:23Z")

</div>

@mineshj1291

I try to do with your code but I need to input `input size` to the summery. How can I find it?

Code:

```auto
from transformers import AutoModel, AutoModelForMaskedLM
import torchsummary
model1 = AutoModel.from_pretrained("bert-base-uncased")
summery1 = torchsummary.summary(model1)

```

Error:

> TypeError: summary() missing 1 required positional argument: ‘input\_size’

If I put input size, it given following error,

```auto
from transformers import AutoModel, AutoModelForMaskedLM
import torchsummary
model1 = AutoModel.from_pretrained("bert-base-uncased")
summery1 = torchsummary.summary(model1, input_size=(512,))

```

Error:

> RuntimeError: Expected tensor for argument #1 ‘indices’ to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)

How did you get it?

---

<div class="post-metadata">

**Author:** ![mineshj1291](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mineshj1291/32/37149_2.png) [@mineshj1291](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmineshj1291)%5C
**Post date:** [October 13, 2022, 1:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F7 "2022-10-13T13:08:11Z")

</div>

you can try upgrading torch-summary and transformers library.  
here is colab for the same, if you want to try out. [transformers\_model\_summary.ipynb](https://colab.research.google.com/drive/1lx2u9TMQzXfzgNkjn3WcuSBYyzIDpgv-?usp=sharing)

---

<div class="post-metadata">

**Author:** ![Indramal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/indramal/32/61744_2.png) [@Indramal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIndramal)%5C
**Post date:** [October 13, 2022, 2:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F8 "2022-10-13T14:08:00Z")

</div>

Thank you very when I update it, it is working.

When I see AutoModel, It has 3 architectures. Included AutoModelForMaskedLM.

> # ====================================================================== Layer (type:depth-idx) Param #
> 
> # ├─BertEmbeddings: 1-1 – | └─Embedding: 2-1 23,440,896 | └─Embedding: 2-2 393,216 | └─Embedding: 2-3 1,536 | └─LayerNorm: 2-4 1,536 | └─Dropout: 2-5 – ├─BertEncoder: 1-2 – | └─ModuleList: 2-6 – | | └─BertLayer: 3-1 7,087,872 | | └─BertLayer: 3-2 7,087,872 | | └─BertLayer: 3-3 7,087,872 | | └─BertLayer: 3-4 7,087,872 | | └─BertLayer: 3-5 7,087,872 | | └─BertLayer: 3-6 7,087,872 | | └─BertLayer: 3-7 7,087,872 | | └─BertLayer: 3-8 7,087,872 | | └─BertLayer: 3-9 7,087,872 | | └─BertLayer: 3-10 7,087,872 | | └─BertLayer: 3-11 7,087,872 | | └─BertLayer: 3-12 7,087,872 ├─BertPooler: 1-3 – | └─Linear: 2-7 590,592 | └─Tanh: 2-8 –
> 
> # Total params: 109,482,240 Trainable params: 109,482,240 Non-trainable params: 0
> 
> # =========================================================================== Layer (type:depth-idx) Param #
> 
> # ├─BertModel: 1-1 – | └─BertEmbeddings: 2-1 – | | └─Embedding: 3-1 23,440,896 | | └─Embedding: 3-2 393,216 | | └─Embedding: 3-3 1,536 | | └─LayerNorm: 3-4 1,536 | | └─Dropout: 3-5 – | └─BertEncoder: 2-2 – | | └─ModuleList: 3-6 85,054,464 ├─BertOnlyMLMHead: 1-2 – | └─BertLMPredictionHead: 2-3 – | | └─BertPredictionHeadTransform: 3-7 592,128 | | └─Linear: 3-8 23,471,418
> 
> # Total params: 132,955,194 Trainable params: 132,955,194 Non-trainable params: 0
> 
> # ====================================================================== Layer (type:depth-idx) Param #
> 
> # ├─BertEmbeddings: 1-1 – | └─Embedding: 2-1 23,440,896 | └─Embedding: 2-2 393,216 | └─Embedding: 2-3 1,536 | └─LayerNorm: 2-4 1,536 | └─Dropout: 2-5 – ├─BertEncoder: 1-2 – | └─ModuleList: 2-6 – | | └─BertLayer: 3-1 7,087,872 | | └─BertLayer: 3-2 7,087,872 | | └─BertLayer: 3-3 7,087,872 | | └─BertLayer: 3-4 7,087,872 | | └─BertLayer: 3-5 7,087,872 | | └─BertLayer: 3-6 7,087,872 | | └─BertLayer: 3-7 7,087,872 | | └─BertLayer: 3-8 7,087,872 | | └─BertLayer: 3-9 7,087,872 | | └─BertLayer: 3-10 7,087,872 | | └─BertLayer: 3-11 7,087,872 | | └─BertLayer: 3-12 7,087,872 ├─BertPooler: 1-3 – | └─Linear: 2-7 590,592 | └─Tanh: 2-8 –
> 
> # Total params: 109,482,240 Trainable params: 109,482,240 Non-trainable params: 0

---

<div class="post-metadata">

**Author:** ![mineshj1291](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mineshj1291/32/37149_2.png) [@mineshj1291](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmineshj1291)%5C
**Post date:** [March 9, 2025, 5:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-bertmodel-automodel-and-automodelformaskedlm%2F24340%2F9 "2025-03-09T17:12:12Z")

</div>

it works fine without input\_size  
here are the library versions.

```auto
torch-summary-1.4.5 transformers-4.49.0

```