# How to freeze layers using trainer?
**URL:**
**Author:**  [@AJHoeh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAJHoeh)%5C
**Post date:** [March 19, 2021, 11:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F1 "2021-03-19T11:44:58Z")
Hey,
I am trying to figure out how to freeze layers of a model and read that I had to use
```auto
for param in model.base_model.parameters():
param.requires_grad = False
```
if I wanted to freeze the encoder of a pretrained MLM for example. But how do I use this with the Trainer?
I tried the following:
```auto
from transformers import BertTokenizer, BertForMaskedLM. LineByLineTextDataset, DataCollatorForLanguageModeling, Trainer, TrainingArguments
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
for param in model.base_model.parameters():
param.requires_grad = False
dataset = LineByLineTextDataset(
tokenizer=tokenizer,
file_path=in_path,
block_size=512,
)
data_collator = DataCollatorForLanguageModeling(
tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)
training_args = TrainingArguments(
output_dir=out_path,
overwrite_output_dir=True,
num_train_epochs=25,
per_device_train_batch_size=48,
save_steps=500,
save_total_limit=2,
seed=1
)
trainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset=dataset
)
trainer.train()
```
If the encoder was frozen I would expect it to produce the same outputs as a fresh instance of the pretrained encoder, but it doesn’t:
```auto
model_fresh = BertForMaskedLM.from_pretrained('bert-base-uncased')
inputs = tokenizer("This is a boring test sentence", return_tensors="pt")
torch.all(model.bert(**inputs)[0].eq(model_fresh.bert(**inputs)[0]))
--> tensor(false)
```
So I must be doing somethin wrong here, I guess the Trainer is reseting the requires\_grad attribute and I have to overwrite it somehow after I instanciated the trainer?
Thanks in advance!
Johannes
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [March 19, 2021, 12:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F2 "2021-03-19T12:56:04Z")
Looking at the [source code of BertForMaskedLM](https://github.com/huggingface/transformers/blob/e8968bd03ad35c2bd0e38ddcbd9243870212944f/src/transformers/models/bert/modeling_bert.py#L1253), the base model is the “bert” attribute, not the “base\_model” attribute. So if you want to freeze the parameters of the base model before training, you should type
```
for param in model.bert.parameters():
param.requires_grad = False
```
instead.
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 19, 2021, 12:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F3 "2021-03-19T12:58:18Z")
@nielsr `base_model` is an attribute that will work on all the `PreTraineModel` (to make it easy to access the encoder in a generic fashion) 🙂
The `Trainer` puts your model into training mode, so your difference might simply come from that (there are dropouts in the model). You should check if putting it back in eval mode solves your problem.
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [March 19, 2021, 12:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F4 "2021-03-19T12:59:04Z")
@sgugger oh didn’t know that, I learn every day!
---
**Author:**  [@AJHoeh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAJHoeh)%5C
**Post date:** [March 19, 2021, 1:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F5 "2021-03-19T13:01:38Z")
OMG! This is so obvious and I cant believe I didn’t realize that. Will test and report! Thanks 🙂
---
**Author:**  [@AJHoeh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAJHoeh)%5C
**Post date:** [March 19, 2021, 4:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F6 "2021-03-19T16:13:03Z")
@sgugger model.eval() should have done the trick, right? I am afraid the results still don’t match ☹
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 19, 2021, 4:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F7 "2021-03-19T16:37:39Z")
You should inspect the weights to see where they difer then. `Trainer` will not change the `requires_grad` value of your parameters.
---
**Author:**  [@AJHoeh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAJHoeh)%5C
**Post date:** [March 22, 2021, 10:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F8 "2021-03-22T10:53:59Z")
@sgugger Thanks, that was important to know for me so I knew I had to be the one screwing up somewhere else and I did somehow manage that 😄
---
**Author:**  [@anon58275033](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon58275033)%5C
**Post date:** [July 14, 2021, 7:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F9 "2021-07-14T19:55:10Z")
Hi,
I tried your code, but I am getting this error:
```auto
AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'
```
---
**Author:**  [@AJHoeh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAJHoeh)%5C
**Post date:** [July 14, 2021, 8:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F10 "2021-07-14T20:26:00Z")
Hey,
yeah, this is because you are using roberta instead of bert, therefore it uses .roberta to store the encoder. I believe there is some model independant keyword like “base\_model” or something, but I dont know right now (im on vacation, but maybe you can try or google it). Hope that helps!
Best
Johannes
---
**Author:**  [@AJHoeh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAJHoeh)%5C
**Post date:** [July 14, 2021, 8:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F11 "2021-07-14T20:30:30Z")
Sorry, answered per mail, sgugger literary provided the base\_model keyword in this thread, so there you go 😉
---
**Author:**  [@Hanqix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHanqix)%5C
**Post date:** [May 26, 2024, 5:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F12 "2024-05-26T17:55:16Z")
Maybe its a problem with using eq instead of isclose?