# How to freeze layers while fine-tuning?
**URL:**
**Author:**  [@luohuashijieyoufengj](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fluohuashijieyoufengj)%5C
**Post date:** [May 15, 2025, 9:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-while-fine-tuning%2F155342%2F1 "2025-05-15T09:17:46Z")
I download pretrained model from huggingface, and add some layers to it.
Now I only want to change layers I added, so I need to freeze layers of pretrained model while training. How can I do it?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [May 15, 2025, 2:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-while-fine-tuning%2F155342%2F2 "2025-05-15T14:19:23Z")
Maybe `requires_grad=False`?
> [@How to freeze layers using trainer?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702)%3A
>
> Hey, I am trying to figure out how to freeze layers of a model and read that I had to use for param in model.base\_model.parameters(): param.requires\_grad = False if I wanted to freeze the encoder of a pretrained MLM for example. But how do I use this with the Trainer? I tried the following: from transformers import BertTokenizer, BertForMaskedLM. LineByLineTextDataset, DataCollatorForLanguageModeling, Trainer, TrainingArguments model = BertForMaskedLM.from\_pretrained('bert-base-uncased'…
>
>
> \### Feature request
>
> Attempt to optimize the training for models with weights/pa…rameters that are set to \`requires\_grad=False\`. This is done by excluding these parameters in the optimizer.
>
> \### Motivation
>
> I am building a Seq2Seq model where I use a pre-trained model for the encoder. I freeze all the parameters of the encoder by setting \`requires\_grad=False\`. I expected the training to speed up compared to a model where both the encoder and decoder weights are trainable. However, I found that there's no difference in speed and also memory.
>
> I investigated a bit and found that all the model parameters, regardless of whether gradients are required to be computed, are included in the optimizer https://github.com/huggingface/transformers/blob/00ba7cadd812437708b380ab078a3cfe8cfaff31/src/transformers/trainer.py#L1021-L1030
>
> I tested an idea and subclassed the \`Seq2SeqTrainer\`. So, I updated the above snippet with this:
>
> \`\`\`Python
> optimizer\_grouped\_parameters = \[
> {
> # Add here the \`p.requires\_grad\` condition
> "params": \[p for n, p in opt\_model.named\_parameters() if (n in decay\_parameters and p.requires\_grad)\],
> "weight\_decay": self.args.weight\_decay,
> },
> {
> # Add here the \`p.requires\_grad\` condition
> "params": \[p for n, p in opt\_model.named\_parameters() if (n not in decay\_parameters and p.requires\_grad)\],
> "weight\_decay": 0.0,
> },
> \]
> \`\`\`
>
> Doing this actually improved both the speed and the memory during the training.
>
> I was wondering if this is something we can add to the codebase. If not, I am curious as to why we shouldn't exclude the parameters that are intended not to be trainable in the optimizer.
>
> \### Your contribution
>
> I can make the PR if this is an acceptable change. 🤗
---
**Author:**  [@luohuashijieyoufengj](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fluohuashijieyoufengj)%5C
**Post date:** [May 16, 2025, 3:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-while-fine-tuning%2F155342%2F3 "2025-05-16T03:11:00Z")
It seems work, thanks.