# Freeze Lower Layers with Auto Classification Model
**URL:**
**Author:**  [@drew2019](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrew2019)%5C
**Post date:** [November 5, 2021, 12:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffreeze-lower-layers-with-auto-classification-model%2F11386%2F1 "2021-11-05T12:39:03Z")
I’ve been unsuccessful in freezing lower pretrained BERT layers when training a classifier using Huggingface. I’m using AutoModelForSequenceClassification particularly, via code below, and I want to freeze the lower X layers (ex: lower 9 layers). Is this possible in HuggingFace, and if so what code would I add to this for functionality?
> tokenizer = AutoTokenizer.from\_pretrained(“bert-base-cased”)
>
> def tokenize\_function(examples):
> return tokenizer(examples[“text”], max\_length = 512, padding=“max\_length”, truncation=True)
>
> tokenized\_train = train.map (tokenize\_function, batched=True)
> tokenized\_test= test.map (tokenize\_function, batched=True)
>
> model = AutoModelForSequenceClassification.from\_pretrained(“bert-base-cased”, num\_labels=1)
>
> for w in model.bert.parameters():
> w.\_trainable= False
>
> training\_args = TrainingArguments(“test\_trainer”, evaluation\_strategy=“epoch”, per\_device\_train\_batch\_size=8)
> trainer = Trainer(model=model, args=training\_args, train\_dataset=tokenized\_train, eval\_dataset=tokenized\_test)
> trainer.train()
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [November 5, 2021, 12:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffreeze-lower-layers-with-auto-classification-model%2F11386%2F2 "2021-11-05T12:58:19Z")
Yes, in PyTorch freezing layers is quite easy. It can be done as follows:
```auto
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(“bert-base-cased”, num_labels=1)
for name, param in model.named_parameters():
if name.startswith("..."): # choose whatever you like here
param.requires_grad = False
```
---
**Author:**  [@drew2019](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrew2019)%5C
**Post date:** [November 5, 2021, 1:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffreeze-lower-layers-with-auto-classification-model%2F11386%2F3 "2021-11-05T13:18:48Z")
> [@nielsr](#):
>
> ```auto
> for name, param in model.named_parameters():
> if name.startswith("..."): # choose whatever you like here
> param.requires_grad = False
>
> ```
Thank you so much nielsr for the quick and useful reply. I believe I got this to work. So to verify, that can be written prior to “Trainer” command and will freeze any specified parameter? So for example, I could write the code below to freeze the first two layers.
> for name, param in model.named\_parameters():
> if name.startswith(“bert.encoder.layer.1”):
> param.requires\_grad = False
> if name.startswith(“bert.encoder.layer.2”):
> param.requires\_grad = False
This question shows my ignorance, but is there a way to print model settings prior to training to verify which layers/parameters are frozen?
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [November 5, 2021, 2:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffreeze-lower-layers-with-auto-classification-model%2F11386%2F4 "2021-11-05T14:01:02Z")
To verify which layers are frozen, you can do:
```auto
for name, param in model.named_parameters():
print(name, param.requires_grad)
```
---
**Author:**  [@jpcompartir](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjpcompartir)%5C
**Post date:** [March 7, 2022, 3:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffreeze-lower-layers-with-auto-classification-model%2F11386%2F5 "2022-03-07T15:47:33Z")
Would just add to this, you probably want to freeze layer 0, and you don’t want to freeze 10, 11, 12 (if using 12 layers for example), so “bert.encoder.layer.1.” rather than “bert.encoder.layer.1” should avoid such things.
---
**Author:**  [@ShaoMinHol](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FShaoMinHol)%5C
**Post date:** [May 5, 2022, 3:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffreeze-lower-layers-with-auto-classification-model%2F11386%2F6 "2022-05-05T03:16:54Z")
May I know for subsequent operations such as model.train and model.eval, does it change the the param.requires\_grad that is specify above? Or do I have to do the above, everytime I change between training and eval mode?
---
**Author:**  [@Morphe](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMorphe)%5C
**Post date:** [May 25, 2023, 1:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffreeze-lower-layers-with-auto-classification-model%2F11386%2F7 "2023-05-25T01:30:50Z")
According to this comment [How to freeze layers using trainer? - #7 by sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-layers-using-trainer%2F4702%2F7) it will not.