# Gradual Layer Freezing with huggingface model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-layer-freezing-with-huggingface-model%2F3566%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 9, 2021, 7:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-layer-freezing-with-huggingface-model%2F3566 "2021-02-09T19:22:45Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![tillfurger](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@tillfurger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftillfurger)%5C
**Post date:** [February 9, 2021, 7:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-layer-freezing-with-huggingface-model%2F3566%2F1 "2021-02-09T19:22:45Z")

</div>

Hi I would like to freeze only the first few encoder layers in a bert model.

I can do it by looping through the 202 layers and freeze them by order

```
model = BertForMaskedLM.from_pretrained('bert-base-uncased')

for param in model.parameters()[6:60]:
   param.requires_grad = False

```

But isn’t there a better way to freeze some layers not having to rely on the order?

Thanks

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [February 10, 2021, 2:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-layer-freezing-with-huggingface-model%2F3566%2F2 "2021-02-10T14:52:40Z")

</div>

Hi,

I don’t think you can freeze the encoder layers without effectively freezing the embedding layer too. (The changes won’t be able to propagate back through the frozen layers to reach the earlier layers.)