# How to freeze distillbert params during finetuning?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-distillbert-params-during-finetuning%2F12930%3E%5C
**Category:** Beginners\
**Created:** [December 17, 2021, 4:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-distillbert-params-during-finetuning%2F12930 "2021-12-17T16:47:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![alghar](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@alghar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falghar)%5C
**Post date:** [December 17, 2021, 4:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-freeze-distillbert-params-during-finetuning%2F12930%2F1 "2021-12-17T16:47:57Z")

</div>

working with TFDistilBertForSequenceClassification and trying to fine tune it. How can I freeze the distillbert layers and only restrict training to params of added classification header layers? Below pls find the layers:

* * *

# Layer (type) Output Shape Param #

distilbert (TFDistilBertMai multiple 66362880  
nLayer)

pre\_classifier (Dense) multiple 590592

classifier (Dense) multiple 1538

dropout\_159 (Dropout) multiple 0

=================================================================  
Total params: 66,955,010  
Trainable params: 66,955,010  
Non-trainable params: 0

* * *