# Training on Domain specific Dataset
**URL:**
**Author:**  [@devesh1412](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdevesh1412)%5C
**Post date:** [March 21, 2021, 12:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-on-domain-specific-dataset%2F4805%2F1 "2021-03-21T00:22:21Z")
Hi ,
I want to train a sentiment analysis multi-label classifier and in addition to training the final output layers, I’d like to train the hidden BERT layers as well.
I want to understand how much improvement can I get in my metric(F1 score) by feeding it my domain-specific data.
All the documents /references I have seen thus far only point to training the final output layer that generates classification. Is there a way to train various hidden layers of BERT using (let’s say) BERT Base?
Thanks in advance,
Devesh
---
**Author:**  [@neuralpat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fneuralpat)%5C
**Post date:** [March 21, 2021, 9:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-on-domain-specific-dataset%2F4805%2F2 "2021-03-21T09:24:37Z")
My understanding is that if you don’t specifically freeze any of the layers you will always train the whole model.
If you want to train only particular layers, you can add a condition to this code:
```
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
for param in model.bert.parameters():
param.requires_grad = False
```
---
**Author:**  [@devesh1412](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdevesh1412)%5C
**Post date:** [March 21, 2021, 5:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-on-domain-specific-dataset%2F4805%2F3 "2021-03-21T17:34:14Z")
Hi @neuralpat ,
Thanks so much for taking the time and responding to my queries.I’m going to try it out.
I am also wondering if there is a way to freeze specific layers(eg. Top 3 ) and train ?
---
**Author:**  [@assansanogo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fassansanogo)%5C
**Post date:** [March 22, 2021, 9:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-on-domain-specific-dataset%2F4805%2F4 "2021-03-22T21:15:20Z")
Hi,
`model = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
```
for param in model.bert.parameters():
param.requires_grad = False
```
as far I understand, the code above is for **ALL** params
but you could easily limit to 3 elements :
`model.bert.parameters()` is a generator
```
i=0
for i,el in enumerate(model.parameters()):
if i<3:
el.requires_grad=False
else:
el.requires_grad=True
```
You can check the result with the result with:
```
for param in model.parameters():
print(param)
```