# How does the finetune on transformer (t5) work
**URL:**
**Author:**  [@mab](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmab)%5C
**Post date:** [April 7, 2022, 2:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-does-the-finetune-on-transformer-t5-work%2F16585%2F1 "2022-04-07T14:42:30Z")
I am using pytorch lightning to finetune t5 transformer on a specific task. However, I was not able to understand how the finetuning works. I always see this code :
tokenizer = AutoTokenizer.from\_pretrained(hparams.model\_name\_or\_path)
model = AutoModelForSeq2SeqLM.from\_pretrained(hparams.model\_name\_or\_path)
I don’t get how the finetuning is done, are they only using fully connected layer on top of the t5 base model being trainable or using fully connected layer on top of the model being non-trainable (frozen) ?
---
**Author:**  [@NimaBoscarino](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNimaBoscarino)%5C
**Post date:** [April 8, 2022, 11:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-does-the-finetune-on-transformer-t5-work%2F16585%2F2 "2022-04-08T23:24:32Z")
Welcome! I’ll take a shot at answering this, but I’m not at expert at this so I may be wrong!
As far as I can understand, when you instantiate a model the weights are not frozen, so if you start finetuning on the model all parameters will be trainable. If you want to freeze weights, that’s something you’ll have to set manually, and the way that you do that will depend on what library (PyTorch, Tensorflow, or Flax) that you’re using.
When you use `AutoModelForSeq2SeqLM` (or any of the other `AutoModelX`) classes to instantiate a model with `.from_pretrained`, the backend that gets used is PyTorch. (As per [Auto Classes](https://huggingface.co/docs/transformers/model_doc/auto#transformers.models.auto.auto_factory._BaseAutoModelClass.from_pretrained)) So once you’ve loaded the model (with the PyTorch backend), if you want to freeze all of the base model’s weights you can access them and freeze them with:
```auto
for param in model.base_model.parameters():
param.requires_grad = False
```
If you were to use one of the Tensorflow or Flax auto-models, then you’d have to follow those libraries’ methods for freezing layers if that’s what you wanted to do.
I hope this helps!
---
**Author:**  [@mab](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmab)%5C
**Post date:** [April 8, 2022, 11:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-does-the-finetune-on-transformer-t5-work%2F16585%2F3 "2022-04-08T23:50:01Z")
Thank you so much for your answer, it is a great help.
By the way, I am trying to add another fully connected layer to the head. Would you know how to do it?
I figured that I can go to the source code library and change the modeling\_t5.py file.
it has the lm\_head defined as
`self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)`.
Thus, I thought if I make it as below, does it make sense, and do you think it is the right way of doing it?
`self.lm_head = nn.Sequential( nn.Linear(config.d_model, config.vocab_size, bias=False), nn.Flatten(), nn.Linear(config.d_model, config.vocab_size, bias=False), )`
---
**Author:**  [@NimaBoscarino](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNimaBoscarino)%5C
**Post date:** [April 11, 2022, 5:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-does-the-finetune-on-transformer-t5-work%2F16585%2F4 "2022-04-11T17:29:40Z")
Ah I’m definitely not the right person to answer this, but I _think_ you should be able to just alter the `model.lm_head` to something like:
```auto
model.lm_head = nn.Sequential(
nn.Linear(in_features=model.lm_head.in_features, out_features=, bias=False),
nn.Linear(in_features=, out_features=model.lm_head.in_features, bias=False)
)
```
And you can add other stuff in there too I think, as long the final output layer matches the expected dimensions (unless you want to change that too [How do I change the classification head of a model? - #19 by nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F19))
If you’re just adding layers to the head I don’t think that you need edit the source code. If you needed to change stuff within the network, e.g. [to make changes to the T5Block](https://github.com/huggingface/transformers/issues/7360), I think that’s when you would dive into the source code.
I’m curious to learn more about this though, so do update this post with anything new that you learn about this!