# Add a classification head to a fine-tuned language model
**URL:**
**Author:**  [@Chris-Palmer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FChris-Palmer)%5C
**Post date:** [July 8, 2021, 10:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-a-classification-head-to-a-fine-tuned-language-model%2F8176%2F1 "2021-07-08T10:49:39Z")
We I have fine-tuned a GPT-2 model with a language model head on medical triage text, and would like to use this model as a classifier. However, as far as I can tell, the Automodel Huggingface library allows me to have either a LM or a classifier etc. head, but I don’t see a way to add a classifier on _top_ of a _fine-tuned_ LM.
Am I mistaken in my understanding of the possibilities? We want to preserve the fine-tuning we have done for the LM task, so don’t want to swap this head for a classifier and lose all of those weights. I suppose we could unlock the top layers of the GPT2 model and try and push the training back into the model, but I suspect that would soon become untenable for our somewhat limited resources. I see this question has been asked [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-bert-model-on-domain-specific-language-and-for-classification%2F3106)%2C and my fellow worker has posted a follow-up question in that thread, but we didn’t see any answer we could relate to, so I am asking again in this new topic.
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 8, 2021, 11:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-a-classification-head-to-a-fine-tuned-language-model%2F8176%2F2 "2021-07-08T11:29:45Z")
You need to use the `AutoModelForSequenceClassification` class to add a classification head on top of your pretrained model.
---
**Author:**  [@Chris-Palmer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FChris-Palmer)%5C
**Post date:** [July 8, 2021, 12:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-a-classification-head-to-a-fine-tuned-language-model%2F8176%2F3 "2021-07-08T12:05:39Z")
Thanks for your reply @sgugger. I have a fine-tuned pretrained model, that is - a `AutoModelForCausalLM.from_pretrained("gpt2-large")`, which I have fine-tuned on my specific language (which is triage text and quite unlike standard English).
I thought that if I called an `AutoModelForSequenceClassification("gpt2-large")` then it would be just like reverting to the standard pretrained model, without any of the fine-tuning I have just done, since my fine-tuning resides in the head of the `AutoModelForCausalLM`.
My question was if it was possible to sit a classifier head on top of the fine tuned language generator model, so as to inherit the weights that I have obtained through the fine-tuning. If I load the weights of the `AutoModelForCausalLM` into the `AutoModelForSequenceClassification` then the fine-tuned weights are discarded, but would it be beneficial to add a linear layer to allow those weights to be preserved for use by a manually added classifier head?
---
**Author:**  [@bdzyubak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbdzyubak)%5C
**Post date:** [May 31, 2024, 6:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-a-classification-head-to-a-fine-tuned-language-model%2F8176%2F4 "2024-05-31T18:11:22Z")
Couple of thoughts, though I am relatively new to LLMs.
1. Typically, if you are trying to have the model learn a new language or writing style (medical data), you may want to fine tune the tokenizer, not the model itself. Fine tuning the model should be done for the specific task - classification, summarization etc - not general language understanding.
2. You could add whatever layers you like to the model.
Full example on my GitHub:
```auto
classifier_layer_names += ['extra_class' + str(layer_ind)]
extra_classifiers.add_module(name=classifier_layer_names[-1],
module=nn.Linear(input_channels,
self.extra_class_layers[layer_ind]))
classifier_dropout_layer_names += ['extra_class_dropout' + str(layer_ind)]
extra_classifiers.add_module(name=classifier_dropout_layer_names[-1],
module=nn.Dropout(p=self.fine_tune_dropout_rate))
# Replacing, rather than adding a layer makes it so that forward() does not need to be extended
self.model.classifier = extra_classifiers
```
[https://github.com/bdzyubak/utils/blob/f0367f3c76cf38c790af43c25abc059d00557ba9/LLM\_pytorch\_lighting\_wrapper.py-](https://github.com/bdzyubak/utils/blob/f0367f3c76cf38c790af43c25abc059d00557ba9/LLM_pytorch_lighting_wrapper.py-)\>replace\_layers\_with\_fc()