# Subclassing a pretrained model for a new objective
**URL:**
**Author:**  [@erickrf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ferickrf)%5C
**Post date:** [October 5, 2021, 9:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F1 "2021-10-05T09:00:21Z")
I would like to use a pretrained model as an encoder for a new task. It is essentially multiple sequence classification objectives, like in the `...ForSequenceClassification` models, but with an output layer for each subtask.
I could just create wrappers around the encoder, but I’d like to subclass from `PreTrainedModel` to better integrate with the Trainer class. How exactly should I do? Do I need to create a config class as well? I will at least need to supply an extra list or dict to the config telling how many classes each subtask has.
Thanks!
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 5, 2021, 12:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F2 "2021-10-05T12:03:51Z")
You can definitely subclass `PretrainedConfig` for your custom config and `PreTrainedModel` for your custom model, then access all the methods of the library.
---
**Author:**  [@erickrf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ferickrf)%5C
**Post date:** [October 5, 2021, 12:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F3 "2021-10-05T12:05:30Z")
@sgugger thanks! But in that case what is needed to make methods like `from_pretrained` work out of the box? I saw that the pretrained model classes have a class attribute called `config_class`, is setting that enough?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 5, 2021, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F4 "2021-10-05T12:08:17Z")
It’s to find the right config in the Transformers library. In your case, you might have to use two steps:
```auto
config = CustomConfig.from_pretrained(path_to_folder_with_config_and_weights)
model = CustomModel.from_pretrained(path_to_folder_with_config_and_weights, config)
```
---
**Author:**  [@erickrf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ferickrf)%5C
**Post date:** [October 5, 2021, 1:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F5 "2021-10-05T13:42:26Z")
Ok. But how can I load the pretrained model (i.e., the encoder inside my class)?
I tried doing `CustomModel.from_pretrained(path_to_pretrained, additional_config_data)`, but that ignored all the weights in the checkpoint (name mismatches, I suppose?).
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 5, 2021, 2:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F6 "2021-10-05T14:02:30Z")
Did you save the corresponding model with `save_pretrained`?
---
**Author:**  [@erickrf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ferickrf)%5C
**Post date:** [October 5, 2021, 2:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F7 "2021-10-05T14:10:30Z")
Nope, I haven’t even fine tuned the model yet 🙂
I’m calling `from_pretrained` in the encoder directly, after creating the classifier object and before training, but that looks hacky.
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 5, 2021, 2:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F8 "2021-10-05T14:54:23Z")
I’m not sure what you want to do, but calling `from_pretrained` on your class with weights saved from another class will not work. If you want to use a pretrained model for part of the your custom model, you should use the `from_pretrained` method when defining that part of your custom model.
---
**Author:**  [@chrico](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchrico)%5C
**Post date:** [August 10, 2022, 5:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsubclassing-a-pretrained-model-for-a-new-objective%2F10521%2F9 "2022-08-10T17:54:53Z")
Would the following implementation work as expected (by which I mean utilize the model weights associated with the Huggingface location parameter, etc.)? I couldn’t figure out how to subclass using AutoModelForSequenceClassification and am hoping this is valid.
```auto
class PretrainedForSequenceCustom(BertForSequenceClassification):
def __init__ (self, num_classes):
# initialize
cfg = AutoConfig.from_pretrained(HUGGINGFACE_LOCATION)
super(). __init__ (cfg)
# redefine bert to use model loaded with AutoModel
self.bert = AutoModel.from_pretrained(HUGGINGFACE_LOCATION)
# redefine classifier and re-run post-initialization
self.classifier = CustomLayer(size_in=cfg.hidden_size, num_classes=num_classes)
self.post_init()
def forward(
self,
input_ids: Optional[torch.Tensor] = None,
attention_mask: Optional[torch.Tensor] = None,
token_type_ids: Optional[torch.Tensor] = None,
position_ids: Optional[torch.Tensor] = None,
head_mask: Optional[torch.Tensor] = None,
inputs_embeds: Optional[torch.Tensor] = None,
labels: Optional[torch.Tensor] = None,
output_attentions: Optional[bool] = None,
output_hidden_states: Optional[bool] = None,
return_dict: Optional[bool] = None,
):
outputs = self.bert(
input_ids,
attention_mask=attention_mask,
token_type_ids=token_type_ids,
position_ids=position_ids,
head_mask=head_mask,
inputs_embeds=inputs_embeds,
output_attentions=output_attentions,
output_hidden_states=output_hidden_states,
return_dict=return_dict,
)
pooled_output = outputs[1]
pooled_output = self.dropout(pooled_output)
logits = self.classifier(pooled_output)
probas = torch.sigmoid(logits)
return logits, probas
def change_bert_grad_mode(self, mode=None):
for param in self.bert.parameters():
if mode is None:
param.requires_grad = not param.requires_grad
else:
param.requires_grad = mode
```