# TFT5ForConditionalGeneration with custom loss

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftft5forconditionalgeneration-with-custom-loss%2F16457%3E%5C
**Category:** Beginners\
**Created:** [April 4, 2022, 3:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftft5forconditionalgeneration-with-custom-loss%2F16457 "2022-04-04T15:19:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![BenM61](https://avatars.discourse-cdn.com/v4/letter/b/b19c9b/32.png) [@BenM61](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBenM61)%5C
**Post date:** [April 4, 2022, 3:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftft5forconditionalgeneration-with-custom-loss%2F16457%2F1 "2022-04-04T15:19:07Z")

</div>

Hey, I want to fine-tune `TFT5ForConditionalGeneration`. Very similar to [this thread](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-user-defined-loss-function%2F566%2F4)%2C but I want to use the teacher forcing (i.e. use the `forward`, rather than `generate` method).

One thing that crossed my mind is to inherit from `TFT5ForConditionalGeneration`, override the forward with a forward exactly like the original (from [the source code](https://github.com/huggingface/transformers/blob/77321481247787c97568c3b9f64b19e22351bab8/src/transformers/models/t5/modeling_t5.py)), but change the loss to my loss. Is that a good idea?

I will add more context below, but my question is how to do something else than `CrossEntropyLoss` for the t5 model loss.

**Context:**  
My task is- given tokenized song lyrics, classify the genres of that song (multilabel classification).

The output I want from t5 is a string containing all the genres predicted, followed by eos token, and than padding tokens to keep all samples with equal length (some songs has more than one genre). For example

```auto
'FUNK,POP,,,</s> <pad> <pad> <pad> <pad> <pad>',

```

When I used the default t5 loss (training via forward), I decreased the loss during training but the eval loss is bad and it predicts nonsense. I think the model can just learn where to place the paddings to get a loss improvement and that’s not what I want.

I’m not sure how to implement differential loss while ignore the pads, but for now I only ask how to change the loss in general.

I can add the code if you want, Thanks