# T5 Model Problems - Constant Loss (doesn't go down)
**URL:**
**Author:**  [@pjahn89](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpjahn89)%5C
**Post date:** [July 14, 2021, 8:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-problems-constant-loss-doesnt-go-down%2F8365%2F1 "2021-07-14T20:58:42Z")
I’ve been trying to train T5 on a custom dataset similar to Squad v2 by modifying the T5 on TPU colab written by Suraj Patil. (I have Colab Pro so I train on a high-ram TPU instance). The link I have shared is training on Squad v2 directly, which seems to have the same problem.
link: [Google Colaboratory](https://colab.research.google.com/drive/1ktVUioN5oK7OhnSpPgAkuYiqk0Uuyke3?usp=sharing)
However, no matter how much I train the error seems to stay constant, i.e. the model does not seem to learn. This seems to be the case both for loss recorded during training, as well as loss during the valuation phase.
Could someone please tell me what I am doing wrong? I am going a little crazy trying to figure it out.
Thank you in advance all.
I’ve corrected the following issues thus far:
1. Different XLA import at start
2. Modification of code to allow for answer-less questions under Squadv2 (as opposed to Squadv1 for Suraj’s original code) under the eos/encoder section
3. Edited data imports to use huggingface’s datasets.load\_datasets instead of NLP
4. Under T2TDataCollator, modify batching to ensure that the inputs are tensors instead of lists, e.g.: torch.FloatTensor(example[‘input\_ids’]).to(torch.int64)
5. Specifying transformers version 2.9.1 to allow for Suraj’s particular usage of T5DataCollator (although I created a version using the current version of transformers, this also has the same problem described above).
[5b. If I use the current version of transformers and not 2.9.1, I make various modifications to T5DataCollator and the labels generated in the training phase to be (‘labels’, ‘decoder\_attention\_mask’ instead of ‘target\_ids’ and ‘target\_attention\_mask’]
---
**Author:**  [@prikmm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprikmm)%5C
**Post date:** [August 8, 2021, 7:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-problems-constant-loss-doesnt-go-down%2F8365%2F3 "2021-08-08T19:22:14Z")
Hey @pjahn89, I am facing a similar issue while trying to finetuning T5 on XSum using TPU/GPU. The training loss is not constant (it varies, but doesn’t converge). But, my validation loss is constant, like literally not even a change in 5th decimal place, I tried many things like creating my nn.Module compatible with the trainer. Subclassed the trainer to modify `compute_loss()`. But, I am not seeing any change. If you have solved this issue, can you please tell me how?
Here’s a [link](https://colab.research.google.com/drive/1KEweUQA8LRk_5VyRfAt04a7g1g6GpgrP?usp=sharing) to my colab for reference.
Thank you!
---
**Author:**  [@EnzoDefi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEnzoDefi)%5C
**Post date:** [August 18, 2023, 8:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-problems-constant-loss-doesnt-go-down%2F8365%2F4 "2023-08-18T20:14:59Z")
Have you solved this issue? Because I am having the same problem