# Fine-tuning T5 with custom datasets
**URL:**
**Author:**  [@TheLongSentance](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTheLongSentance)%5C
**Post date:** [July 30, 2021, 6:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F1 "2021-07-30T18:34:10Z")
Hi folks,
I am a newbie to T5 and transformers in general so apologies in advance for any stupidity or incorrect assumptions on my part!
I am trying to put together an example of fine-tuning the T5 model to use a custom dataset for a custom task. I have the “[How to fine-tune a model on summarization](https://github.com/huggingface/notebooks/blob/master/examples/summarization.ipynb)” example notebook working but that example uses a pre-configured HF dataset via “load\_dataset()” not a custom dataset that I load from disk. So I was wanting to combine that example with the guidance given at “[Fine-tuning with custom datasets](https://huggingface.co/transformers/master/custom_datasets.html)” but with T5 and not DistilBert as in the fine-tuning example shown.
I think my main problem is knowing how to construct a dataset object that the pre-configured T5 model can consume. So here is my use of the tokenizer and my attempt at formating the tokenized sequencies into datasets:
But I get the following error back when I call trainer.train():
I have seen the post “[Defining a custom dataset for fine-tuning translation](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdefining-a-custom-dataset-for-fine-tuning-translation%2F6913)%E2%80%9D but the solution offered there seems to be write your own custom Dataset loading class rather than directly providing a solution to the problem - I can try to learn/do this but it would be great to get this working equivalent to “Fine-tuning with custom datasets” but for the T5 model I want to use.
I also found “[Fine Tuning Transformer for Summary Generation](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)” which is where I got the idea to change the **getitem** method of my ToxicDataset class to return “input\_ids” “input\_mask” “output\_ids” “output\_mask” but I am guessing really, I can’t find any documentation of what is needed (sorry!).
Any help or pointers to find what I need would be very much appreciated!
---
**Author:**  [@TheLongSentance](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTheLongSentance)%5C
**Post date:** [August 3, 2021, 9:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F2 "2021-08-03T09:11:56Z")
I think I may have found a way around this issue (or at least the trainer starts and completes!). The subclassing of a `torch.utils.data.Dataset` object for the distilbert example in “[Fine-tuning with custom datasets](https://huggingface.co/transformers/master/custom_datasets.html)” needs changing as follows. I guess because the distilbert model provides just a list of integers whereas the T5 model has output texts and I assume the DataCollatorForSeq2Seq() takes care of preprocessing the labels (the output encodings) into the features needed by forward function of T5 model (I am guessing, but this is what I am assuming from what I have read). Code changes below:
---
**Author:**  [@TheLongSentance](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTheLongSentance)%5C
**Post date:** [August 4, 2021, 9:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F3 "2021-08-04T09:39:41Z")
Response via github from sgugger:
_This tutorial is out of date and will be rewritten soon. You should have a look at the [maintained examples](https://github.com/huggingface/transformers/tree/master/examples) or the [example notebooks](https://huggingface.co/transformers/notebooks.html) instead._
---
**Author:**  [@alexyalunin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falexyalunin)%5C
**Post date:** [October 29, 2021, 9:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F4 "2021-10-29T09:42:35Z")
So what example in the links above is appropriate for fine-tuning T5?
---
**Author:**  [@TheLongSentance](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTheLongSentance)%5C
**Post date:** [October 29, 2021, 10:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F5 "2021-10-29T10:07:11Z")
Clone the transformers repo and I used transformers/examples/pytorch/summarization/run\_summarization.py. It is not limited to summarization with T5, just configure the input/output text to what you need.
---
**Author:**  [@sheoran95](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsheoran95)%5C
**Post date:** [February 28, 2023, 3:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F6 "2023-02-28T15:21:10Z")
Hey! Can you pls share the notebook or code snippet that helped you finally train the custom dataset with T5. I’ve been struggling with the same problem. Please help!
It would be great if you you could share how you create dataset, loader and finally train it.
---
**Author:**  [@TheLongSentance](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTheLongSentance)%5C
**Post date:** [February 28, 2023, 6:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F7 "2023-02-28T18:55:30Z")
Hi @sheoran95, well I did this about a year back (so there might be something better/different now, please check out other options too) but as I said in my post above, the example code I got running was **run\_summarization.py** from [transformers/examples/pytorch/summarization at main · huggingface/transformers (github.com)](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization). Take a look at the **README.md** for how to get started with a example dataset. You want the T5ForConditionalGeneration model and note you need to provide the --source\_prefix "summarize: " parameter (which you can later change to other T5 supported prefixes to do other things).
---
**Author:**  [@sheoran95](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsheoran95)%5C
**Post date:** [March 1, 2023, 8:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F8 "2023-03-01T08:24:54Z")
Thanks for such a fast response!
Sure, I’ll check this out.
I actually tried the code changes you mentioned previously and it got my Trainer working.
I was facing an error: “ValueError: You have to specify either decoder\_input\_ids or decoder\_inputs\_embeds” but after using AutoModelForSeq2SeqLM instead of AutoModel, it was also resolved.
Is there any specific reason why you shifted to the run\_summarization.py file if your Trainer was working?
---
**Author:**  [@TheLongSentance](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTheLongSentance)%5C
**Post date:** [March 2, 2023, 5:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F9 "2023-03-02T17:08:39Z")
I switched to that file because the trainer displayed on the hugging face website was out of date and didn’t work, and the Hugging Face folks said that run\_summarization.py was working as it is maintained as part of the Transformers repository. Hope that provides some explanation.
---
**Author:**  [@sheoran95](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsheoran95)%5C
**Post date:** [March 13, 2023, 5:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F10 "2023-03-13T05:19:31Z")
Thank you!
---
**Author:**  [@sachin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsachin)%5C
**Post date:** [March 13, 2023, 7:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F11 "2023-03-13T07:32:13Z")
@sheoran95 in case you still need help, this was how I trained it using pytorch lightning: [T5 for grammar correction | Kaggle](https://www.kaggle.com/code/sachin/t5-for-grammar-correction)
---
**Author:**  [@sheoran95](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsheoran95)%5C
**Post date:** [March 13, 2023, 8:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F12 "2023-03-13T08:02:38Z")
Thanks for sharing Sachin!
---
**Author:**  [@arohablue](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farohablue)%5C
**Post date:** [August 29, 2024, 7:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-t5-with-custom-datasets%2F8858%2F13 "2024-08-29T19:36:42Z")
Check out this repo how i training codeT5+ on custom data : [GitHub - arohablue/ai-code-generators-data-poisoning: Exploring Targeted Data Poisoning Attacks on Code Generators](https://github.com/arohablue/ai-code-generators-data-poisoning)