# Training the t5
**URL:**
**Author:**  [@Samuel-Fipps](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSamuel-Fipps)%5C
**Post date:** [August 16, 2022, 2:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-the-t5%2F21657%2F1 "2022-08-16T02:23:33Z")
Hello, on the T5 page it says to train it using just
> loss = model(input\_ids=input\_ids, lm\_labels=labels).loss
> This did nothing but give me a loss value.
However when i do this [Summarization](https://huggingface.co/docs/transformers/tasks/summarization)
I get the error text, and summary are not recognized.
---
**Author:**  [@sason](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsason)%5C
**Post date:** [August 16, 2022, 5:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-the-t5%2F21657%2F2 "2022-08-16T05:25:59Z")
Could you please show your code and the respective error?
---
**Author:**  [@Samuel-Fipps](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSamuel-Fipps)%5C
**Post date:** [August 16, 2022, 12:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-the-t5%2F21657%2F3 "2022-08-16T12:54:07Z")
> from datasets import load\_dataset
> from transformers import T5Tokenizer, T5ForConditionalGeneration
> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer
>
> billsum = load\_dataset(“billsum”, split=“ca\_test”)
> billsum = billsum.train\_test\_split(test\_size=0.2)
> from transformers import AutoTokenizer
>
> tokenizer = T5Tokenizer.from\_pretrained(“google/t5-efficient-xxl-nl4”)
> model = AutoModelForSeq2SeqLM.from\_pretrained(“/media/samuel/New Volume/models/t5-efficient-xxl-nl4”)
>
> prefix = "summarize: "
>
> def preprocess\_function(examples):
> inputs = [prefix + doc for doc in examples[“text”]]
> model\_inputs = tokenizer(inputs, max\_length=1024, truncation=True)
>
> ```
> with tokenizer.as_target_tokenizer():
> labels = tokenizer(examples["summary"], max_length=128, truncation=True)
>
> model_inputs["labels"] = labels["input_ids"]
> return model_inputs
>
> ```
>
> tokenized\_billsum = billsum.map(preprocess\_function, batched=True)
>
> from transformers import DataCollatorForSeq2Seq
>
> data\_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)
>
> training\_args = Seq2SeqTrainingArguments(
> output\_dir=“./results”,
> evaluation\_strategy=“epoch”,
> learning\_rate=2e-5,
> per\_device\_train\_batch\_size=16,
> per\_device\_eval\_batch\_size=16,
> weight\_decay=0.01,
> save\_total\_limit=3,
> num\_train\_epochs=1,
> fp16=True,
> )
>
> trainer = Seq2SeqTrainer(
> model=model,
> args=training\_args,
> train\_dataset=tokenized\_billsum[“train”],
> eval\_dataset=tokenized\_billsum[“test”],
> tokenizer=tokenizer,
> data\_collator=data\_collator,
> )
>
> trainer.train()
Errior:
> The following columns in the training set don’t have a corresponding argument in `T5ForConditionalGeneration.forward` and have been ignored: title, summary, text. If title, summary, text are not expected by `T5ForConditionalGeneration.forward`, you can safely ignore this message.
---
**Author:**  [@Samuel-Fipps](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSamuel-Fipps)%5C
**Post date:** [August 16, 2022, 1:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-the-t5%2F21657%2F4 "2022-08-16T13:09:58Z")
I believe that error is fine but it doesn’t change my model when i train on this.
the training loss is 0.0
and the eval loss is nan
---
**Author:**  [@Samuel-Fipps](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSamuel-Fipps)%5C
**Post date:** [August 16, 2022, 6:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-the-t5%2F21657%2F5 "2022-08-16T18:14:51Z")
I figured it out. you get the loss value from the model and call
> loss = model(input\_ids=input\_ids, lm\_labels=labels).loss
> optimizer.zero\_grad()
> loss.backward()
> optimizer.step()
I just figure it would do this part for you