# T5 model for summarization far from SOTA results

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-for-summarization-far-from-sota-results%2F7934%3E%5C
**Category:** Models\
**Created:** [July 2, 2021, 8:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-for-summarization-far-from-sota-results%2F7934 "2021-07-02T20:26:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![cvillela](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@cvillela](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcvillela)%5C
**Post date:** [July 2, 2021, 8:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-for-summarization-far-from-sota-results%2F7934%2F1 "2021-07-02T20:26:00Z")

</div>

Hello!

I am having a hard time achieving State of the Art results after fine-tuning T5-base for text summarization.

I am trying the full implementation with the Transformers library (meaning, using the Seq2SeqTrainer() class). I am also using the XSum Dataset for fine-tuning the model.

I have preprocessed the dataset according to the documentation:

- "summarize: " prefix, and " " token at the end of the label texts
- max\_input\_length == 512

I am also only using the attention\_mask provided by the tokenization of the input sequence.

As to my training arguments:

- I have tried using both AdamW and Adafactor, with a learning rate of 3e-4 and weight\_decay of 5e-5.
- my batch\_size is currently 4, with gradient\_accumulation\_steps = 64, eval\_accumulation\_steps = 64
- I am using predict\_with\_generate = True, for obvious reasons
- For what I’ve read, FP16 had performance problems on this model, and so fp16=False.

Anyhow, I am only scoring aproximately 28 at the ROUGE-1 Score, where as the model reached ~43 ROUGE1 on the paper, even after I purposedly used 32 training examples in 100 epochs of training to force overfitting.

The Summarization example notebook by Huggingface also reaches only about 28 at the ROUGE1 Score.

Are there any tips for the fine tune? Should I implement my own trainer with PyTorch Lightning?  
I have already checked the discussion at [T5 Finetuning Tips](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-finetuning-tips%2F684)%2C but the results I’m getting aren’t improving at all.

Thanks a lot!