# Run\_summarization.py t5 model output inconsistent results

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-summarization-py-t5-model-output-inconsistent-results%2F55951%3E%5C
**Category:** Models\
**Created:** [September 22, 2023, 3:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-summarization-py-t5-model-output-inconsistent-results%2F55951 "2023-09-22T03:30:19Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mht0228](https://avatars.discourse-cdn.com/v4/letter/m/ac91a4/32.png) [@mht0228](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmht0228)%5C
**Post date:** [September 22, 2023, 3:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-summarization-py-t5-model-output-inconsistent-results%2F55951%2F1 "2023-09-22T03:30:19Z")

</div>

I am working on T5 finetune model , and I found the [t5-small-finetuned-samsum-en](https://huggingface.co/santiviquez/t5-small-finetuned-samsum-en) model on huggingface and the rouge metric on [samsum Benchmark (Summarization) | Papers With Code](https://paperswithcode.com/sota/summarization-on-samsum)  
It shows: (I have delete other models except t5-small-finetuned-samsum-en)

| Rank | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSUM | gen\_len | loss | Details | Year | Tags |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 12 | t5-small-finetuned-samsum-en | 40.039 | 15.85 | 31.808 | 36.089 | 18.107 | 2.192 | | 2022 | |

But when I use transformers examples code run\_summarization.py run the model I have download from huggingface [t5-small-finetuned-samsum-en] , The result is not consistent with above.  
My result is: my ROUGE-1 is 3.34 but the result above is 40.039, It is very different. Here is my result: (The value should \*100)  
\*\*\*\*\* eval metrics \*\*\*\*\*  
eval\_gen\_len = 12.2066  
eval\_loss = 7.7219  
eval\_rouge1\_high\_fmeasure = 0.0383  
eval\_rouge1\_high\_precision = 0.2029  
eval\_rouge1\_high\_recall = 0.0225  
eval\_rouge1\_low\_fmeasure = 0.0334  
eval\_rouge1\_low\_precision = 0.1792  
eval\_rouge1\_low\_recall = 0.0194  
eval\_rouge1\_mid\_fmeasure = 0.0358  
eval\_rouge1\_mid\_precision = 0.1909  
eval\_rouge1\_mid\_recall = 0.0209  
eval\_rouge2\_high\_fmeasure = 0.0021  
eval\_rouge2\_high\_precision = 0.0114  
eval\_rouge2\_high\_recall = 0.0012  
eval\_rouge2\_low\_fmeasure = 0.0011  
eval\_rouge2\_low\_precision = 0.006  
eval\_rouge2\_low\_recall = 0.0006  
eval\_rouge2\_mid\_fmeasure = 0.0016  
eval\_rouge2\_mid\_precision = 0.0086  
eval\_rouge2\_mid\_recall = 0.0009  
eval\_rougeL\_high\_fmeasure = 0.0334  
eval\_rougeL\_high\_precision = 0.1803  
eval\_rougeL\_high\_recall = 0.0197  
eval\_rougeL\_low\_fmeasure = 0.0293  
eval\_rougeL\_low\_precision = 0.1587  
eval\_rougeL\_low\_recall = 0.0171  
eval\_rougeL\_mid\_fmeasure = 0.0314  
eval\_rougeL\_mid\_precision = 0.1688  
eval\_rougeL\_mid\_recall = 0.0184  
eval\_rougeLsum\_high\_fmeasure = 0.0362  
eval\_rougeLsum\_high\_precision = 0.1951  
eval\_rougeLsum\_high\_recall = 0.0213  
eval\_rougeLsum\_low\_fmeasure = 0.0317  
eval\_rougeLsum\_low\_precision = 0.1708  
eval\_rougeLsum\_low\_recall = 0.0184  
eval\_rougeLsum\_mid\_fmeasure = 0.034  
eval\_rougeLsum\_mid\_precision = 0.1832  
eval\_rougeLsum\_mid\_recall = 0.0199  
eval\_runtime = 0:00:54.57  
eval\_samples = 818  
eval\_samples\_per\_second = 14.988  
eval\_steps\_per\_second = 14.988  
And I have not change the run\_summarization.py script ,  
My shell script is:  
python ./run\_summarization\_.py  
–model\_name\_or\_path $T5\_DIR  
–do\_eval  
–source\_prefix "summarize: "  
–output\_dir ./output/huggingface-summarization  
–dataset\_name $samsum\_path  
–dataset\_config “3.0.0”  
–per\_device\_train\_batch\_size=1  
–per\_device\_eval\_batch\_size=1  
–overwrite\_output\_dir  
–predict\_with\_generate \

Can anyone help me on this issue? Thanks a lot.