# Cannot reproduce the results **URL:** **Author:** ![chz816](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@chz816](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchz816)%5C **Post date:** [October 2, 2020, 6:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-reproduce-the-results%2F1345%2F1 "2020-10-02T18:02:23Z") Hi I try to reproduce the result related to BART and the result is not comparable to the claimed performance. I tried `sshleifer/distilbart-cnn-12-6` and `facebook/bart-large-cnn` and met the same problem. My generation process is modified based on the released summarization pipeline. ```auto python run_eval.py sshleifer/distilbart-cnn-12-6 $DATA_DIR/test.source $OUTPUT_FILE \ --reference_path $DATA_DIR/test.target \ --task summarization \ --device cuda \ --fp16 \ --bs 32 ``` My performance without post-processing: ```auto 1 ROUGE-1 Average_R: 0.48286 (95%-conf.int. 0.48036 - 0.48554) 1 ROUGE-1 Average_P: 0.33581 (95%-conf.int. 0.33356 - 0.33802) 1 ROUGE-1 Average_F: 0.38536 (95%-conf.int. 0.38338 - 0.38737) --------------------------------------------- 1 ROUGE-2 Average_R: 0.20405 (95%-conf.int. 0.20148 - 0.20648) 1 ROUGE-2 Average_P: 0.14260 (95%-conf.int. 0.14067 - 0.14449) 1 ROUGE-2 Average_F: 0.16314 (95%-conf.int. 0.16108 - 0.16517) --------------------------------------------- 1 ROUGE-L Average_R: 0.40419 (95%-conf.int. 0.40174 - 0.40665) 1 ROUGE-L Average_P: 0.28191 (95%-conf.int. 0.27984 - 0.28396) 1 ROUGE-L Average_F: 0.32309 (95%-conf.int. 0.32111 - 0.32509) ``` My performance with post-posting (from ProphetNet): ```auto 1 ROUGE-1 Average_R: 0.49758 (95%-conf.int. 0.49505 - 0.50028) 1 ROUGE-1 Average_P: 0.35663 (95%-conf.int. 0.35421 - 0.35889) 1 ROUGE-1 Average_F: 0.40406 (95%-conf.int. 0.40200 - 0.40607) --------------------------------------------- 1 ROUGE-2 Average_R: 0.21882 (95%-conf.int. 0.21622 - 0.22125) 1 ROUGE-2 Average_P: 0.15750 (95%-conf.int. 0.15543 - 0.15947) 1 ROUGE-2 Average_F: 0.17794 (95%-conf.int. 0.17576 - 0.17998) --------------------------------------------- 1 ROUGE-L Average_R: 0.41627 (95%-conf.int. 0.41375 - 0.41881) 1 ROUGE-L Average_P: 0.29928 (95%-conf.int. 0.29712 - 0.30132) 1 ROUGE-L Average_F: 0.33860 (95%-conf.int. 0.33658 - 0.34056) ``` The expected performance for `sshleifer/distilbart-cnn-12-6` is ?/21.26/30.59 and I can only achieve 40.41/17.79/33.86. So is the trick related to the post-processing, or how can I achieve the expected performance? Thank you! ---
**Author:** ![chz816](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@chz816](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchz816)%5C **Post date:** [October 5, 2020, 2:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-reproduce-the-results%2F1345%2F2 "2020-10-05T02:34:49Z")
For anyone may see this post, the problem is solved by using larger batch size. Above result is using batch size 32 on GeForce 2080 Ti, and I change to Tesla V100 today with batch size 128. The result is pretty closed to expected: ```auto 1 ROUGE-1 Average_R: 0.53399 (95%-conf.int. 0.53146 - 0.53669) 1 ROUGE-1 Average_P: 0.39205 (95%-conf.int. 0.38963 - 0.39451) 1 ROUGE-1 Average_F: 0.44179 (95%-conf.int. 0.43972 - 0.44408) --------------------------------------------- 1 ROUGE-2 Average_R: 0.25584 (95%-conf.int. 0.25312 - 0.25867) 1 ROUGE-2 Average_P: 0.18821 (95%-conf.int. 0.18605 - 0.19056) 1 ROUGE-2 Average_F: 0.21172 (95%-conf.int. 0.20940 - 0.21420) --------------------------------------------- 1 ROUGE-L Average_R: 0.44976 (95%-conf.int. 0.44719 - 0.45244) 1 ROUGE-L Average_P: 0.33090 (95%-conf.int. 0.32864 - 0.33330) 1 ROUGE-L Average_F: 0.37251 (95%-conf.int. 0.37040 - 0.37465) ``` ---
**Author:** ![LiuYangyang](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@LiuYangyang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLiuYangyang)%5C **Post date:** [October 5, 2020, 6:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-reproduce-the-results%2F1345%2F3 "2020-10-05T06:57:03Z")
a little confused. why would batch-size affect result? ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C **Post date:** [October 5, 2020, 9:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-reproduce-the-results%2F1345%2F4 "2020-10-05T09:30:56Z")
Different batch size = different average gradients per batch. Depending on the total size of the data and the training time, this can have a big effect on final performance. Gradient accumulation should help with that. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C **Post date:** [October 5, 2020, 10:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-reproduce-the-results%2F1345%2F5 "2020-10-05T10:36:51Z")
cc @sshleifer This post might also help > [@Facebook/bart-large-cnn has a low rouge score on cnn\_dailymail](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffacebook-bart-large-cnn-has-a-low-rouge-score-on-cnn-dailymail%2F673%2F11)%3A > > I had similar issues reproducing the ROUGE scores of facebook/bart-large-cnn on the CNN/Daily Mail dataset. First I tried the evaluation script examples/seq2seq/eval.py with the follwoing arguments: export DATA\_DIR=cnn\_dm python3 ./run\_eval.py facebook/bart-large-cnn $DATA\_DIR/val.source dbart\_val\_generations.txt --reference\_path $DATA\_DIR/val.target --score\_path cnn\_rouge.json --task summarization --device cuda --max\_source\_length 1024 --max\_target\_length 56 --f… ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C **Post date:** [October 5, 2020, 1:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-reproduce-the-results%2F1345%2F6 "2020-10-05T13:25:41Z")
@valhalla I think that is a totally separate issue (and already fixed in `calculate_rouge_score` on master)