# \`\`\`google/pegasus-cnn\_dailymail\`\`\` generates blank files

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgoogle-pegasus-cnn-dailymail-generates-blank-files%2F5535%3E%5C
**Category:** Models\
**Created:** [April 15, 2021, 6:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgoogle-pegasus-cnn-dailymail-generates-blank-files%2F5535 "2021-04-15T18:25:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![chz816](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@chz816](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchz816)%5C
**Post date:** [April 15, 2021, 6:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgoogle-pegasus-cnn-dailymail-generates-blank-files%2F5535%2F1 "2021-04-15T18:25:28Z")

</div>

I am working on generating summaries using SOTA models. Recently I am using pegasus-related models to generate results. It surprises me that `google/pegasus-xsum` returns the perfect results while `google/pegasus-cnn_dailymail` returns very bad performance.

I am running the code from [transformers/examples/legacy/seq2seq at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/master/examples/legacy/seq2seq), and my transformers’ version is 4.2.0. The detailed implementation is:

```bash
python -m torch.distributed.launch --nproc_per_node=3 run_distributed_eval.py \
    --model_name google/pegasus-cnn_dailymail \
    --save_dir $OUTPUT_DIR \
    --data_dir $DATA_DIR \
    --bs 32 \
    --fp16

```

The performance from `google/pegasus-xsum`:

```auto
{'rouge1': 47.0271, 'rouge2': 24.4924, 'rougeL': 39.2529}

```

The performance from `google/pegasus-cnn_dailymail `:

```auto
{'rouge1': 0.1602, 'rouge2': 0.084, 'rougeL': 0.1134}

```

I also check the `test_generations.txt` file to try to figure out the reasons for abnormal low ROUGE score. It turns out most of the lines are blank. Does anyone meet similar problems when trying to generate summaries using `google/pegasus-cnn_dailymail`?