# Pegasus Questions

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 24, 2020, 2:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838 "2020-08-24T14:56:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [August 24, 2020, 2:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F1 "2020-08-24T14:56:22Z")

</div>

Q: Max model input size varies between checkpoints, what is the max num input tokens that each model can process?  
A:

```auto
max_model_length = {
    "xsum": 512,
    "cnn_dailymail": 1024,
    "newsroom": 512,
    "wikihow": 512,
    "multi_news": 1024,
    "reddit_tifu": 512,
    "big_patent": 1024,
    "arxiv": 1024,
    "pubmed": 1024,
    "gigaword": 128,
    "aeslc": 512,
    "billsum": 1024,
    "large": 1024,
}

```

that constant is defined here:

> <https://github.com/sshleifer/transformers_fork/blob/f69cac3347641beaba5037b9a6fca1a46f423639/src/transformers/configuration_pegasus.py#L67-L67>

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [August 24, 2020, 2:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F2 "2020-08-24T14:59:48Z")

</div>

So does that mean `max_position_embedding` is reduced for fine-tuned models (gigaword, wikihow). i.e if the `max_position_embedding` for the pre-trained model is 1024 then all fine-tuned models should also have same, right ?

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [August 24, 2020, 3:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F3 "2020-08-24T15:42:51Z")

</div>

The positional embeddings are static, so the max model\_input\_length is more a reflection of how the model was finetuned than a fundamental limitation.

I got the params from: [https://github.com/google-research/pegasus/blob/master/pegasus/params/public\_params.py](https://github.com/google-research/pegasus/blob/master/pegasus/params/public_params.py)

For some of the saved `state_dict`s on S3, I simply removed the `embed_positions.weight` to allow passing `max_position_embeddings=ANY_INT` to `from_pretrained` allocate more at **init**. Let me know if you try that and it fails and I can delete the `embed_positions.weight` as needed.

---

<div class="post-metadata">

**Author:** ![Buckeyes2019](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Buckeyes2019](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBuckeyes2019)%5C
**Post date:** [August 24, 2020, 3:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F4 "2020-08-24T15:47:49Z")

</div>

## Was suggested I re-direct my questions from this thread: [Questions about Pegasus for Summarization](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-about-pegasus-for-summarization%2F841)

Two questions: (1) does anyone have good examples of the google/pegasus models being used for summarization using the Automodal and Autotokenizer classes?

(2) Using the google/pegasus models, how do you set the minimum and/or maximum length for the generated summary?

Thanks!

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [August 24, 2020, 7:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F5 "2020-08-24T19:50:37Z")

</div>

Thanks for redirecting

[Docs](https://huggingface.co/transformers/master/model_doc/pegasus.html#usage-example) have a usage example, that I’ll put here

```auto
import torch
from transformers import PegasusForConditionalGeneration, PegasusTokenizer
src_text = [
    """ PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."""
]

model_name = 'google/pegasus-xsum'
torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = PegasusTokenizer.from_pretrained(model_name)
model = PegasusForConditionalGeneration.from_pretrained(model_name).to(torch_device)
batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest').to(torch_device)
translated = model.generate(**batch)
tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
assert tgt_text[0] == "California's largest electricity provider has turned off power to tens of thousands of customers."

```

For all seq2seq/ForConditionalGeneration models, you can control generated summary lengths with the `min_length` and `max_length` keyword arguments to the generate function.

If you don’t pass them, the (good) defaults from config will be used. The defaults from config were set by the authors to get a good validation rouge on the finetuning dataset.

---

<div class="post-metadata">

**Author:** ![LiuYangyang](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@LiuYangyang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLiuYangyang)%5C
**Post date:** [August 25, 2020, 1:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F6 "2020-08-25T01:46:52Z")

</div>

In the original paper Section 6:

> We used sinusoidal positional encoding

and Section 6.2:

> This would present a problem for position embeddings which would never be updated for longer input lengths, but we confirm the postulation that sinusoidal positional encodings (Vaswani et al., 2017) generalize well when fine-tuning PEGASUSLARGE beyond the input lengths observed in training up to Linput = 1024 tokens.

So you can use longer input than training, but you have to observe if model still generalize with longer input.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [August 25, 2020, 8:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F7 "2020-08-25T08:58:45Z")

</div>

So if we want to use longer input we’ll need to resize the embeddings (as these are static) and these newly added embeddings will be randomly initialised and need to be trained.

Is this correct ?

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [August 25, 2020, 3:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F8 "2020-08-25T15:18:49Z")

</div>

Ideally you won’t need to manually resize, and can just pass `max_position_embeddings` as a kwarg to `from_pretrained`

---

<div class="post-metadata">

**Author:** ![tuner007](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tuner007/32/2168_2.png) [@tuner007](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftuner007)%5C
**Post date:** [August 27, 2020, 8:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F9 "2020-08-27T20:54:36Z")

</div>

Something is wrong ? 🤔**UPDATE : Normal behavior…as pubmed dataset(biomedical text) is used for finetuning**

**Text :** The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.

[`google/pegasus-pubmed`](https://huggingface.co/google/pegasus-pubmed) **summary** :  
the tower of so paulo ( tower of so paulo , brazil ) is the tallest building in the brazilian amazon . its height is determined by measuring the distance from the top of the building to the base of the building . the building is located in so paulo , brazil , on the so paulo river . its height is determined by measuring the distance from the top of the building to the base of the building .

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [August 28, 2020, 12:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F10 "2020-08-28T00:18:26Z")

</div>

I cant tell from that. It might be so different from a pubmed article that the model gets confused. Try finding the pubmed dataset and pasting in an example.

---

<div class="post-metadata">

**Author:** ![tuner007](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tuner007/32/2168_2.png) [@tuner007](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftuner007)%5C
**Post date:** [August 28, 2020, 6:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F11 "2020-08-28T18:46:51Z")

</div>

Yes ! tested with authors script/finetuned checkpoints and got similar results…

Thanks !

---

<div class="post-metadata">

**Author:** ![tuner007](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tuner007/32/2168_2.png) [@tuner007](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftuner007)%5C
**Post date:** [September 12, 2020, 2:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F12 "2020-09-12T14:11:49Z")

</div>

@sshleifer I was trying to convert pegasus TF checkpoint to pytorch using [this](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_pegasus_tf_to_pytorch.py) script in transformers.

Can you please help me with below queries:

1.While replacing the keys i noticed below replacement which is not matching with BART keys so i have changed it accordingly:  
**Pegasus:**  
decoder/LayerNorm/beta -\> decoder.layer\_norm.bias  
decoder/LayerNorm/gamma -\> decoder.layer\_norm.weight  
**BART:**  
decoder.layernorm\_embedding.weight  
decoder.layernorm\_embedding.bias

2.In below code all pegasus dict\_keys should be present in BART keys right ? but as Pegasus has more number of layers compared to BART…so it will never get keys like “decoder.layers.12.final\_layer\_norm.bias” in BART keys

```auto
def convert_pegasus_to_bart(tf_weights: dict, cfg_updates: dict) -> PegasusForConditionalGeneration:
    cfg_kwargs = DEFAULTS.copy()
    cfg_kwargs.update(cfg_updates)
    cfg = PegasusConfig(**cfg_updates)
    bart = PegasusForConditionalGeneration(cfg)
    sd = bart.model.state_dict()
    mapping = {}
    for k, v in tf_weights.items():
        new_k = rename_state_dict_key(k)
        if new_k not in sd:
            raise ValueError(f"could not find new key {new_k} in state dict. (converted from {k})")
        if "dense" in k or "proj" in new_k:
            v = v.T

```

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [September 12, 2020, 5:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F13 "2020-09-12T17:36:30Z")

</div>

1. `rename_state_dict_key` should handle that
2. `cfg.encoder_layers` will be 16 there, so `decoder.layers.12` will definitely exist.

If the code breaks, pls send a command and traceback.

---

<div class="post-metadata">

**Author:** ![tuner007](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tuner007/32/2168_2.png) [@tuner007](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftuner007)%5C
**Post date:** [September 12, 2020, 6:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F14 "2020-09-12T18:01:45Z")

</div>

**(update: works for me now)**  
cfg\_kwargs.update(cfg\_updates)  
cfg = PegasusConfig(\*\ ***cfg\_updates** ) -\> cfg = PegasusConfig(\*\ ***cfg\_kwargs** ) 😅

Also, even got an error with model name in tokenizer which i later changed from “sshleifer/pegasus” to “google/pegasus-aeslc”  
**command:** `python convert_pegasus_tf_to_pytorch.py gs://pegasus_ckpt/arxiv/model.ckpt-32000 save_dir`

Thanks !

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [September 12, 2020, 8:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F15 "2020-09-12T20:41:45Z")

</div>

great catch, PR’d fix [https://github.com/huggingface/transformers/pull/7094](https://github.com/huggingface/transformers/pull/7094)

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [September 12, 2020, 8:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F16 "2020-09-12T20:42:03Z")

</div>

You may know this but we already have

`google/pegasus-arxiv` available fwiw

---

<div class="post-metadata">

**Author:** ![tuner007](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tuner007/32/2168_2.png) [@tuner007](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftuner007)%5C
**Post date:** [September 12, 2020, 8:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F17 "2020-09-12T20:55:47Z")

</div>

Yes, actually i wanted to add pegasus for qa and paraphrasing 🙂

---

<div class="post-metadata">

**Author:** ![LiuYangyang](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@LiuYangyang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLiuYangyang)%5C
**Post date:** [September 15, 2020, 2:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F18 "2020-09-15T14:44:06Z")

</div>

Are there any script available to distill model and how long does it take? I need the small model trained on multi-news, but found nothing.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 15, 2020, 3:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F19 "2020-09-15T15:03:19Z")

</div>

Pegasus distillation is now supported in the seq2seq examples.  
[https://github.com/huggingface/transformers/tree/master/examples/seq2seq](https://github.com/huggingface/transformers/tree/master/examples/seq2seq)

You’ll need to pass pegaus model path instead of bart and multi-news directory formatted according to the format specified in the readme.

---

<div class="post-metadata">

**Author:** ![Jeeves](https://avatars.discourse-cdn.com/v4/letter/j/73ab20/32.png) [@Jeeves](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJeeves)%5C
**Post date:** [November 3, 2020, 7:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838%2F20 "2020-11-03T07:18:38Z")

</div>

Hi Sshleifer,

@sshleifer Sorry in advance if this is a silly question. When inferring on pegasus-arxiv, specially on long documents like the Arxiv dataset, can we only infer on the maximum input length (ie: 1024) or the whole document at once. Or do we have to send in the input in batches according to max length of 1024?

[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-questions%2F838.md%3Fpage%3D2)