# Show Submodels of PegasusTokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshow-submodels-of-pegasustokenizer%2F17286%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [April 28, 2022, 12:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshow-submodels-of-pegasustokenizer%2F17286 "2022-04-28T12:33:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rmeier](https://avatars.discourse-cdn.com/v4/letter/r/45deac/32.png) [@rmeier](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frmeier)%5C
**Post date:** [April 28, 2022, 12:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshow-submodels-of-pegasustokenizer%2F17286%2F1 "2022-04-28T12:33:33Z")

</div>

Hi,

for understanding the process of the PegasusTokenizer, I would like to print the different outputs of the Tokenizer after each submodule (Normalization, Pre Tokenization and so on.

I used the google/pegasus-cnn\_dailymail model with the PegasusTokenizer.  
Is there s way to show on one data example how the Tokenizer transforms the input text to the output integer sequences?

I would like to see the transformation step wise for each step the Tokenizer is performing?

It would be great if somebody can point me in one direction because it is very urgent.

Thanks  
Ralf

---

<div class="post-metadata">

**Author:** ![rmeier](https://avatars.discourse-cdn.com/v4/letter/r/45deac/32.png) [@rmeier](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frmeier)%5C
**Post date:** [April 28, 2022, 1:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshow-submodels-of-pegasustokenizer%2F17286%2F2 "2022-04-28T13:01:47Z")

</div>

I have some ideas but I don’t think this is the right solution?

```auto
import sentencepiece as spm

s = spm.SentencePieceProcessor(tokenizer.vocab_file)
str(s.encode(sample_text, out_type=str, enable_sampling=True, alpha=0.1, nbest_size=-1))[:1000]

```