# T5 generate() output doesn't produce \<extra\_id\_0\>

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-generate-output-doesnt-produce-extra-id-0%2F9403%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 22, 2021, 6:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-generate-output-doesnt-produce-extra-id-0%2F9403 "2021-08-22T18:51:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Niloofar](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Niloofar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNiloofar)%5C
**Post date:** [August 22, 2021, 6:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-generate-output-doesnt-produce-extra-id-0%2F9403%2F1 "2021-08-22T18:51:50Z")

</div>

Hi,

From the tutorial and my understanding, in the unsupervised denoising training, the model is working like two complementary pieces of the puzzle. Given the example in the tutorial:  
‘The \<extra\_id\_0\> walks in \<extra\_id\_1\> park’  
spans like ‘The’ and ‘walks in’ and ‘park’, apart from being one or multiple tokens are replaced with sentinel tokens. and the sentinel tokens ‘\<extra\_id\_0\>’ and ‘\<extra\_id\_1\>’ are replaced with a predicted tokens again one or more.  
so the expected generated output is ‘\<extra\_id\_0\> cute dog \<extra\_id\_1\> the \<extra\_id\_2\>’.

The problem is where I have tested a sentence in which the generated output doesn’t start with \<extra\_id\_0\> and the first sentinel token is \<extra\_id\_1\>. after few trials I found out that the space after to before a ‘.’ cause this problem. I will share the exact sentence that I generated and you can recreate that.

from transformers import T5ForConditionalGeneration, T5Tokenizer

model = T5ForConditionalGeneration.from\_pretrained(“t5-small”)  
tokenizer = T5Tokenizer.from\_pretrained(“t5-small”)

input = tokenizer(‘Rate leap into darkness 1 points .\<extra\_id\_0\> object\<extra\_id\_1\> rating\<extra\_id\_2\> point . RateBook’, return\_tensors=‘pt’).input\_ids  
output = model.generate(input\_ids=input, num\_beams=3, num\_return\_sequences=3,max\_length=input.size(1))

for i in range(output.size(0)):  
print(tokenizer.decode(output[i]))

#######output#######  
Rate\<extra\_id\_1\> rating\<extra\_id\_2\> 1\<extra\_id\_3\> rating rating 1\<extra\_id\_4\> 1\<extra\_id\_5\> Rate\<extra\_id\_6\> rating 1\<extra\_id\_7\> Rate  
Rate\<extra\_id\_1\> rating\<extra\_id\_2\> 1\<extra\_id\_3\> rating rating 1\<extra\_id\_4\> 1\<extra\_id\_5\> Rate\<extra\_id\_6\> rating rating 1\<extra\_id\_7\>  
Rate\<extra\_id\_1\> rating\<extra\_id\_2\> 1\<extra\_id\_3\> Rating\<extra\_id\_4\> 1\<extra\_id\_5\> Rate\<extra\_id\_6\> rating rating 1\<extra\_id\_7\> Rate\<extra\_id\_8\>  
###################

but if I remove the space before ‘RateBook’ then the output starts with \<extra\_id\_0\>.  
to test you can replace the sentence below:  
‘Rate leap into darkness 1 points .\<extra\_id\_0\> object\<extra\_id\_1\> rating\<extra\_id\_2\> point .RateBook’

the output is as follow:

\<extra\_id\_0\>RateBook\<extra\_id\_1\> rating\<extra\_id\_2\> 1\<extra\_id\_3\>RateBook\<extra\_id\_4\> 1\<extra\_id\_5\>RateBook\<extra\_id\_6\>  
\<extra\_id\_0\>RateBook is an\<extra\_id\_1\> of\<extra\_id\_2\> 1\<extra\_id\_3\>RateBook\<extra\_id\_4\> 1\<extra\_id\_5\>Rate  
\<extra\_id\_0\>RateBook is an\<extra\_id\_1\> with a\<extra\_id\_2\> of 1\<extra\_id\_3\>RateBook\<extra\_id\_4\> 1

I would be happy if someone can explain the reason for this behavior.

---

<div class="post-metadata">

**Author:** ![ashokurlana](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@ashokurlana](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fashokurlana)%5C
**Post date:** [July 18, 2022, 12:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-generate-output-doesnt-produce-extra-id-0%2F9403%2F2 "2022-07-18T12:45:21Z")

</div>

I also facing the same issue, highly appreciated if someone resolve it. Thanks in advance.