# GPT2 returns sequence of \<|endoftext|\> after finetuning

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-returns-sequence-of-endoftext-after-finetuning%2F70418%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 24, 2024, 1:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-returns-sequence-of-endoftext-after-finetuning%2F70418 "2024-01-24T13:33:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tempdeltavalue](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tempdeltavalue/32/28454_2.png) [@tempdeltavalue](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftempdeltavalue)%5C
**Post date:** [January 24, 2024, 1:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-returns-sequence-of-endoftext-after-finetuning%2F70418%2F1 "2024-01-24T13:33:40Z")

</div>

Here’s I tried to finetune gpt2

> <https://github.com/tempdeltavalue/temp_l/blob/main/finetune_seq2seq.ipynb>

but after few iterations I started to get same sequences, for ex.  
prompt: Prime number is

model output: \<|endoftext|\>\<|endoftext|\>\<|endoftext|\>\<|endoftext|\>\<|endoftext|\>\<|endoftext|\>\<|endoftext|\>\<|endoftext|\>

By the way, my text divided into sentences without [EOS] tokens, in other words every item is separate sentence

---

<div class="post-metadata">

**Author:** ![tempdeltavalue](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tempdeltavalue/32/28454_2.png) [@tempdeltavalue](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftempdeltavalue)%5C
**Post date:** [January 27, 2024, 12:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-returns-sequence-of-endoftext-after-finetuning%2F70418%2F2 "2024-01-27T00:20:23Z")

</div>

upd:

It’s reproducible even without finetuning

 ![Screenshot 2024-01-27 021120](https://us1.discourse-cdn.com/hellohellohello/original/3X/7/d/7d76adda64ab398dcbb44446de7705ed0b72ad8b.png)

---

<div class="post-metadata">

**Author:** ![tempdeltavalue](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tempdeltavalue/32/28454_2.png) [@tempdeltavalue](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftempdeltavalue)%5C
**Post date:** [January 31, 2024, 11:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-returns-sequence-of-endoftext-after-finetuning%2F70418%2F4 "2024-01-31T11:05:56Z")

</div>

one top screenshot - correct output

 ![Screenshot 2024-01-31 124015](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/8/d87835efa393c13169c06f4cab61af1b5994fb4c.png)

on bottom - eos sequence

 ![Screenshot 2024-01-31 12395922](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/9/e94c788d781129bc1a67a62facec1a775979e546.png)

But code generally the same