# transformers.Tokenizer produce unexpected results

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-produce-unexpected-results%2F37790%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 26, 2023, 11:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-produce-unexpected-results%2F37790 "2023-04-26T11:53:06Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JeffTian](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@JeffTian](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJeffTian)%5C
**Post date:** [April 26, 2023, 11:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-tokenizer-produce-unexpected-results%2F37790%2F1 "2023-04-26T11:53:06Z")

</div>

When using GPT2Tokenizer, I need to add an additional special token, let’s say \<|special|\>. But when adding it by using `'additional_special_tokens'`, it gives out unexpected results.

```auto
from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained(direction)
tokenizer.add_special_tokens({'additional_special_tokens': ['<|special|>']})

text_expected = '<|endoftext|> bye'
tokenizer(text_expected, return_tensors='pt').input_ids
-> tensor([[50256, 33847]])

text_unexpected = '<|special|> bye'
tokenizer(text_unexpected, return_tensors='pt').input_ids
-> tensor([[50257, 16390]])

```

the id of **‘bye’** changed, seems the tokenizer regards the \<|endoftext|\> as part of the sentence but not for \<|special|\>? is this expected to be seen? Thanks for any explanations.