# 2 possible bugs for adding new tokens to T5
**URL:**
**Author:**  [@berkayberabi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fberkayberabi)%5C
**Post date:** [November 11, 2020, 9:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F2-possible-bugs-for-adding-new-tokens-to-t5%2F1984%2F1 "2020-11-11T09:58:54Z")
Hi,
I think that I found 2 issues while trying to add new tokens to the T5 tokenizer. My goal was to add smaller sign “\<” to the vocabulary of T5. However, doing this prevents the model from extracting the eos\_token and unk\_token correctly. When the model encounters or , the model splits it as \< and unk\> or \< and /s\>. (I am adding the eos tokens to the end by myself and they will be extracted as normal tokens and also occur in the prediction even if skip\_special\_tokens=True)
So to overcome this, I decided to change the eos\_token to ~end~. This works but then I observed that the prediction results were much worse! Especially at the end of the prediction. Actually everything was okay but suddenly at the end there were always some strange additional generation. I tracked down the issue and I found it.
The custom eos\_token is added to the vocabulary as a new token, instead of overwriting the existing eos\_token. This means “\” has id 1, when I add my own custom eos\_token, it does not overwrite 1, or it will not be mapped to 1. It adds a new token to vocabulary but then all the pretaining for eos token is gone! The model has to learn the eos\_token from stracth and this degenerates the results remarkably. If you could run the code below and observe the output, you will understand better what I mean!
```
tokenizer = T5Tokenizer.from_pretrained(model_name,)
print('len_tokenizer with default eos token: ', len(tokenizer))
tokenizer = T5Tokenizer.from_pretrained(model_name, eos_token='~end~')
print('len_tokenizer with custom eos token: ', len(tokenizer))
print('id 1 before adding <:', tokenizer.decode(1, skip_special_tokens=False))
print('id 2 before adding <:', tokenizer.decode(2, skip_special_tokens=False))
tokenizer.add_tokens(['{', '}', '<', '>', '\\'])
print('id 1 after adding <:', tokenizer.decode(1, skip_special_tokens=False))
print('id 2 after adding <:', tokenizer.decode(2, skip_special_tokens=False))
custom_eos_id = tokenizer.encode("~end~", return_tensors='pt', truncation=True, padding=True)
print('custom_eos id: ', custom_eos_id)
```
Thank you for your time!
---
**Author:**  [@thomwolf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthomwolf)%5C
**Post date:** [November 12, 2020, 12:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F2-possible-bugs-for-adding-new-tokens-to-t5%2F1984%2F2 "2020-11-12T12:48:08Z")
Hm currently we don’t have a simple high-level way to change the string associated to a token.
You could manually edit the tokenizer.JSON file generated by the fast version of the T5 tokenizer I guess but that’s a bit hacky.
---
**Author:**  [@leestevennz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fleestevennz)%5C
**Post date:** [June 21, 2024, 11:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F2-possible-bugs-for-adding-new-tokens-to-t5%2F1984%2F3 "2024-06-21T23:36:58Z")
Hi @thomwolf ,
I am wanting to replace a token in the existing vocab. After manually editing the tokenizer.json file, when I try to load the tokenizer I’m getting the following error:
Exception: data did not match any variant of untagged enum ModelWrapper at line 356367 column 3
Any ideas how to fix this?
---
**Author:**  [@leestevennz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fleestevennz)%5C
**Post date:** [June 25, 2024, 3:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F2-possible-bugs-for-adding-new-tokens-to-t5%2F1984%2F4 "2024-06-25T03:47:04Z")
You can ignore this sorry. I found the issue. if you change the vocab in anyway, you need to make sure you also update the merges accordingly.