# "Add\_tokens" breaks words when encoding **URL:** **Author:** ![sduran](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sduran](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsduran)%5C **Post date:** [August 2, 2022, 2:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-tokens-breaks-words-when-encoding%2F21154%2F1 "2022-08-02T14:42:53Z") I am using the add\_tokens function in order to get a larger vocabulary for the distilgpt2 pre-trained tokenizer. But, when I do it, it changes the tokenizer’s behaviour when doing the encoding: it breaks words, in order to identify my new token. I don’t understand why it does it, and whether is possible to force it to work as it was before. I provide an example: ```auto from transformers import AutoTokenizer, TFAutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("distilgpt2") model = TFAutoModelForCausalLM.from_pretrained("distilgpt2") ``` If I use the word ‘crypt’, it does not break it, even if there are existing subwords in it: ```auto tokenizer.encode('crypt') ## [29609] tokenizer.encode('cry') ## [20470] tokenizer.encode('pt') ## [457] ``` But if I add “ryp”, it breaks the words (and I don’t want them broken! I just want to add the full word “ryp”) ```auto tokenizer.add_tokens('ryp') model.resize_token_embeddings(len(tokenizer)) tokenizer.encode('crypt') # [66, 50257, 83] ``` Would you know what this happens, and how I can force it to work as before? I read the docs about the BPE encoding, but I don’t find how to force it to use the largest found token. Thanks ! ---
**Author:** ![AlexandrosChariton](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alexandroschariton/32/17809_2.png) [@AlexandrosChariton](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAlexandrosChariton)%5C **Post date:** [February 13, 2023, 5:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-tokens-breaks-words-when-encoding%2F21154%2F2 "2023-02-13T17:09:28Z")
```auto from transformers import AutoTokenizer, TFAutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("distilgpt2") from transformers.tokenization_utils import AddedToken tokenizer.add_tokens(AddedToken('ryp', single_word = True)) ``` AddedToken accepts single strings. Then as you can see: ```auto tokenizer.encode('crypt') #[29609] ``` Huggingface source is here: [Added Tokens](https://huggingface.co/docs/tokenizers/api/added-tokens) ---
**Author:** ![TunyTrinh](https://avatars.discourse-cdn.com/v4/letter/t/a9adbd/32.png) [@TunyTrinh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTunyTrinh)%5C **Post date:** [August 22, 2023, 4:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-tokens-breaks-words-when-encoding%2F21154%2F3 "2023-08-22T04:05:55Z")
I would like to express my gratitude to @AlexandrosChariton for providing a solution to a problem I was facing. However, I have tried implementing the suggested method, and while it does work, it doesn’t entirely address my specific case. Here is my situation: I have checked the tokens “He” and “Hello” in the vocabulary of the ‘bert-base-cased’ model, and they exist. The results are as follows: ```auto tokenizer.tokenize("Hello") # Output is ['Hello'] tokenizer.tokenize("He") # Output is ['He'] tokenizer.tokenize("Helloe") # Output is ['Hello', 'e'] tokenizer.tokenize("Hee") # Output is ['He', 'e'] ``` However, when I attempt to use the same method to add a new token, “pr,” using the following code: ```auto tokenizer.add_tokens(AddedToken('pr', single_word=True)) ``` The method works in some cases: ```auto tokenizer.tokenize("present") # Output is ['present'] tokenizer.tokenize("pr") # Output is ['pr'] ``` But it doesn’t work in other cases: ```auto tokenizer.tokenize("prmention") # Output is ['p', '##rm', '##ent', '##ion'] # My expected output is ['pr', '##ment', '##ion'], similar to the output of tokenizer.tokenize('privatemention') which is ['private', '##ment', '##ion'] ``` Could anyone advise me on how to solve this problem? Thank you in advance for your assistance.