# Adding tokens, but tokenizer doesn't use them

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadding-tokens-but-tokenizer-doesnt-use-them%2F78674%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [March 25, 2024, 11:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadding-tokens-but-tokenizer-doesnt-use-them%2F78674 "2024-03-25T11:18:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![WonderYear1905](https://avatars.discourse-cdn.com/v4/letter/w/c89c15/32.png) [@WonderYear1905](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FWonderYear1905)%5C
**Post date:** [March 25, 2024, 11:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadding-tokens-but-tokenizer-doesnt-use-them%2F78674%2F1 "2024-03-25T11:18:58Z")

</div>

Hi, Im trying to add tokens to a pretrained tokenizer.  
First I initialized the tokenizer:

> tokenizer = AutoTokenizer.from\_pretrained(“mistralai/Mistral-7B-Instruct-v0.2”)

Next I created a training iterator and trained a new tokenizer:

> training\_corpus = get\_training\_corpus()  
> new\_tokenizer = tokenizer.train\_new\_from\_iterator(training\_corpus,vocab\_size=10000)#,10000)

Taking the diff:

> tokens\_to\_add = list(set(new\_tokenizer.vocab.keys())- set(tokenizer.vocab.keys()))  
> output = tokenizer.add\_tokens(tokens\_to\_add)

tokenizer is updated, I can see its now in the correct size (i.e original size=32000+ added tokens), I can also see the new token under `added_tokens_decoder` and `added_tokens_encoder`. Everything seems great.

But, when Im trying to tokenize my input data:

> tokenizer.tokenize(x, return\_tensors=“pt”)

The tokenizer just doesnt use the new tokens.  
Any idea what Im doing wrong?!

Thanks!

---

<div class="post-metadata">

**Author:** ![Butanium](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/butanium/32/49190_2.png) [@Butanium](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FButanium)%5C
**Post date:** [August 14, 2024, 11:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadding-tokens-but-tokenizer-doesnt-use-them%2F78674%2F2 "2024-08-14T11:57:57Z")

</div>

Just experienceed the same issue with llama2-7b, did you find a solution to that?

In my case adding `normalized=False` fixed the issue I had on my single example.