# Extending the tokenizer affects model generation

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fextending-the-tokenizer-affects-model-generation%2F132162%3E%5C
**Category:** Intermediate\
**Created:** [December 19, 2024, 1:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fextending-the-tokenizer-affects-model-generation%2F132162 "2024-12-19T01:45:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![suruti94](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/suruti94/32/37030_2.png) [@suruti94](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsuruti94)%5C
**Post date:** [December 19, 2024, 1:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fextending-the-tokenizer-affects-model-generation%2F132162%2F1 "2024-12-19T01:45:29Z")

</div>

Hi,

I followed the example and extended the tokenizer for a new language. Something like this:

llama\_tokenizer = AutoTokenizer.from\_pretrained(base\_model)  
new\_tokenizer = llama\_tokenizer.train\_new\_from\_iterator(training\_corpus, 138000)

The new tokenizer itsels seems to work fine where I am able to decode in both the original and new language. But If I use the model to generate new text, it does not work anymore:

model = AutoModelForCausalLM.from\_pretrained(base\_model)  
tokenizer = AutoTokenizer.from\_pretrained(\<new\_tokenizer\>)  
model.resize\_token\_embeddings(len(tokenizer))

If i use this model to generate and batch\_decode in the original language of the base model, it produces answers mixing text from original and new language. Why is this happening ? I have also trained the model with more text from new language and that also does not seem to help.

Thanks  
Mohan

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 19, 2024, 2:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fextending-the-tokenizer-affects-model-generation%2F132162%2F2 "2024-12-19T02:02:02Z")

</div>

```py
tokenizer = AutoTokenizer.from_pretrained(<new_tokenizer>)

```

This is just a hypothesis, but I think that **AutoTokenizer** may not recognize your new tokenizer.  
If the structure itself has not changed, you can use the **LlamaTokenizer** class, and if the structure itself has changed, you can use it by **defining a new class**.

> **[Auto Classes](https://huggingface.co/docs/transformers/model_doc/auto)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Llama2](https://huggingface.co/docs/transformers/v4.47.1/en/model_doc/llama2#transformers.LlamaTokenizer)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![suruti94](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/suruti94/32/37030_2.png) [@suruti94](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsuruti94)%5C
**Post date:** [December 19, 2024, 4:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fextending-the-tokenizer-affects-model-generation%2F132162%2F3 "2024-12-19T04:24:47Z")

</div>

Thanks John. I tried some of your approaches. It did not help much (may be I did implement correctly). But that gave me some clues. Finally I tried this.

model = AutoModelForCausalLM.from\_pretrained(base\_model)  
llama\_tokenizer = AutoTokenizer.from\_pretrained(\<base\_llama\_model\>)  
tokenizer = AutoTokenizer.from\_pretrained(\<new\_tokenizer)

diff\_tokens = set(tokenizer.vocab).difference(llama\_tokenizer.vocab)  
llama\_tokenizer.add\_tokens(list(diff\_tokens))  
model.resize\_token\_embeddings(len(llama\_tokenizer))

This seems to work. It is still not clear to me as to why using the new tokenizer I showed before does not work ? Isn’t that expected to work ? I followed the example (train\_from\_new\_iterator) from the example in the document.

Thanks  
Mohan

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 19, 2024, 4:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fextending-the-tokenizer-affects-model-generation%2F132162%2F4 "2024-12-19T04:37:50Z")

</div>

Well, whatever the case, I’m glad it seems to be working.😀

> It is still not clear to me as to why using the new tokenizer I showed before does not work ? Isn’t that expected to work ?

It seems that the problem with the tokenizer not updating properly has been around for a while…  
Apparently, the function itself is working as expected and there is no bug, but it’s not easy for users to understand.

> <https://stackoverflow.com/questions/76198051/how-to-add-new-tokens-to-an-existing-huggingface-tokenizer>