# Does tokenizer changed during model training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-tokenizer-changed-during-model-training%2F19213%3E%5C
**Category:** Beginners\
**Created:** [June 15, 2022, 7:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-tokenizer-changed-during-model-training%2F19213 "2022-06-15T19:01:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Talha](https://avatars.discourse-cdn.com/v4/letter/t/a5b964/32.png) [@Talha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTalha)%5C
**Post date:** [June 15, 2022, 7:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-tokenizer-changed-during-model-training%2F19213%2F1 "2022-06-15T19:01:15Z")

</div>

I want to know during model training, does tokenizer alter it self, (i mean does it train) or it remain the same as original one.  
For example i used the  
`AutoTokenizer.from_pretrained(model_name,local_files_only=False)`  
after model training, is there any change to tokenizer?

---

<div class="post-metadata">

**Author:** ![Slinae](https://avatars.discourse-cdn.com/v4/letter/s/a183cd/32.png) [@Slinae](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSlinae)%5C
**Post date:** [June 16, 2022, 1:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-tokenizer-changed-during-model-training%2F19213%2F2 "2022-06-16T01:44:22Z")

</div>

Hi, Talha,

Tokenizer id does not change. For example, the token\_id of `[['sep']]` is 102. It will be 102 through the training. But the embedding hidden state will be changed.

---

<div class="post-metadata">

**Author:** ![ShieldHero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shieldhero/32/4451_2.png) [@ShieldHero](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FShieldHero)%5C
**Post date:** [August 11, 2022, 1:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-tokenizer-changed-during-model-training%2F19213%2F3 "2022-08-11T13:47:47Z")

</div>

One doubt. So lets say I have trained two models of “bert-base-uncased” to do different classification tasks(same number of classes).  
So, now, can I use the tokenizer of one model1 to tokenize the other one? Will the result be different compared to using two different tokenizers?
