# What exacly is changed in the tokenizer after training it?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-exacly-is-changed-in-the-tokenizer-after-training-it%2F20971%3E%5C
**Category:** Beginners\
**Created:** [July 28, 2022, 5:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-exacly-is-changed-in-the-tokenizer-after-training-it%2F20971 "2022-07-28T17:26:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdanieli](https://avatars.discourse-cdn.com/v4/letter/m/ba9def/32.png) [@mdanieli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmdanieli)%5C
**Post date:** [July 28, 2022, 5:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-exacly-is-changed-in-the-tokenizer-after-training-it%2F20971%2F1 "2022-07-28T17:26:01Z")

</div>

Only the new words it learns are added? Or anything else? (other words removed, or more things?)  
I ask since after I trained the tokenizer, new words of the dataset indeed were added, but the model (which I used with this pre-trained tokenizer) performance was reduced. How can I debug what made the performance to reduce (no change in the model bedore and after the tokenizer training)?

Thanks,  
Michal