# Simple Save/Load of tokenizer not working

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-save-load-of-tokenizer-not-working%2F1864%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 3, 2020, 12:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-save-load-of-tokenizer-not-working%2F1864 "2020-11-03T00:29:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhi](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@abhi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhi)%5C
**Post date:** [November 3, 2020, 12:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-save-load-of-tokenizer-not-working%2F1864%2F1 "2020-11-03T00:29:59Z")

</div>

I am training a DistilBert pretrained model for sequence classification with a pretrained tokenizer.

I currently save the model like this:

`> model.save_pretrained(dir)`  
`> tokenizer.save_pretrained(dir)`

And load like this:

`> model.from_pretrained(dir)`  
`> tokenizer.from_pretrained(dir)).`

Weirdly this produces bad results (by over 10%) because the tokenizer has somehow changed. Instead this works much better:

`> model.from_pretrained(dir)`  
`> (DistilBertTokenizer.from_pretrained('distilbert-base-cased')).`

There must be something simple I’m missing here. Why won’t it work properly if I just load the tokenizer directly from dir.

**PROGRESS**

Upon further investigation I noticed that

`> tokenizer.from_pretrained(dir)).`

defaults to

`> set_lower_case=True.`

Manually setting this fixes my issue. However, this doesn’t make sense to me as the model before saving had `set_lower_case=False.` Is there any way it can be saved directly with this option?

---

<div class="post-metadata">

**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthomwolf)%5C
**Post date:** [November 3, 2020, 8:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-save-load-of-tokenizer-not-working%2F1864%2F2 "2020-11-03T08:28:02Z")

</div>

Yes this was fixed in ([https://github.com/huggingface/transformers/pull/8006](https://github.com/huggingface/transformers/pull/8006)) and the fix will be in the next release 🙂

---

<div class="post-metadata">

**Author:** ![abhi](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@abhi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhi)%5C
**Post date:** [November 4, 2020, 12:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-save-load-of-tokenizer-not-working%2F1864%2F3 "2020-11-04T00:03:34Z")

</div>

Great, thanks!