# Issue in loading the saved tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-in-loading-the-saved-tokenizer%2F67978%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [January 4, 2024, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-in-loading-the-saved-tokenizer%2F67978 "2024-01-04T13:54:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![hadnoor](https://avatars.discourse-cdn.com/v4/letter/h/aeb1de/32.png) [@hadnoor](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhadnoor)%5C
**Post date:** [January 4, 2024, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-in-loading-the-saved-tokenizer%2F67978%2F1 "2024-01-04T13:54:07Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/f/5/f5025d6c1a84495a18d2ec6d6cb94b3be711a482.png)

---

<div class="post-metadata">

**Author:** ![hadnoor](https://avatars.discourse-cdn.com/v4/letter/h/aeb1de/32.png) [@hadnoor](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhadnoor)%5C
**Post date:** [January 4, 2024, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-in-loading-the-saved-tokenizer%2F67978%2F2 "2024-01-04T13:54:43Z")

</div>

i was using the following code :

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from\_pretrained(“Salesforce/xgen-7b-8k-base”,trust\_remote\_code=True)

tokenizer.pad\_token = tokenizer.eos\_token

tokenizer.save\_pretrained(“toksav”)

loaded\_tokenizer = AutoTokenizer.from\_pretrained(“toksav”)