# AutoTokenizer is very slow when loading llama tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotokenizer-is-very-slow-when-loading-llama-tokenizer%2F41547%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [May 31, 2023, 2:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotokenizer-is-very-slow-when-loading-llama-tokenizer%2F41547 "2023-05-31T02:38:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![coincheung](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@coincheung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcoincheung)%5C
**Post date:** [May 31, 2023, 2:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotokenizer-is-very-slow-when-loading-llama-tokenizer%2F41547%2F1 "2023-05-31T02:38:35Z")

</div>

Code is like this:

```auto
    tokenizer = LlamaTokenizer.from_pretrained('decapoda-research/llama-13b-hf', use_fast=True)
    tokenizer.save_pretrained('./llama_tok')

    tokenizer = LlamaTokenizer.from_pretrained('./llama_tok/') # very fast to load
    tokenizer = AutoTokenizer.from_pretrained('./llama_tok/') # very slow to load

```

What is the cause of this and how could I fix it please?

---

<div class="post-metadata">

**Author:** ![asbljy](https://avatars.discourse-cdn.com/v4/letter/a/cab0a1/32.png) [@asbljy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fasbljy)%5C
**Post date:** [June 22, 2023, 10:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotokenizer-is-very-slow-when-loading-llama-tokenizer%2F41547%2F2 "2023-06-22T22:34:47Z")

</div>

I also found this issue.

---

<div class="post-metadata">

**Author:** ![reece124](https://avatars.discourse-cdn.com/v4/letter/r/779978/32.png) [@reece124](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Freece124)%5C
**Post date:** [October 31, 2023, 5:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotokenizer-is-very-slow-when-loading-llama-tokenizer%2F41547%2F3 "2023-10-31T05:39:30Z")

</div>

Same issue here also for loading Vicuna tokenizers. Got it workaround by replacing AutoTokenizer with LlamaTokenizer.