# Errors with Tokenizers on Llama

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferrors-with-tokenizers-on-llama%2F132264%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [December 19, 2024, 11:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferrors-with-tokenizers-on-llama%2F132264 "2024-12-19T11:52:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dristi](https://avatars.discourse-cdn.com/v4/letter/d/f17d59/32.png) [@dristi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdristi)%5C
**Post date:** [December 19, 2024, 11:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferrors-with-tokenizers-on-llama%2F132264%2F1 "2024-12-19T11:52:38Z")

</div>

I tried downloading llama3 on my server and train. I downloaded it from meta and I tried to convert the raw meta data. I have protobuf and sentencepiece downloaded still it says I have to download it.

ValueError: Failed to instantiate tokenizer. Please, make sure you have sentencepiece and protobuf installed.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 19, 2024, 1:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferrors-with-tokenizers-on-llama%2F132264%2F2 "2024-12-19T13:19:09Z")

</div>

Assuming that PyTorch is installed. This should be fine, but there is a possibility that the actual error is occurring elsewhere.

```auto
pip install protobuf sentencepiece transformers

```

---

<div class="post-metadata">

**Author:** ![Invicto69](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/invicto69/32/40711_2.png) [@Invicto69](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FInvicto69)%5C
**Post date:** [February 8, 2025, 5:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferrors-with-tokenizers-on-llama%2F132264%2F3 "2025-02-08T17:11:47Z")

</div>

> <https://github.com/huggingface/transformers/issues/35326#issuecomment-2645846778>
>
> \### System Info
> 
> python == 3.10.12
> transformers == 4.45.1
> 
> 
> \### Who can help?
> …
> \_No response\_
> 
> \### Information
> 
> \- \[X\] The official example scripts
> \- \[\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> I ran the \`convert\_llama\_weightsto\_hf.py\` script on the llama-3.3-70B-Instruct checkpoint weights 
> \`!python convert\_llama\_weights\_to\_hf.py --input\_dir "llama\_3\_3\_model/Llama-3.3-70B-Instruct/original" --model\_size 70B --output\_dir "llama\_3\_3\_model/Llama-3.3-70B-Instruct/hf-converted"\`, 
> and got the 
> \`ValueError: Failed to instantiate tokenizer.Pleae, make sure you have sentencepiece and protobuf.\`
> I tried upgrading both sentencepiece and protobuf but same error is occurring again
> 
> 
> \### Expected behavior
> 
> downloaded from HuggingFace Files.