# Loading pre-trained models with AddedTokens
**URL:**
**Author:**  [@stefanomezza](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstefanomezza)%5C
**Post date:** [May 30, 2024, 12:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-pre-trained-models-with-addedtokens%2F89292%2F1 "2024-05-30T00:42:23Z")
I have pre-trained a `"meta-llama/Llama-2-7b-chat-hf"` model using the `transformers` library. Since my model uses additional tokens, I added them to the tokeniser before training and fine-tuned the “embed\_tokens” module of the network. My training code looked like this:
```auto
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf",trust_remote_code=True, token=hf_token)
tokenizer.add_special_tokens({ "additional_special_tokens":[AddedToken("<|move|>"),
AddedToken("<|endmove|>"),
AddedToken("<|end|>")]})
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map=device_map,
token=hf_token
)
model.resize_token_embeddings(len(tokenizer))
peft_config = LoraConfig(
lora_alpha=lora_alpha,
lora_dropout=lora_dropout,
r=lora_r,
bias="none",
modules_to_save= ["embed_tokens", "lm_head"],
task_type="CAUSAL_LM",
)
```
The model trained and saved successfully. However, when trying to load it using `AutoModelForCausalLM.from_pretrained`, I get the following error:
```auto
Error(s) in loading state_dict for LlamaForCausalLM:
size mismatch for model.embed_tokens.modules_to_save.default.weight: copying a param with shape torch.Size([32003, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).
size mismatch for lm_head.modules_to_save.default.weight: copying a param with shape torch.Size([32003, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096])
```
I appreciate the error is due to the fact that the fine-tuned model has three additional tokens and that causes a mismatch, but how should I load a pre-trained model with a different input shape like mine?
I looked into the transformers API docs for a way to load models with AddedTokens, but I couldn’t find anything. I read a blog post mentioning that passing `ignore_mismatched_sizes=True` to the from\_pretrained function would solve the issue, but it didn’t work for me.
---
**Author:**  [@thatone](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthatone)%5C
**Post date:** [October 14, 2024, 2:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-pre-trained-models-with-addedtokens%2F89292%2F2 "2024-10-14T14:07:38Z")
was this ever solved? i have basically the exact problem
RuntimeError: Error(s) in loading state\_dict for MistralForCausalLM:
size mismatch for model.embed\_tokens.weight: copying a param with shape torch.Size([32002, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).
size mismatch for lm\_head.weight: copying a param with shape torch.Size([32002, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 14, 2024, 2:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-pre-trained-models-with-addedtokens%2F89292%2F3 "2024-10-14T14:14:48Z")
This is a post dealing with a completely different issue, but sometimes there are bugs specific to the version of PEFT, so you might want to try changing the version of each library.
My recent experience has been that some programs do not work if the transformers library is newer than 4.44.
> [@LLaMa3.1 8B Instruct Prompt Tuning for Text Classification doesn't improve test accuracy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama3-1-8b-instruct-prompt-tuning-for-text-classification-doesnt-improve-test-accuracy%2F108838)%3A
>
> Hello everyone! I am trying to fine-tune LLaMa3.1 8B Instruct for text classification with prompt tuning from peft library (prepending a trainable matrix before input embeddings). For instance, I am using the QNLI dataset to classify whether a question and sentence are in entailment. I have preprocessed the dataset so that each data looks like the following: \<|begin\_of\_text|\>\<|start\_header\_id|\>system\<|end\_header\_id|\> Cutting Knowledge Date: December 2023 Today Date: 26 Jul 2024 \<|eot\_id|\>\<|…