# Unable to load checkpoint after finetuning
**URL:**
**Author:**  [@Sifal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSifal)%5C
**Post date:** [August 14, 2023, 8:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-checkpoint-after-finetuning%2F50693%2F1 "2023-08-14T08:38:57Z")
Hi,
So I’m running into trouble when trying to load the model weights after finetuning a pretrained base model, steps I’m taking:
```auto
model= BertForSequenceClassification.from_pretrained("model/repo")
model.classifier = torch.nn.Sequential(
torch.nn.Dropout(0.4),
torch.nn.Linear(model.config.hidden_size, 256),
torch.nn.ReLU(),
torch.nn.Dropout(0.4),
torch.nn.Linear(256, 2),
)
```
getting a warning here which I guess is fine for the moment:
```auto
Some weights of the model checkpoint at model/repo were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at model/repo and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```
But after finetuning using hyperparameter search :
```auto
trainer.hyperparameter_search(
direction="maximize",
backend="optuna",
hp_space=optuna_hp_space,
n_trials=10,
)
```
and trying to load the model with the following config I’m still getting the same waring as before:
```auto
from transformers import AutoConfig, AutoModel
model = AutoModel.from_pretrained("./run-3/checkpoint-1880",config=model.config)
```
Is there something I’m missing?
Thanks for looking into this.
---
**Author:**  [@lakshmi97](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flakshmi97)%5C
**Post date:** [November 16, 2023, 11:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-checkpoint-after-finetuning%2F50693%2F2 "2023-11-16T23:06:40Z")
I have the same issue. I try to resume my model training and I get a similar behavior. I don’t see why only classification head weights aren’t saved while all other weights are saved.
_**There were missing keys in the checkpoint model loaded: [‘cls.predictions.decoder.weight’,‘cls.predictions.decoder.bias’].**_
---
**Author:**  [@MohammedAljahdali](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohammedAljahdali)%5C
**Post date:** [November 23, 2023, 4:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-checkpoint-after-finetuning%2F50693%2F3 "2023-11-23T16:07:38Z")
I have the same issue, when resuming from a checkpoint, I get this warning: `There were missing keys in the checkpoint model loaded: ['lm_head.weight'].`. I am not sure what is the cause, but trying to figure it out.
---
**Author:**  [@MohammedAljahdali](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohammedAljahdali)%5C
**Post date:** [November 23, 2023, 4:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-checkpoint-after-finetuning%2F50693%2F4 "2023-11-23T16:57:30Z")
I solved the issue on my end by setting `save_safetensors=False,` in the TrainingArguments.
---
**Author:**  [@Rudrashk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRudrashk)%5C
**Post date:** [January 24, 2024, 7:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-checkpoint-after-finetuning%2F50693%2F6 "2024-01-24T19:19:55Z")
When using `save_safetensors=False`, it throws errors while loading the prior checkpoints as it cannot detect the pytorch\_model.bin file. Any suggestion to restore?
---
**Author:**  [@alimotahharynia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falimotahharynia)%5C
**Post date:** [February 21, 2024, 9:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-checkpoint-after-finetuning%2F50693%2F7 "2024-02-21T21:15:59Z")
I also did this and it worked. Wonder what’s the problem with model.safetensors, cause when I check the model.state\_dict of safetensors, the lm\_head.weight was present.