# Load model from checkpoints occurs degraded performance
**URL:**
**Author:**  [@Hanzhi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHanzhi)%5C
**Post date:** [July 7, 2023, 11:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-model-from-checkpoints-occurs-degraded-performance%2F46000%2F1 "2023-07-07T11:17:42Z")
Hi, I’m trying to load a pre-trained model from the local checkpoint. However, I get significantly different results when I evaluate the performance on the same validation set used in the training phase.
The code I’m using is an example notebook: “[https://github.com/huggingface/peft/blob/main/examples/image\_classification/image\_classification\_peft\_lora.ipynb](https://github.com/huggingface/peft/blob/main/examples/image_classification/image_classification_peft_lora.ipynb)”
After the training, I save the model using the following:
trainer.save\_model()
trainer.log\_metrics(“train”, train\_results.metrics)
trainer.save\_metrics(“train”, train\_results.metrics)
trainer.save\_state()
Then, I load the model using:
new\_model = AutoModelForImageClassification.from\_pretrained(‘vit-base-patch16-224-in21k-finetuned-lora/checkpoint-18’)
new\_trainer = Trainer(new\_model, args,
eval\_dataset=val\_ds, tokenizer=image\_processor,
compute\_metrics=compute\_metrics, data\_collator=collate\_fn)
new\_trainer.evaluate(val\_ds)
However, two trainers gave me completely different results:
{‘eval\_loss’: 0.4082214832305908,
‘eval\_accuracy’: 0.88,
‘eval\_runtime’: 6.8795,
‘eval\_samples\_per\_second’: 72.68,
‘eval\_steps\_per\_second’: 0.581,
‘epoch’: 2.0}
and
{‘eval\_loss’: 4.637523651123047,
‘eval\_accuracy’: 0.016,
‘eval\_runtime’: 6.0346,
‘eval\_samples\_per\_second’: 82.856,
‘eval\_steps\_per\_second’: 0.663}
Thus, it seems I’m not loading the model with random weights. Could you please tell me how to load the architecture and the weights?
---
**Author:**  [@Hanzhi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHanzhi)%5C
**Post date:** [July 7, 2023, 11:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-model-from-checkpoints-occurs-degraded-performance%2F46000%2F2 "2023-07-07T11:46:32Z")
Ahh, My problem. I was using LoRA thus the new model can not load the corresponding weights.
---
**Author:**  [@Hanzhi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHanzhi)%5C
**Post date:** [July 7, 2023, 12:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-model-from-checkpoints-occurs-degraded-performance%2F46000%2F3 "2023-07-07T12:10:14Z")
For anyone wanna know how to load a pre-trained PEFT model, here is an example:
new\_model = AutoModelForImageClassification.from\_pretrained(
‘“google/vit-base-patch16-224-in21k”’,
label2id=label2id,
id2label=id2label,
ignore\_mismatched\_sizes=True,
)
lora\_model = get\_peft\_model(new\_model, config)
lora\_model = lora\_model.from\_pretrained(new\_model,
model\_id=‘vit-base-patch16-224-in21k-finetuned-lora/checkpoint-18’)