**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [August 3, 2020, 12:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F2 "2020-08-03T12:44:52Z")
There is no parameter for that yet, keeping this in mind for the future development of Trainer.
---
**Author:**  [@Tanuj](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTanuj)%5C
**Post date:** [April 29, 2021, 8:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F3 "2021-04-29T08:34:19Z")
Is this implemented yet? If not, is there any other way to do this manually (read: hackishly) with the Trainer api? (Except of course, saving the model every step)
---
**Author:**  [@karthikcs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkarthikcs)%5C
**Post date:** [May 19, 2021, 3:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F4 "2021-05-19T15:31:54Z")
Hi Tanuj,
I have not seen any parameter for that. However, there is a workaround.
Use following combinations
evaluation\_strategy =âstepsâ,
eval\_steps = 10, # Evaluation and Save happens every 10 steps
save\_total\_limit = 5, # Only last 5 models are saved. Older ones are deleted.
load\_best\_model\_at\_end=True,
When I tried with the above combination, at any time 5 previous models will be saved in output directory, but if the best model is not one among them, it will keep the best model as well. So it will be 1 + 5 models. You can change save\_total\_limit = 1 so it will serve your purpose
---
**Author:**  [@monta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmonta)%5C
**Post date:** [July 7, 2021, 6:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F5 "2021-07-07T06:30:37Z")
An option to use Keras callback:
>
---
**Author:**  [@whydinkov](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwhydinkov)%5C
**Post date:** [August 26, 2021, 6:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F6 "2021-08-26T18:53:29Z")
@sgugger So if I understand this correctly, in this post [Checkpoints and disk storage - #9 by sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcheckpoints-and-disk-storage%2F1412%2F9) you actually âconfirmâ that the proposed configuration from @karthikcs will actually achieve the asked functionality (from the original question), namely having
```auto
save_total_limit=1
```
will actually save and load ONLY the best performing checkpoint (save because the best is always on top and ânever gets deletedâ, load because there is only one endpoint left). Is this correct?
---
**Author:**  [@michael-kingston](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmichael-kingston)%5C
**Post date:** [March 5, 2022, 2:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F7 "2022-03-05T02:13:45Z")
this is a great question. was it ever clarified?
---
**Author:**  [@jayelm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjayelm)%5C
**Post date:** [March 15, 2022, 4:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F8 "2022-03-15T04:09:36Z")
Can confirm that setting `save_total_limit` to whatever you want, even 1, will not interfere with Trainerâs ability to load the best model at end. Look at the [source](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/trainer.py#L2200-L2212):
First, [`_sorted_checkpoints`](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/trainer.py#L2189-L2193) prioritizes checkpoints to keep while also making sure not to delete the best model, by rotating the best model to the front of the queue:
```auto
checkpoints_sorted = [checkpoint[1] for checkpoint in checkpoints_sorted]
# Make sure we don't delete the best model.
if self.state.best_model_checkpoint is not None:
best_model_index = checkpoints_sorted.index(str(Path(self.state.best_model_checkpoint)))
for i in range(best_model_index, len(checkpoints_sorted) - 2):
checkpoints_sorted[i], checkpoints_sorted[i + 1] = checkpoints_sorted[i + 1], checkpoints_sorted[i]
return checkpoints_sorted
```
Finally, after sorting, if `save_total_limit=1`, this number is actually increased to 2, so that you always keep the best model:
```auto
# If save_total_limit=1 with load_best_model_at_end=True, we could end up deleting the last checkpoint, which
# we don't do to allow resuming.
save_total_limit = self.args.save_total_limit
if (
self.state.best_model_checkpoint is not None
and self.args.save_total_limit == 1
and checkpoints_sorted[-1] != self.state.best_model_checkpoint
):
save_total_limit = 2
```
---
**Author:**  [@dashapyly](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdashapyly)%5C
**Post date:** [April 22, 2022, 9:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F9 "2022-04-22T21:17:17Z")
What I am looking to do is save checkpoints frequently as not to waste compute time if I need to restart (so I set save\_total\_limit = 5 and save often). But I also want to keep bigger steps, like epochs, for future analysis for training rate success and so on. I definitely need to delete smaller steps because then I quickly use up the space. Is there an option to do this? Thanks!
---
**Author:**  [@Thang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FThang)%5C
**Post date:** [June 7, 2022, 5:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F10 "2022-06-07T05:50:39Z")
It may depend on the size of your save\_steps. In my case, I use:
save\_steps = len(train\_data[âtrainâ])//batch\_size # after each epoch
warmup\_steps = save\_steps//10 # 10% of save\_steps
---
**Author:**  [@AlhitawiMohammed22](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAlhitawiMohammed22)%5C
**Post date:** [April 5, 2023, 5:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F11 "2023-04-05T05:48:56Z")
Does this setting param save the best checkpoint after training finishes?
If we set :
```auto
save_total_limit = 1,
save_best_checkpoint = True
```
or those settings:
```auto
predict_with_generate=True,
evaluation_strategy="steps",
per_device_train_batch_size= 4,
per_device_eval_batch_size= 4,
num_train_epochs= 10 ,
fp16= True,
learning_rate= 4e-5,
output_dir=f'models/',
logging_steps=10,
save_steps=500,
eval_steps=500,
save_total_limit = 1,
report_to=["tensorboard"],
```
if not of the 2 above: what is the correct setting param should I use to keep the best checkpoint at the end?
---
**Author:**  [@AlhitawiMohammed22](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAlhitawiMohammed22)%5C
**Post date:** [April 5, 2023, 5:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F12 "2023-04-05T05:56:02Z")
do those settings save the best checkpoint at the end?
if I set `save_total_limit=1` with ` save_best_checkponit`
---
**Author:**  [@SUNM](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSUNM)%5C
**Post date:** [May 20, 2023, 1:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F13 "2023-05-20T01:10:18Z")
HI @Thang , I hope you are well. sorry during training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can save the best model , my code is as follow, what I missed in the code?
```auto
training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)
Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
'attention_mask': torch.stack([f[1] for f in data]),
'labels': torch.stack([f[0] for f in data])}).train()
```
---
**Author:**  [@SUNM](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSUNM)%5C
**Post date:** [May 20, 2023, 1:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F14 "2023-05-20T01:13:01Z")
Hi @AlhitawiMohammed22, I hope you are well. sorry during training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell me how I can save the best model , my code is as follow, what I missed in the code?
```auto
training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)
Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
'attention_mask': torch.stack([f[1] for f in data]),
'labels': torch.stack([f[0] for f in data])}).train()
```
---
**Author:**  [@AlhitawiMohammed22](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAlhitawiMohammed22)%5C
**Post date:** [May 20, 2023, 11:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F15 "2023-05-20T11:57:39Z")
It works when I pass a combination of three parameters:
```auto
evaluation_strategy="steps",
save_total_limit = 1,
load_best_model_at_end =True,
```
In addition(optional), you can save the model at the end
`trainer.save_model(output_dir = path/to/out`
so you just need to change epoch to steps
---
**Author:**  [@SUNM](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSUNM)%5C
**Post date:** [May 20, 2023, 2:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F16 "2023-05-20T14:00:29Z")
@AlhitawiMohammed22 many thanks for your reply and time. Sorry , in this way it will give me 2 models one best model and other last model? Based on what metrics it give us the best model? Can we define the metric? And how I can get train and validation loss? Many thanks.
---
**Author:**  [@AlhitawiMohammed22](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAlhitawiMohammed22)%5C
**Post date:** [May 20, 2023, 3:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F17 "2023-05-20T15:58:24Z")
you can use `model.eval()` method on your `test_set`
or in another way to use your own metrics
```auto
def compute_metrics():
# do your metrics
pass to the trainer :
` compute_metrics= compute_metrics,`
otherwise, it will use the default one you have to look to trainer `repo`
```
---
**Author:**  [@SUNM](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSUNM)%5C
**Post date:** [May 20, 2023, 11:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F18 "2023-05-20T23:06:21Z")
@AlhitawiMohammed22 Sorry, I mean thr training and validation loss during training. I want to have those numbers. Except tensorboard do u know other ways? Many thanksss
---
**Author:**  [@AlhitawiMohammed22](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAlhitawiMohammed22)%5C
**Post date:** [May 21, 2023, 2:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F19 "2023-05-21T02:00:28Z")
@SUNM no problem at all.
It should be in your output dir/checkpoint/trainer\_state.json
If you have ckp
---
**Author:**  [@SUNM](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSUNM)%5C
**Post date:** [May 23, 2023, 1:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-only-the-best-performing-checkpoint%2F552%2F20 "2023-05-23T01:26:53Z")
@AlhitawiMohammed22 , sorry, do you know how I can get them from jason file to make my graph?