# Saving checkpoint is too slow with deepspeed
**URL:**
**Author:**  [@quovadiss](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fquovadiss)%5C
**Post date:** [August 9, 2023, 8:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-checkpoint-is-too-slow-with-deepspeed%2F50099%2F1 "2023-08-09T08:36:20Z")
A custom model is nn.Module with self.lm(large language model, CLM). I’m trying to train the model with deepspeed zero3. I’m using A100 x8 gpus. I made ds\_config.json file and ran sh file like this.
`deepspeed finetune/run.py --num_gpus=8 --deepspeed ds_config.json ...`
Also, I changed Trainer’s method “\_save” like this.
```auto
def _save(self, output_dir: Optional[str] = None):
output_dir = output_dir if output_dir is not None else self.args.output_dir
os.makedirs(output_dir, exist_ok=True)
logger.info("Saving model checkpoint to %s", output_dir)
state_dict = self.accelerator.get_state_dict(self.deepspeed)
self.model.lm.save_pretrained(output_dir, state_dict=state_dict)
self.tokenizer.save_pretrained(output_dir)
# Good practice: save your training arguments together with the trained model
torch.save(self.args, os.path.join(output_dir, "training_args.bin"))
```
But, saving checkpoint doesn’t finish, and I couldn’t get pytorch\_model.bin file. I think it’s too slow to save checkpoint gathering all models’ parameters from 8 gpus.
How can I solve this problem?
---
**Author:**  [@berkayberabi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fberkayberabi)%5C
**Post date:** [September 11, 2023, 4:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-checkpoint-is-too-slow-with-deepspeed%2F50099%2F2 "2023-09-11T16:44:13Z")
Hi, I have the same problem with saving `codet5-6b` with `zero3`. The logs says `pytorch_model.bin` is saved but it is not there and the process hangs. Did you find a solution?
---
**Author:**  [@theblackhacker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftheblackhacker)%5C
**Post date:** [November 14, 2023, 8:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-checkpoint-is-too-slow-with-deepspeed%2F50099%2F3 "2023-11-14T08:48:09Z")
With 8xA100, i think you should want to use ZeRO-2, it’ll be a lot faster in training and in model saving too.
The reason ZeRO-3 saves the model is very slow because it only uses 1 GPU[0] to combine all layers, and only combines 1 layer at a time.
---
**Author:**  [@jakobsal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjakobsal)%5C
**Post date:** [March 2, 2024, 5:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-checkpoint-is-too-slow-with-deepspeed%2F50099%2F4 "2024-03-02T17:20:13Z")
I’m using 4xL4 with ZeRO-2 to fine-tune sdxl. All 4 GPUs are running at 100% when the checkpoint is being saved but nothing is saved despite waiting for 30 mins. Is this expected or am I doing something wrong here?
---
**Author:**  [@jax2000](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjax2000)%5C
**Post date:** [March 6, 2024, 7:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-checkpoint-is-too-slow-with-deepspeed%2F50099%2F6 "2024-03-06T07:39:12Z")
I’m in a situation just like yours, do you have a good solution for it now? @jakobsal
---
**Author:**  [@jakobsal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjakobsal)%5C
**Post date:** [March 6, 2024, 8:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-checkpoint-is-too-slow-with-deepspeed%2F50099%2F7 "2024-03-06T08:23:52Z")
Hey @jax2000, I’m getting back to this today so not yet. What I understand so far though is that you might have to set some environmental variables that will depend on your setup. I’ve found this [Github thread](https://github.com/huggingface/accelerate/issues/314) to contain some interesting stuff to try out. This [NVIDIA env variables docs](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html) is probably also useful.