# Questions about deepspeed resume training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-about-deepspeed-resume-training%2F22765%3E%5C
**Category:** 🤗Accelerate\
**Created:** [September 8, 2022, 1:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-about-deepspeed-resume-training%2F22765 "2022-09-08T13:43:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cyk1337](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cyk1337/32/24460_2.png) [@cyk1337](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcyk1337)%5C
**Post date:** [September 8, 2022, 1:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-about-deepspeed-resume-training%2F22765%2F1 "2022-09-08T13:43:43Z")

</div>

1. Background: Use deepspeed (use ZeRO-1) for multi-node training, save optimizers to resume training.
2. I find that `accelerate.save_state` only save the partitioned optimizer state on each machine. if we use 32 gpus (4 node), each node only saves 8 partitions. For the 1st node, it saves `bf16_zero_pp_rank_(0-7)_mp_rank_00_optim_states.pt`. When call `load_state` to resume training for deepspeed, the 1 st node reports the issue of `The following zero checkpoints paths are missing: ['./output/test/step_5/pytorch_model/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt', './output/test/step_5/pytorch_model/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt']`

It requires to merge the all states from different nodes. Any easier solution to synchronize them?

---

<div class="post-metadata">

**Author:** ![smangrul](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smangrul/32/3791_2.png) [@smangrul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmangrul)%5C
**Post date:** [September 13, 2022, 5:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-about-deepspeed-resume-training%2F22765%2F2 "2022-09-13T05:27:17Z")

</div>

Hello @cyk1337, we have discussed this with the DeepSpeed team and they have said that the current codebase assumes the user has a shared filesystem.

1. As a workaround, use a shared filesystem.
2. Raised issue on DeepSpeed repo as per their recommendation here: [[BUG] Can’t load checkpoint without having shared filesystem in multi-node training when multi-node setup config remains same · Issue #2319 · microsoft/DeepSpeed (github.com)](https://github.com/microsoft/DeepSpeed/issues/2319). Please follow it up there. Once that gets resolved, you should be able to load the state when not having shared filesystem.

---

<div class="post-metadata">

**Author:** ![cyk1337](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cyk1337/32/24460_2.png) [@cyk1337](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcyk1337)%5C
**Post date:** [October 21, 2022, 4:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-about-deepspeed-resume-training%2F22765%2F3 "2022-10-21T04:40:12Z")

</div>

Thanks for your reply! I have posted an issue on DeepSpeed repo. Hope it will be solved ASAP;)