# Deepspeed resume training from saved states

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeepspeed-resume-training-from-saved-states%2F22768%3E%5C
**Category:** 🤗Accelerate\
**Created:** [September 8, 2022, 1:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeepspeed-resume-training-from-saved-states%2F22768 "2022-09-08T13:50:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![cyk1337](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cyk1337/32/24460_2.png) [@cyk1337](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcyk1337)%5C
**Post date:** [September 8, 2022, 1:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeepspeed-resume-training-from-saved-states%2F22768%2F1 "2022-09-08T13:50:54Z")

</div>

To resume deepspeed zero-1 multi-node training, I use `accelerate.save_state` to save the optimizer states. However, each node only saves one partitioned states (e.g., 1/4 states for 4 machines). When loading states for resuming training, the `load_state` reports missing other checkpoints `The following zero checkpoints paths are missing: ['./output/test/step_5/pytorch_model/bf16_zero_pp_rank_{8-31}_mp_rank_00_optim_states.pt', ...]`, and the subprocesses are killed.

It requires to manually synchronize the partitioned states from multiple nodes. Any easier solution to it?