Questions about deepspeed resume training

smangrul
Tue, 13 Sep 2022 05:27:17 +0000

Hello @cyk1337, we have discussed this with the DeepSpeed team and they have said that the current codebase assumes the user has a shared filesystem.

1. As a workaround, use a shared filesystem.

2. Raised issue on DeepSpeed repo as per their recommendation here: [BUG] Can’t load checkpoint without having shared filesystem in multi-node training when multi-node setup config remains same · Issue #2319 · microsoft/DeepSpeed (github.com). Please follow it up there. Once that gets resolved, you should be able to load the state when not having shared filesystem.