# Load a single GPU checkpoint to 2 GPUS (deepspeed)

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-a-single-gpu-checkpoint-to-2-gpus-deepspeed%2F19806%3E%5C
**Category:** Intermediate\
**Created:** [June 29, 2022, 6:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-a-single-gpu-checkpoint-to-2-gpus-deepspeed%2F19806 "2022-06-29T18:33:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ierezell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ierezell/32/2517_2.png) [@ierezell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fierezell)%5C
**Post date:** [June 29, 2022, 6:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-a-single-gpu-checkpoint-to-2-gpus-deepspeed%2F19806%2F1 "2022-06-29T18:33:51Z")

</div>

Hello 🙂

When training a model on one GPU with deep speed, I was able to save the model and to convert the weights to a `pytorch_model.bin` with the given `zero_to_fp32.py` script and train again.  
(see here : [[Solved] Cannot restart training from deepspeed checkpoint](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsolved-cannot-restart-training-from-deepspeed-checkpoint%2F19786))

However, I’m not able to load again this model and train it this time on two gpus :

```auto
deepspeed.runtime.zero.utils.ZeRORuntimeException: The checkpoint being loaded used a DP world size of 1 but the current world size is 2. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.

```

The error seems clear but is there any way to load the model (CPU or GPU) like for inference and _then_ split it among the two gpus ?

Thanks in advance,  
Have a great day