# How to load a model and make in parallel (T5)

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-model-and-make-in-parallel-t5%2F3829%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 22, 2021, 8:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-model-and-make-in-parallel-t5%2F3829 "2021-02-22T20:13:50Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![peggy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/peggy/32/286_2.png) [@peggy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpeggy)%5C
**Post date:** [February 22, 2021, 8:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-model-and-make-in-parallel-t5%2F3829%2F1 "2021-02-22T20:13:50Z")

</div>

Hi,  
I’m using the almost-new `model.parallelize` feature for T5 models, and at some point, I need to save my model checkpoint and load it later to continue training. How can I do these together? With `model.load_state_dict` everything loads on the device I provide, but how can it be loaded on several devices like it’s done in ` model.parallelize`?