# Move Trainer out of GPU

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmove-trainer-out-of-gpu%2F16773%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 13, 2022, 3:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmove-trainer-out-of-gpu%2F16773 "2022-04-13T03:34:59Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![clone](https://avatars.discourse-cdn.com/v4/letter/c/d26b3c/32.png) [@clone](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclone)%5C
**Post date:** [April 13, 2022, 3:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmove-trainer-out-of-gpu%2F16773%2F1 "2022-04-13T03:34:59Z")

</div>

Hello! I’m using the Trainer API, which is great, to train a causal language model (gpt) using 2 GPUs in parallel.

Now, I have to train multiple similar models for a few epochs at a time, and I would like to iteratively swap out of the GPUs the current Trainer, and swap in the next one. After a while the cycle would restart, so I would need to maintain the state of all the Trainers at all time.  
I would like to achieve something like:

```python
while not done:
  for trainer in trainer_list:
    move_to_cuda(trainer)
    for epoch in range(epochs):  
      trainer.train()
    move_out_of_cuda(trainer)

```

I would also like to keep using both GPUs to train each model.

Does anyone know how I could move a (multi-GPU) Trainer out of vram?

edit: typos