# Running mT5 on multiple GPUs

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-mt5-on-multiple-gpus%2F20895%3E%5C
**Category:** Models\
**Created:** [July 26, 2022, 7:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-mt5-on-multiple-gpus%2F20895 "2022-07-26T19:49:47Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![rafae015](https://avatars.discourse-cdn.com/v4/letter/r/77aa72/32.png) [@rafae015](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frafae015)%5C
**Post date:** [July 26, 2022, 7:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-mt5-on-multiple-gpus%2F20895%2F1 "2022-07-26T19:49:47Z")

</div>

Hi everyone,  
I am trying to finetune the Google mT5 XXL model for MTNT French-English data. I use 2 32GB GPUs and I am using accelerate package for the model finetuning. However, I am getting a “CUDA out of memory” error for both GPUs. It appears that the script first runs on GPU 0 and then tries running on GPU 1 and therefore give the memory error for both GPUs.  
Thank you.