# Flan-t5 multi gpu training issue

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fflan-t5-multi-gpu-training-issue%2F35387%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 3, 2023, 8:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fflan-t5-multi-gpu-training-issue%2F35387 "2023-04-03T08:26:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![shreyans92dhankhar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shreyans92dhankhar/32/4909_2.png) [@shreyans92dhankhar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fshreyans92dhankhar)%5C
**Post date:** [April 3, 2023, 8:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fflan-t5-multi-gpu-training-issue%2F35387%2F1 "2023-04-03T08:26:18Z")

</div>

Hi,

I am loading flan t5 xxl sharded version using “philschmid/flan-t5-xxl-sharded-fp16” for finetuning. I want to do multi-gpu training using this. After loading the model from AutoModelForSeq2SeqLM, i am setting this parameter setattr(model, ‘model\_parallel’, True) but i am getting following error while training using trainer.  
AttributeError: ‘T5Stack’ object has no attribute ‘first\_device’

Can someone help with this issue?