# Trainer) training one batch with multiple GPUs

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-training-one-batch-with-multiple-gpus%2F43722%3E%5C
**Category:** DeepSpeed\
**Created:** [June 19, 2023, 2:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-training-one-batch-with-multiple-gpus%2F43722 "2023-06-19T02:24:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeesoo](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@jeesoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjeesoo)%5C
**Post date:** [June 19, 2023, 2:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-training-one-batch-with-multiple-gpus%2F43722%2F1 "2023-06-19T02:24:29Z")

</div>

Hi, there.  
I’m using huggingFace Trainer code to train gpt-based large language model.  
The size is more than 8b.  
when I use input sequence length = 2048 tokens, and the per\_device\_train\_batch\_size=1, it seems it doesn’t fit on A100 (40GB) GPU. How can I load one batch to multiple gpus? It seems like that I ‘must’ load more than one batch on one gpu.