# Distributed training large models on cloud resources
**URL:**
**Author:**  [@rmiller3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frmiller3)%5C
**Post date:** [March 20, 2024, 5:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-large-models-on-cloud-resources%2F78307%2F1 "2024-03-20T17:22:29Z")
I’m trying to fine tune mosaicml/mpt-7b-instruct using cloud resources ([modal.com](http://modal.com)) with mutliple GPUs but still running out of memory even on the large GPUs.
I’m using the huggingface trainer to try this and confirmed the training\_args resolved n\_gpu to 2 or 4 depending on how many I configure modal to have.
Can someone help me understand if separate scripts like torchrun (or maybe accelerate launch…) are required? Some information I read seems to conflict.
For example, the official docs show using torchrun (and run\_clm for other forms):
[https://huggingface.co/docs/transformers/en/perf\_train\_gpu\_many](https://huggingface.co/docs/transformers/en/perf_train_gpu_many)
This stackoverflow seems to agree: [machine learning - How to use Huggingface Trainer with multiple GPUs? - Stack Overflow](https://stackoverflow.com/questions/75814047/how-to-use-huggingface-trainer-with-multiple-gpus)
* * *
However, these forum posts imply that the trainer should work 'out of the box' if n\_gpu is proper: \> /static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-gpt2-using-multiple-gpu-and-trainer%2F796%2F10
This says someone was surprised that their process was forked: [How to restrict training to one GPU if multiple are available, co - #3 by dropout05](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-restrict-training-to-one-gpu-if-multiple-are-available-co%2F1244%2F3)
* * *
As a separate question maybe warranting a separate thread later, what should the torchrun training script do? Load the model, preprocess the dataset and call train? No special parameters to tell it to run on multiple machines?
* * *
Is there any way torchrun can be encapsulated in a single training call with parameters to split the load across all GPU?
Note: I’m running this on a cloud service ([modal.com](http://modal.com)) so I’m assuming it’s not running on multiple GPUs but I can’t really see the GPU load as far as I know.
---
**Author:**  [@rmiller3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frmiller3)%5C
**Post date:** [March 20, 2024, 7:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-large-models-on-cloud-resources%2F78307%2F2 "2024-03-20T19:37:00Z")
Perhaps the seemingly conflicting information comes from the definition of distributed (distributed across multiple nodes / machines vs distributed across GPUs).
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [March 20, 2024, 7:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-large-models-on-cloud-resources%2F78307%2F3 "2024-03-20T19:46:19Z")
> [@rmiller3](#):
>
> Can someone help me understand if separate scripts like torchrun (or maybe accelerate launch…) are required? Some information I read seems to conflict.
Have you configured something like DeepSpeed or FSDP? Native DDP (what it’s doing rn) still loads all of the model on each GPU. You want a sharding mechanism that will split the model up onto each GPU during training
---
**Author:**  [@rmiller3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frmiller3)%5C
**Post date:** [March 20, 2024, 8:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-large-models-on-cloud-resources%2F78307%2F4 "2024-03-20T20:34:44Z")
Thanks, I’ll look into those now. I hadn’t.
---
**Author:**  [@singhjagpreet](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsinghjagpreet)%5C
**Post date:** [March 27, 2024, 7:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-large-models-on-cloud-resources%2F78307%2F5 "2024-03-27T07:28:41Z")
Have you tried accelerate
> **[Accelerate](https://huggingface.co/docs/accelerate/en/index)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@rmiller3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frmiller3)%5C
**Post date:** [March 27, 2024, 6:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-large-models-on-cloud-resources%2F78307%2F6 "2024-03-27T18:38:50Z")
I got it working from a tutorial on [modal.com](http://modal.com) which uses accelerate. Working backwards off the github mentioned here. [Fine-tune an LLM in minutes (ft. Llama 2, CodeLlama, Mistral, etc.) | Modal Docs](https://modal.com/docs/examples/llm-finetuning)
mosaicml/mpt-7b had some hiccups since the default configs use peft and it was missing some function that needed it, but I was able to do some finetuning there.
The tutorial uses accelerate, but has a deprecated code using torch.distributed which I’m also trying to get working. I’m not sure of the pros and cons of one vs the other yet (torch.distributed vs accelerate).
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [March 27, 2024, 7:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-large-models-on-cloud-resources%2F78307%2F7 "2024-03-27T19:25:15Z")
> [@rmiller3](#):
>
> (torch.distributed vs accelerate)
They are synonymous, accelerate just wraps around it 🙂 (For on GPUs)