# Distributed Training w/ Trainer
**URL:**
**Author:**  [@josephgatto](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjosephgatto)%5C
**Post date:** [July 7, 2021, 4:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F1 "2021-07-07T16:21:39Z")
Does anyone have an end-to-end example of how to do multi-gpu, multi-node distributed training using the trainer? I can’t seem to find one anywhere.
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 7, 2021, 5:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F2 "2021-07-07T17:52:14Z")
All the examples using the `Trainer` run in multi-gpu multi-node, you just have to use the PyTorch launcher to properly launch a multi-GPU multinode training.
---
**Author:**  [@josephgatto](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjosephgatto)%5C
**Post date:** [July 7, 2021, 6:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F3 "2021-07-07T18:03:40Z")
So is there no code adjustments that need to be made, only how the file is launched?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 7, 2021, 6:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F4 "2021-07-07T18:08:41Z")
Yes, the `Trainer` will deal with all the rest by itself.
---
**Author:**  [@SSamDav](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSSamDav)%5C
**Post date:** [July 23, 2021, 8:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F5 "2021-07-23T08:33:27Z")
Hi I’m trying to run a multi-node training using the Trainer class, for that I run my script with the `python -m torch.distributed.launch --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr="IP" \ --master_port=1234`, however, the script doesn’t wait for the master node. Also when I run in the master node the script doesn’t wait for the child node. Should I set up any env variable? The only thing that I’m doing is passing the local\_rank to the TrainingArgs.
Thanks for the help!
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 26, 2021, 1:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F6 "2021-07-26T13:05:20Z")
It’s hard to know what the problem could be without seeing the script you are launching.
---
**Author:**  [@Lee1990](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLee1990)%5C
**Post date:** [April 24, 2022, 9:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F7 "2022-04-24T09:02:37Z")
what do you mean by “use the PyTorch launcher to properly launch a multi-GPU multinode training” ?
---
**Author:**  [@nlp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnlp)%5C
**Post date:** [May 31, 2022, 10:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F8 "2022-05-31T10:10:45Z")
It doesnt work with Longformer. Is this expected ?
---
**Author:**  [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [May 31, 2022, 1:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F9 "2022-05-31T13:31:53Z")
[https://pytorch.org/docs/stable/distributed.html#launch-utility](https://pytorch.org/docs/stable/distributed.html#launch-utility)
---
**Author:**  [@keval-sha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkeval-sha)%5C
**Post date:** [April 14, 2025, 8:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F10 "2025-04-14T20:27:41Z")
Are there any examples of running a distributed training job using pytorch on k8s, specifically GKE?
---
**Author:**  [@theharshithh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftheharshithh)%5C
**Post date:** [May 20, 2025, 1:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F11 "2025-05-20T13:09:14Z")
Use raylib for distributed training. here is a guide put up by verl for mutli-node training in ray.
> **[Multinode Training — verl documentation](https://verl.readthedocs.io/en/latest/start/multinode.html#set-up-multinode-ray-cluster)**
---
**Author:**  [@dag332](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdag332)%5C
**Post date:** [June 3, 2025, 6:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-training-w-trainer%2F8151%2F12 "2025-06-03T18:14:56Z")
if you want to do multi-node, multi-GPU training make sure to use the `local_process_index` instead of the `process_index`. Otherwise the device count doesn’t account for the GPUs on the separate node
```auto
from trl import SFTConfig, SFTTrainer
from accelerate import PartialState
device_string = PartialState().local_process_index
model = AutoModelForCausalLM.from_pretrained(
...
device_map={'':device_string}
)
```