# Learning rate for the \`Trainer\` in a multi gpu setup
**URL:**
**Author:**  [@marouen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarouen)%5C
**Post date:** [April 29, 2024, 2:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flearning-rate-for-the-trainer-in-a-multi-gpu-setup%2F84360%2F1 "2024-04-29T14:20:38Z")
Im training using the `trainer` class on a multi gpu setup.
I know that when using `accelerate` ([Comparing performance between different device setups](https://huggingface.co/docs/accelerate/concept_guides/performance#learning-rates)), in order to train with the desired learning rate we have to explicitely multiply by the number of gpus.
- Is that also the case when using the trainer class?
- In the case of warmup steps: should the same be applied? i.e. `n_warmup_steps *= n_gpus` ?
- In the case of a learning rate scheduler: should the same be applied too?
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [April 29, 2024, 4:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flearning-rate-for-the-trainer-in-a-multi-gpu-setup%2F84360%2F2 "2024-04-29T16:21:58Z")
> [@marouen](#):
>
> Is that also the case when using the trainer class?
Yes.
Wrt warmup steps, it may need to be, Im’ unsure off the top of my head
---
**Author:**  [@marouen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarouen)%5C
**Post date:** [April 29, 2024, 4:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flearning-rate-for-the-trainer-in-a-multi-gpu-setup%2F84360%2F3 "2024-04-29T16:41:50Z")
Thanks for answering, so if I pass some **`lr`** to either `TrainingArguments` `learning_rate` or to the `Trainer` `optimizers`, backprop actually occurs with **`lr / n_gpus`**. Is my understanding correct?
In that case, wouldnt it be less prone to confusion to call it (similarly to the batch size), `learning_rate_per_device`?
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [April 29, 2024, 5:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flearning-rate-for-the-trainer-in-a-multi-gpu-setup%2F84360%2F4 "2024-04-29T17:32:57Z")
> [@marouen](#):
>
> learning\_rate\_per\_device
Not necessarily, because it’s a huristic that people recommend to do so, but it’s also recommended to test yourself at your discretion.
What’s _really_ happening is the number of steps increases that we’re stepping the learning rate, so if you want the same LR from situation A to B you should try multiplying the learning rate.
However again: test yourself first. Sometimes it’s not necessary
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [April 30, 2024, 8:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flearning-rate-for-the-trainer-in-a-multi-gpu-setup%2F84360%2F5 "2024-04-30T08:44:12Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.