# Multi gpu not working
**URL:**
**Author:**  [@anon47283947](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon47283947)%5C
**Post date:** [February 3, 2023, 3:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-not-working%2F31059%2F1 "2023-02-03T15:53:14Z")
Hi,
I am using huggingface run\_clm.py to train gptj-6b model with 8 gpu’s. But it is not using all gpus and throwing cuda out of memory error. I have tried changing batch\_size with multiple of gpus. is there anything I have to mention for using all gpus?
python3 run\_clm.py
–model\_name\_or\_path EleutherAI/gpt-j-6B
–dataset\_name glue
–dataset\_config\_name cola
–per\_device\_train\_batch\_size 8
–per\_device\_eval\_batch\_size 8
–do\_train
–do\_eval
–output\_dir /tmp/gptneo20b\_100
–num\_train\_epochs=100
–weight\_decay=0.01
–learning\_rate=1e-5
this is what am currently running.
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [February 3, 2023, 7:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-not-working%2F31059%2F2 "2023-02-03T19:50:43Z")
> [@anon47283947](#):
>
> python3 run\_clm.py
You need to launch with `torchrun --n_procs_per_node=NGPUS run_clm.py ...` in order to enable multi-GPU. Another option is to use Accelerate’s CLI launcher directly:
```bash
accelerate launch --multi_gpu --num_processes=NGPUS run_clm.py...
```
---
**Author:**  [@anon47283947](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon47283947)%5C
**Post date:** [February 3, 2023, 8:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-not-working%2F31059%2F3 "2023-02-03T20:11:08Z")
> [@muellerzr](#):
>
> torch
Here, it shows all 8 gpu’s with same memory error. And why pytorch reserving memory in all 8gpu’s? and not 1.
(Using accelerate launcher)