# Why is Trainer only using 1 (not 4) GPUs?
**URL:**
**Author:**  [@drscotthawley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrscotthawley)%5C
**Post date:** [June 2, 2022, 8:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-trainer-only-using-1-not-4-gpus%2F18667%2F1 "2022-06-02T20:05:37Z")
The Transformers Trainer is only using 1 out of 4 possible GPUs. Why is that?
Hi, I’ve set `CUDA_VISIBLE_DEVICES=0,1,2,3` and `torch.cuda.device_count()` shows 4. But when I run my Trainer, nvtop shows that only GPU 0 is computing anything. I would expect all 4 GPU usage bars in the following screenshot to be all the way up, but devices 1-3 show 0% usage:
I even tried manually setting `trainer.args._n_gpu = 4` (instead of leaving it alone) but it had no effect.
Someone will ask to see the full code for what I’m doing, which is understandable. It is a clone of this Kaggle notebook: [Music Genre Classification with Wav2Vec2 | Kaggle](https://www.kaggle.com/code/lujar1762/music-genre-classification-with-wav2vec2/notebook)
PS- In searching for answers, I notice people asking [how to limit the number of GPUs to 1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-restrict-training-to-one-gpu-if-multiple-are-available-co%2F1244) but I"m trying to get it to do the opposite – use all GPUs! If the latter is supposed to be the default… ?
Do the HuggingFace models not automatically invoke DistributedDataParallel? If that omission is the issue, then…that would be understandable. I don’t see any usage of “parallel” or “accelerate” anywhere in the code. So perhaps I need to add that manually?
---
**Author:**  [@drscotthawley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrscotthawley)%5C
**Post date:** [June 2, 2022, 9:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-trainer-only-using-1-not-4-gpus%2F18667%2F2 "2022-06-02T21:08:27Z")
Tried taking a look at `accelerate` example notebook for SimpleNLP, but it crashes with SIGSEV for me:
> [@Simple NLP Example not working](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F7)%3A
>
> I too am seeing the SIGSEV error when I run the Simple NLP example on an A100 pod: --------------------------------------------------------------------------- ProcessExitedException Traceback (most recent call last) Input In [25], in \() 1 from accelerate import notebook\_launcher ----\> 3 notebook\_launcher(training\_function, num\_processes=4) File ~/envs/wav2vec/lib/python3.9/site-packages/accelerate/launchers.py:129, in notebook\_launcher(function, args, num…
|