# More processes than GPUs with DeepSpeed launcher

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmore-processes-than-gpus-with-deepspeed-launcher%2F70633%3E%5C
**Category:** DeepSpeed\
**Created:** [January 25, 2024, 8:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmore-processes-than-gpus-with-deepspeed-launcher%2F70633 "2024-01-25T20:07:17Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lucius42](https://avatars.discourse-cdn.com/v4/letter/l/8e8cbc/32.png) [@Lucius42](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLucius42)%5C
**Post date:** [January 25, 2024, 8:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmore-processes-than-gpus-with-deepspeed-launcher%2F70633%2F1 "2024-01-25T20:07:17Z")

</div>

Hello,

I am training LoRA adaptation of a T5 model in a one-machine multiple GPU setup.  
I am using Transformers 4.26.1 and DeepSpeed 0.9.2 and launching my script with deepspeed (thus the parallelization setup is Distributed Data Parallel).

Just before training, I first get four processes in my first GPU (probably the four models loaded in right?)

When the training starts, I have four processes running in my first GPU and then one process for each other GPU. Although I am using regular PyTorch I have the same issue as in this repo : [extra process when running ddp across multiple GPUs · Lightning-AI/pytorch-lightning · Discussion #9864 · GitHub](https://github.com/Lightning-AI/pytorch-lightning/discussions/9864).

See image

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/a/3/a33c2eeca37cd7b9b738b7e07712eb2aa7f252df.png)

Is it normal? I have only seen examples with one process per GPU. I would be greatly interested in an explanation.

Thanks !