# Stable diffusion \`train\_text\_to\_image.py\` only on one gpu

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-train-text-to-image-py-only-on-one-gpu%2F31466%3E%5C
**Category:** 🧨 Diffusers\
**Created:** [February 10, 2023, 6:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-train-text-to-image-py-only-on-one-gpu%2F31466 "2023-02-10T18:38:45Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 10, 2023, 6:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-train-text-to-image-py-only-on-one-gpu%2F31466%2F1 "2023-02-10T18:38:45Z")

</div>

I’m training a stable diffusion model using a modified version of the `train_text_to_image.py` script. I’m noticing that it’s only running on one (of two) gpus. Is there a simple fix for this? Is it possible to run this script on multiple gpus? (It seems like it should be, since there are references to both “parallel” and “distributed” in the code…)

How do I enable multi-gpu? Or, how might I approach debugging why it’s not working?

EDIT: I see that Accelerate is supposed to handle this dynamically. So I guess I’m curious where to start looking for the problem.

---

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 11, 2023, 12:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-train-text-to-image-py-only-on-one-gpu%2F31466%2F2 "2023-02-11T00:24:28Z")

</div>

I’m seeing references to `accelerator.num_processes`, but I can’t set it… ?? Can I only set it from a config file?

More info…

config:

```auto
compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: MULTI_GPU
downcast_bf16: 'no'
dynamo_backend: 'NO'
fsdp_config: {}
gpu_ids: all
machine_rank: 0
main_training_function: main
megatron_lm_config: {}
mixed_precision: fp16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
use_cpu: false

```

At launch I’m seeing this:

```auto
02/10/2023 18:46:44 - INFO - __main__ - Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda
Mixed precision type: fp16

```

Is there something I have to set to _use_ the `default_config.yaml` file?

`accelerate test` seems okay:

```auto
stdout: Distributed environment: MULTI_GPU Backend: nccl
stdout: Num processes: 2
stdout: Process index: 1
stdout: Local process index: 1
stdout: Device: cuda:1
stdout: Mixed precision type: fp16
stdout: 
stdout: **Initialization**
stdout: Testing, testing. 1, 2, 3.
stdout: Distributed environment: MULTI_GPU Backend: nccl
stdout: Num processes: 2
stdout: Process index: 0
stdout: Local process index: 0
stdout: Device: cuda:0
stdout: Mixed precision type: fp16
stdout: 
stdout: 
stdout: **Test random number generator synchronization**
stdout: All rng are properly synched.
stdout: 
stdout: **DataLoader integration test**
stdout: 0 1 tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
stdout: 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
stdout: 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
stdout: 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], device='cuda:0') <class 'accelerate.data_loader.DataLoaderShard'>
stdout: tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
stdout: 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
stdout: 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
stdout: 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], device='cuda:1') <class 'accelerate.data_loader.DataLoaderShard'>
stdout: Non-shuffled dataloader passing.
stdout: Shuffled dataloader passing.
stdout: Non-shuffled central dataloader passing.
stdout: Shuffled central dataloader passing.
stdout: 
stdout: **Training integration test**
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Training yielded the same results on one CPU or distributed setup with no batch split.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: FP16 training check.
stdout: Training yielded the same results on one CPU or distributes setup with batch split.
stdout: FP16 training check.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: BF16 training check.
stdout: BF16 training check.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
Test is a success! You are ready for your distributed training!

```

---

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [February 11, 2023, 1:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-train-text-to-image-py-only-on-one-gpu%2F31466%2F3 "2023-02-11T01:10:09Z")

</div>

oh… my… goodness…

not  
`python /my/bloody/script.py`  
but  
`accelerate launch /my/bloody/script.py`… yikes… 🤦‍♂️

---

<div class="post-metadata">

**Author:** ![alexchenyu](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@alexchenyu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falexchenyu)%5C
**Post date:** [April 28, 2023, 5:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-train-text-to-image-py-only-on-one-gpu%2F31466%2F4 "2023-04-28T17:31:07Z")

</div>

Hi did you figure it out? Can you run the script on multiple gpus now?

---

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [April 28, 2023, 8:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-train-text-to-image-py-only-on-one-gpu%2F31466%2F5 "2023-04-28T20:18:17Z")

</div>

Yeah, you have run the `accelerate config` script, but once you’ve done that, just call it with `accelerate launch` rather than `python`. Then accelerate does all the heavy lifting for you!

---

<div class="post-metadata">

**Author:** ![williamberman](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/williamberman/32/14112_2.png) [@williamberman](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwilliamberman)%5C
**Post date:** [May 2, 2023, 8:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-train-text-to-image-py-only-on-one-gpu%2F31466%2F6 "2023-05-02T20:50:08Z")

</div>

Hey @jbmaxwell yes that is correct, sorry for the confusion. For context, any python script in multi gpu is usually launched differently than just running a standard script as multiple processes have to be spawned and orchestrated.

Here’s some more related docs if they’re helpful

> **[Launching your 🤗 Accelerate scripts](https://huggingface.co/docs/accelerate/basic_tutorials/launch#using-accelerate-launch)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

[https://pytorch.org/docs/stable/elastic/run.html](https://pytorch.org/docs/stable/elastic/run.html)