# No GPUs found in distributed mode

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-distributed-mode%2F32773%3E%5C
**Category:** 🤗Accelerate\
**Created:** [March 1, 2023, 5:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-distributed-mode%2F32773 "2023-03-01T05:26:02Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Trusure](https://avatars.discourse-cdn.com/v4/letter/t/df705f/32.png) [@Trusure](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTrusure)%5C
**Post date:** [March 1, 2023, 5:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-distributed-mode%2F32773%2F1 "2023-03-01T05:26:02Z")

</div>

I have use accelerate config to set multi-GPU distributed environment, as below:

```auto
- `Accelerate` version: 0.16.0
- Platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.31
- Python version: 3.9.13
- Numpy version: 1.21.5
- PyTorch version (GPU?): 1.13.0+cu116 (True)
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: fp16
        - use_cpu: False
        - dynamo_backend: NO
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: [0,1]
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {}
        - fsdp_config: {}
        - megatron_lm_config: {}
        - downcast_bf16: no

```

But when I use accelerate test to check it out, I found accelerate cannot find GPUs. I am sure that there are many GPUs in my machine.

```auto
Running: accelerate-launch --config_file=None /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py
stderr: ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_scrip │
stderr: │ t.py:336 in <module> │
stderr: │ │
stderr: │ 333 │
stderr: │ 334 │
stderr: │ 335 if __name__ == " __main__": │
stderr: │ ❱ 336 │ main() │
stderr: │ 337 │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_scrip │
stderr: │ t.py:305 in main │
stderr: │ │
stderr: │ 302 │
stderr: │ 303 │
stderr: │ 304 def main(): │
stderr: │ ❱ 305 │ accelerator = Accelerator() │
stderr: │ 306 │ state = accelerator.state │
stderr: │ 307 │ if state.local_process_index == 0: │
stderr: │ 308 │ │ print(" **Initialization**") │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/accelerator.py:323 in │
stderr: │ __init__ │
stderr: │ │
stderr: │ 320 │ │ │ │ │ │ self.init_handler = handler │
stderr: │ 321 │ │ │
stderr: │ 322 │ │ kwargs = self.init_handler.to_kwargs() if self.init_handler is not None else {} │
stderr: │ ❱ 323 │ │ self.state = AcceleratorState( │
stderr: │ 324 │ │ │ mixed_precision=mixed_precision, │
stderr: │ 325 │ │ │ cpu=cpu, │
stderr: │ 326 │ │ │ dynamo_backend=dynamo_backend, │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/state.py:162 in __init__ │
stderr: │ │
stderr: │ 159 │ │ │ elif int(os.environ.get("LOCAL_RANK", -1)) != -1 and not cpu: │
stderr: │ 160 │ │ │ │ self.distributed_type = DistributedType.MULTI_GPU │
stderr: │ 161 │ │ │ │ if not torch.distributed.is_initialized(): │
stderr: │ ❱ 162 │ │ │ │ │ torch.distributed.init_process_group(backend="nccl", **kwargs) │
stderr: │ 163 │ │ │ │ │ self.backend = "nccl" │
stderr: │ 164 │ │ │ │ self.num_processes = torch.distributed.get_world_size() │
stderr: │ 165 │ │ │ │ self.process_index = torch.distributed.get_rank() │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:76 │
stderr: │ 1 in init_process_group │
stderr: │ │
stderr: │ 758 │ │ │ # different systems (e.g. RPC) in case the store is multi-tenant. │
stderr: │ 759 │ │ │ store = PrefixStore("default_pg", store) │
stderr: │ 760 │ │ │
stderr: │ ❱ 761 │ │ default_pg = _new_process_group_helper( │
stderr: │ 762 │ │ │ world_size, │
stderr: │ 763 │ │ │ rank, │
stderr: │ 764 │ │ │ [], │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:89 │
stderr: │ 7 in _new_process_group_helper │
stderr: │ │
stderr: │ 894 │ │ │ │ pg_options.is_high_priority_stream = False │
stderr: │ 895 │ │ │ │ pg_options._timeout = timeout │
stderr: │ 896 │ │ │ │
stderr: │ ❱ 897 │ │ │ pg = ProcessGroupNCCL(prefix_store, group_rank, group_size, pg_options) │
stderr: │ 898 │ │ │ # In debug mode and if GLOO is available, wrap in a wrapper PG that │
stderr: │ 899 │ │ │ # enables enhanced collective checking for debugability. │
stderr: │ 900 │ │ │ if get_debug_level() == DebugLevel.DETAIL: │
stderr: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
stderr: RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!
stderr: ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_scrip │
stderr: │ t.py:336 in <module> │
stderr: │ │
stderr: │ 333 │
stderr: │ 334 │
stderr: │ 335 if __name__ == " __main__": │
stderr: │ ❱ 336 │ main() │
stderr: │ 337 │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_scrip │
stderr: │ t.py:305 in main │
stderr: │ │
stderr: │ 302 │
stderr: │ 303 │
stderr: │ 304 def main(): │
stderr: │ ❱ 305 │ accelerator = Accelerator() │
stderr: │ 306 │ state = accelerator.state │
stderr: │ 307 │ if state.local_process_index == 0: │
stderr: │ 308 │ │ print(" **Initialization**") │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/accelerator.py:323 in │
stderr: │ __init__ │
stderr: │ │
stderr: │ 320 │ │ │ │ │ │ self.init_handler = handler │
stderr: │ 321 │ │ │
stderr: │ 322 │ │ kwargs = self.init_handler.to_kwargs() if self.init_handler is not None else {} │
stderr: │ ❱ 323 │ │ self.state = AcceleratorState( │
stderr: │ 324 │ │ │ mixed_precision=mixed_precision, │
stderr: │ 325 │ │ │ cpu=cpu, │
stderr: │ 326 │ │ │ dynamo_backend=dynamo_backend, │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/state.py:162 in __init__ │
stderr: │ │
stderr: │ 159 │ │ │ elif int(os.environ.get("LOCAL_RANK", -1)) != -1 and not cpu: │
stderr: │ 160 │ │ │ │ self.distributed_type = DistributedType.MULTI_GPU │
stderr: │ 161 │ │ │ │ if not torch.distributed.is_initialized(): │
stderr: │ ❱ 162 │ │ │ │ │ torch.distributed.init_process_group(backend="nccl", **kwargs) │
stderr: │ 163 │ │ │ │ │ self.backend = "nccl" │
stderr: │ 164 │ │ │ │ self.num_processes = torch.distributed.get_world_size() │
stderr: │ 165 │ │ │ │ self.process_index = torch.distributed.get_rank() │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:76 │
stderr: │ 1 in init_process_group │
stderr: │ │
stderr: │ 758 │ │ │ # different systems (e.g. RPC) in case the store is multi-tenant. │
stderr: │ 759 │ │ │ store = PrefixStore("default_pg", store) │
stderr: │ 760 │ │ │
stderr: │ ❱ 761 │ │ default_pg = _new_process_group_helper( │
stderr: │ 762 │ │ │ world_size, │
stderr: │ 763 │ │ │ rank, │
stderr: │ 764 │ │ │ [], │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:89 │
stderr: │ 7 in _new_process_group_helper │
stderr: │ │
stderr: │ 894 │ │ │ │ pg_options.is_high_priority_stream = False │
stderr: │ 895 │ │ │ │ pg_options._timeout = timeout │
stderr: │ 896 │ │ │ │
stderr: │ ❱ 897 │ │ │ pg = ProcessGroupNCCL(prefix_store, group_rank, group_size, pg_options) │
stderr: │ 898 │ │ │ # In debug mode and if GLOO is available, wrap in a wrapper PG that │
stderr: │ 899 │ │ │ # enables enhanced collective checking for debugability. │
stderr: │ 900 │ │ │ if get_debug_level() == DebugLevel.DETAIL: │
stderr: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
stderr: RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!

```