# Why my Accelerate just doesn't work? **URL:** **Author:** ![beyond](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beyond/32/28447_2.png) [@beyond](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbeyond)%5C **Post date:** [March 6, 2022, 2:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-my-accelerate-just-doesnt-work%2F15409%2F1 "2022-03-06T14:09:23Z") **env:** torch1.7 -cuda11.0 -nccl2.7.8 8 V100 GPUs ubuntu **cmds:** ```python pip install transformers pip install accelerate ``` then I set up with `accelerate config`: ```auto Which type of machine are you using? ([0] No distributed training, [1] multi-CPU, [2] multi-GPU, [3] TPU): 2 How many different machines will you use (use more than 1 for multi-node training)? [1]: 1 Do you want to use DeepSpeed? [yes/NO]: No How many processes in total will you use? [1]: 4 Do you wish to use FP16 (mixed precision)? [yes/NO]: no ``` Now the error comes out, when I check with `accelerate test`, I got: ```auto Running: accelerate-launch --config_file=None /home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py stdout: ***************************************** stdout: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. stdout: ***************************************** stdout: Distributed environment: MULTI_GPU Backend: nccl stdout: Num processes: 4 stdout: Process index: 2 stdout: Local process index: 2 stdout: Device: cuda:2 stdout: Use FP16 precision: False stdout: stdout: Distributed environment: MULTI_GPU Backend: nccl stdout: Num processes: 4 stdout: Process index: 3 stdout: Local process index: 3 stdout: Device: cuda:3 stdout: Use FP16 precision: False stdout: stdout: Distributed environment: MULTI_GPU Backend: nccl stdout: Num processes: 4 stdout: Process index: 1 stdout: Local process index: 1 stdout: Device: cuda:1 stdout: Use FP16 precision: False stdout: stdout: **Initialization** stdout: Testing, testing. 1, 2, 3. stdout: Distributed environment: MULTI_GPU Backend: nccl stdout: Num processes: 4 stdout: Process index: 0 stdout: Local process index: 0 stdout: Device: cuda:0 stdout: Use FP16 precision: False stdout: stdout: stdout: **Test random number generator synchronization** stdout: All rng are properly synched. stdout: stdout: **DataLoader integration test** stdout: Non-shuffled dataloader passing. stdout: Shuffled dataloader passing. stderr: Traceback (most recent call last): stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 291, in stderr: Traceback (most recent call last): stderr: main() stderr: Traceback (most recent call last): stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 283, in main stderr: Traceback (most recent call last): stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 291, in stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 291, in stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 291, in stderr: central_dl_preparation_check() stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 120, in central_dl_preparation_check stderr: for batch in dl: stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/data_loader.py", line 362, in __iter__ stderr: broadcast_object_list(batch_info) stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/utils.py", line 403, in broadcast_object_list stderr: torch.distributed.broadcast_object_list(object_list, src=from_process) stderr: AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list' stderr: main() stderr: main() stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 283, in main stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 283, in main stderr: main() stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 283, in main stderr: central_dl_preparation_check() stderr: central_dl_preparation_check() stderr: central_dl_preparation_check() stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 120, in central_dl_preparation_check stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 120, in central_dl_preparation_check stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 120, in central_dl_preparation_check stderr: for batch in dl: stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/data_loader.py", line 362, in __iter__ stderr: for batch in dl: stderr: for batch in dl: stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/data_loader.py", line 362, in __iter__ stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/data_loader.py", line 362, in __iter__ stderr: broadcast_object_list(batch_info) stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/utils.py", line 403, in broadcast_object_list stderr: broadcast_object_list(batch_info) stderr: broadcast_object_list(batch_info) stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/utils.py", line 403, in broadcast_object_list stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/utils.py", line 403, in broadcast_object_list stderr: torch.distributed.broadcast_object_list(object_list, src=from_process) stderr: torch.distributed.broadcast_object_list(object_list, src=from_process) stderr: torch.distributed.broadcast_object_list(object_list, src=from_process) stderr: AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list' stderr: AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list' stderr: AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list' stderr: Traceback (most recent call last): stderr: File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main stderr: " __main__", mod_spec) stderr: File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code stderr: exec(code, run_globals) stderr: File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 261, in stderr: main() stderr: File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 257, in main stderr: cmd=cmd) stderr: subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-u', '/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py']' returned non-zero exit status 1. stderr: Traceback (most recent call last): stderr: File "/home/me/.local/bin/accelerate-launch", line 8, in stderr: sys.exit(main()) stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/commands/launch.py", line 390, in main stderr: launch_command(args) stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/commands/launch.py", line 378, in launch_command stderr: multi_gpu_launcher(args) stderr: File "/home/me/.local/lib/python3.6/site-packages/accelerate/commands/launch.py", line 176, in multi_gpu_launcher stderr: raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) stderr: subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'torch.distributed.launch', '--use_env', '--nproc_per_node', '4', '/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py']' returned non-zero exit status 1. Traceback (most recent call last): File "/home/me/.local/bin/accelerate", line 8, in sys.exit(main()) File "/home/me/.local/lib/python3.6/site-packages/accelerate/commands/accelerate_cli.py", line 41, in main args.func(args) File "/home/me/.local/lib/python3.6/site-packages/accelerate/commands/test.py", line 52, in test_command result = execute_subprocess_async(cmd, env=os.environ.copy()) File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/testing.py", line 135, in execute_subprocess_async f"'{cmd_str}' failed with returncode {result.returncode}\n\n" RuntimeError: 'accelerate-launch --config_file=None /home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py' failed with returncode 1 The combined stderr from workers follows: Traceback (most recent call last): File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 291, in Traceback (most recent call last): main() Traceback (most recent call last): File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 283, in main Traceback (most recent call last): File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 291, in File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 291, in File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 291, in central_dl_preparation_check() File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 120, in central_dl_preparation_check for batch in dl: File "/home/me/.local/lib/python3.6/site-packages/accelerate/data_loader.py", line 362, in __iter__ broadcast_object_list(batch_info) File "/home/me/.local/lib/python3.6/site-packages/accelerate/utils.py", line 403, in broadcast_object_list torch.distributed.broadcast_object_list(object_list, src=from_process) AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list' main() main() File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 283, in main File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 283, in main main() File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 283, in main central_dl_preparation_check() central_dl_preparation_check() central_dl_preparation_check() File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 120, in central_dl_preparation_check File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 120, in central_dl_preparation_check File "/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py", line 120, in central_dl_preparation_check for batch in dl: File "/home/me/.local/lib/python3.6/site-packages/accelerate/data_loader.py", line 362, in __iter__ for batch in dl: for batch in dl: File "/home/me/.local/lib/python3.6/site-packages/accelerate/data_loader.py", line 362, in __iter__ File "/home/me/.local/lib/python3.6/site-packages/accelerate/data_loader.py", line 362, in __iter__ broadcast_object_list(batch_info) File "/home/me/.local/lib/python3.6/site-packages/accelerate/utils.py", line 403, in broadcast_object_list broadcast_object_list(batch_info) broadcast_object_list(batch_info) File "/home/me/.local/lib/python3.6/site-packages/accelerate/utils.py", line 403, in broadcast_object_list File "/home/me/.local/lib/python3.6/site-packages/accelerate/utils.py", line 403, in broadcast_object_list torch.distributed.broadcast_object_list(object_list, src=from_process) torch.distributed.broadcast_object_list(object_list, src=from_process) torch.distributed.broadcast_object_list(object_list, src=from_process) AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list' AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list' AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list' Traceback (most recent call last): File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main " __main__", mod_spec) File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 261, in main() File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 257, in main cmd=cmd) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-u', '/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py']' returned non-zero exit status 1. Traceback (most recent call last): File "/home/me/.local/bin/accelerate-launch", line 8, in sys.exit(main()) File "/home/me/.local/lib/python3.6/site-packages/accelerate/commands/launch.py", line 390, in main launch_command(args) File "/home/me/.local/lib/python3.6/site-packages/accelerate/commands/launch.py", line 378, in launch_command multi_gpu_launcher(args) File "/home/me/.local/lib/python3.6/site-packages/accelerate/commands/launch.py", line 176, in multi_gpu_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'torch.distributed.launch', '--use_env', '--nproc_per_node', '4', '/home/me/.local/lib/python3.6/site-packages/accelerate/test_utils/test_script.py']' returned non-zero exit status 1. ``` what are all these things? I have googled the msg `AttributeError: module 'torch.distributed' has no attribute 'broadcast_object_list'` for hundreds of times but I don’t know how to fix it. Could any one help me to figure out? Thanks a lot!!! ---
**Author:** ![beyond](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beyond/32/28447_2.png) [@beyond](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbeyond)%5C **Post date:** [March 7, 2022, 10:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-my-accelerate-just-doesnt-work%2F15409%2F2 "2022-03-07T10:13:09Z")
@sgugger could you please help me? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C **Post date:** [March 7, 2022, 1:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-my-accelerate-just-doesnt-work%2F15409%2F3 "2022-03-07T13:00:39Z")
Will have a look later today. I think it’s a problem with older versions of PyTorch so if you can upgrade, it should solve the issue, but will fix for older versions 🙂