# Accelerate socket timeout on multi-node LLM training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-socket-timeout-on-multi-node-llm-training%2F92170%3E%5C
**Category:** Intermediate\
**Created:** [June 14, 2024, 9:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-socket-timeout-on-multi-node-llm-training%2F92170 "2024-06-14T09:33:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![LayZ007](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@LayZ007](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLayZ007)%5C
**Post date:** [June 14, 2024, 9:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-socket-timeout-on-multi-node-llm-training%2F92170%2F1 "2024-06-14T09:33:49Z")

</div>

I have been trying recently to distribute the fine-tuning of an LLM model on multi-node (multiple machines). I have two machines with a GTX 3060 each the first machine (host, 172.21.202.40) is configured as follows :  
`compute_environment: LOCAL_MACHINE debug: true distributed_type: MULTI_GPU downcast_bf16: 'no' enable_cpu_affinity: false gpu_ids: '[0]' machine_rank: 0 main_process_ip: 172.21.202.40 main_process_port: 5000 main_training_function: main mixed_precision: bf16 num_machines: 2 num_processes: 2 rdzv_backend: static same_network: true tpu_env: [] tpu_use_cluster: false tpu_use_sudo: false use_cpu: false`

The other machine (172.21.201.195) is configured like the following :  
`compute_environment: LOCAL_MACHINE debug: true distributed_type: MULTI_GPU downcast_bf16: 'no' enable_cpu_affinity: false gpu_ids: '[0]' machine_rank: 0 main_process_ip: 172.21.202.40 main_process_port: 5000 main_training_function: main mixed_precision: bf16 num_machines: 2 num_processes: 2 rdzv_backend: static same_network: true tpu_env: [] tpu_use_cluster: false tpu_use_sudo: false use_cpu: false`

I try to run the tst file given by Accelerate using the command `accelerate test`, the two machines seem to be hanging until i get a timeout error. I checked the traffic between the two machines using wireshark and they do seem to exchange data. The stream i caught is as follows :

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/6/7/67b92a3f605e3b195f0059a3d4170c65f69ab069.png)

I am not sure what the binary code is. But the data transfer seems to be working fine, both my terminals hang on this line `Running: accelerate-launch /home/ubuntu/.local/lib/python3.10/site-packages/accelerate/test_utils/scripts/test_script.py` and after while it send a socket timeout error.