# Using Accelerate on an HPC (Slurm) **URL:** **Author:** ![CamilleP](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@CamilleP](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCamilleP)%5C **Post date:** [May 21, 2021, 8:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F1 "2021-05-21T08:52:48Z") Hi, I am performing some tests with Accelerate on an HPC (where slurm is usually how we distribute computation). It works on one node and multiple GPU but now I want to try a multi node setup. I will use your launcher `accelerate launch --config_file ` but then I need to be able to update a couple of the fields from the json file in my script (so during the creation of the Accelerator ?) : - main\_process\_ip - machine\_rank How can I do that ? Will it be working ? I am right to think that if my setup is two nodes, each one with 4 GPU, the (range of) value(s) should be: - for “num\_process”: 8 (the number of gpu) - for “num\_machine”: 2 - for “machine rank”: [0,1] - for “distributed\_type” : “MULTI\_GPU” Thanks ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C **Post date:** [May 21, 2021, 11:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F2 "2021-05-21T11:50:35Z")
How do you usually distribute in multi-node with slurm? In PyTorch distributed, the `main_process_ip` is the IP address of the machine of rank 0, so it should work if you enter that. ---
**Author:** ![CamilleP](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@CamilleP](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCamilleP)%5C **Post date:** [May 21, 2021, 1:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F3 "2021-05-21T13:24:37Z")
In slurm, there is srun that launches as many instances of the scripts as there is number of nodes x task (ie process ) Then, from within the script we can retrieve all the slurm environment variable that we need (specifically for the master task and the (local) rank of a process - that is all that is necessary for “dist.init\_process\_group” in pure pytorch ddp. What I see as a problem is that I can’t know in advance the nodes (among hundreds available) on which my script will run, so I can’t fully build the config.json file in advance. That is why I would like to be able to update the main\_process\_ip and the machine rank within the script. I don’t know if that makes sense (it is also how we use horovod, ie the setup happens inside the script). ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C **Post date:** [May 21, 2021, 1:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F4 "2021-05-21T13:58:53Z")
It doesn’t look like the Accelerate launcher can help you here, but there is no problem using your usual launching script. ---
**Author:** ![CamilleP](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@CamilleP](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCamilleP)%5C **Post date:** [May 21, 2021, 2:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F5 "2021-05-21T14:01:31Z")
Ok thanks, that is probably easier. But then how can I let accelerate know who is the master node ? Is there some arg in Accelerator() that can be set ? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C **Post date:** [May 21, 2021, 2:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F6 "2021-05-21T14:02:53Z")
It detects everything from the environment (like PyTorch DDP) so you shouldn’t have to worry about anything. ---
**Author:** ![CamilleP](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@CamilleP](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCamilleP)%5C **Post date:** [May 21, 2021, 2:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F7 "2021-05-21T14:04:15Z")
Ok, I will give it a try. It works on one node with several GPU, so I only have one more step to overcome. Thanks for your answers ! ---
**Author:** ![limk1m](https://avatars.discourse-cdn.com/v4/letter/l/ee7513/32.png) [@limk1m](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flimk1m)%5C **Post date:** [September 2, 2021, 8:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F8 "2021-09-02T08:43:57Z")
hi, do u solve this problem? I have the same problem with u. I can’t know which machine node in advance, so how to config accelerate ---
**Author:** ![CamilleP](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@CamilleP](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCamilleP)%5C **Post date:** [September 8, 2021, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F9 "2021-09-08T11:07:41Z")
Hello, Sorry I can’t help you right now. I was actually waiting for the tool to be more mature and/or more documented before trying again. I will post in this thread if I find the time to resume my tests. Don’t hesitate to post if you find a solution, I would be very interested to read it and try it! ---
**Author:** ![Dahoas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dahoas/32/5492_2.png) [@Dahoas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDahoas)%5C **Post date:** [July 25, 2022, 7:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F10 "2022-07-25T07:42:20Z")
Any update on this? ---
**Author:** ![smangrul](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smangrul/32/3791_2.png) [@smangrul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmangrul)%5C **Post date:** [July 27, 2022, 5:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-accelerate-on-an-hpc-slurm%2F6286%2F11 "2022-07-27T05:05:26Z")
Hello @Dahoas , A user seems to have provided some approach in this comment on a related issue: [Using accelerate config for SLURM cluster with dist\_url input · Issue #145 · huggingface/accelerate · GitHub](https://github.com/huggingface/accelerate/issues/145#issuecomment-1184817922) Here is the Git Gist from that comment: [distributed dalle2 laion (github.com)](https://gist.github.com/rom1504/474f97a95a526d40ae44a3fc3c657a2e) Could you please try it out and let us know if that works?