# Multi-GPU is slower than single GPU when running examples
**URL:**
**Author:**  [@duncanswilson](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fduncanswilson)%5C
**Post date:** [July 11, 2024, 8:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-is-slower-than-single-gpu-when-running-examples%2F96885%2F1 "2024-07-11T20:17:43Z")
I have a machine with 3 3090s and have been using accelerate with [lm\_eval](https://github.com/EleutherAI/lm-evaluation-harness/) to speed up inference and seeing sensible results.
I wrote some custom training scripts using accelerate but noticed about a 3x slowdown vs the single GPU case. While debugging, I decided to try the `nlp_example.py` and I’m getting significant slowdowns between the single GPU and multi-GPU case there too.
I’ve found this [topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsingle-gpu-is-faster-than-multiple-gpus%2F71383) on a similar slowdown, but I am able to see performance gains with accelerate and lm\_eval, so I doubt that it is a CUDA/pytorch version incompatibility…
For reference, when running `nlp_example.py` on one GPU I am getting 44 secs total for three epochs and for the multiGPU case I get 4 mins.
I’m happy to provide any other information (package versions and CUDA versions) if that is needed.
---
**Author:**  [@duncanswilson](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fduncanswilson)%5C
**Post date:** [July 24, 2024, 7:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-is-slower-than-single-gpu-when-running-examples%2F96885%2F2 "2024-07-24T19:33:21Z")
If anyone else runs into this issue: it turned out to be a BIOS level change that was needed in order to fix the communication overhead. Specifically changing the link speed on the PCI ports from Gen 1 to Gen 4 resulted in seeing speedups using multiple GPUs for fine-tuning!
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [July 25, 2024, 7:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-is-slower-than-single-gpu-when-running-examples%2F96885%2F3 "2024-07-25T07:33:26Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.