# Multi-gpu inference
**URL:**
**Author:**  [@somesaba](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsomesaba)%5C
**Post date:** [May 13, 2024, 11:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-inference%2F86567%2F1 "2024-05-13T23:59:53Z")
Hello, I am trying to maximize inference speed of a single prompt on a small (7B) model. I have a server with 4 GPUs. It seems possible to use accelerate to speed up inference. Does anyone have example code? I only see examples of splitting multiple prompts across GPUs but I only have 1 prompt at a time.
Thank you.
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [May 14, 2024, 12:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-inference%2F86567%2F2 "2024-05-14T00:26:52Z")
If your model fits on a single GPU, that will always be faster than multiple GPUs as doing so would involve communication bandwidth and slow things down some
---
**Author:**  [@somesaba](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsomesaba)%5C
**Post date:** [May 14, 2024, 1:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-gpu-inference%2F86567%2F3 "2024-05-14T01:01:42Z")
Thank you. Is there any other way to improve inference speed? I’m already using an H100 with 4-bit quantization and flash attention 2.