# Inference on Multi-GPU/multinode
**URL:**
**Author:**  [@gfatigati](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgfatigati)%5C
**Post date:** [December 21, 2022, 10:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-on-multi-gpu-multinode%2F28217%2F1 "2022-12-21T10:59:37Z")
Dear Huggingface community,
I’m using Owl-Vit in order to analyze a lot of input images, passing a set of labels. At the moment, my code works well but run just on 1 GPU:
```auto
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-large-patch14")
processor = OwlViTProcessor.from_pretrained("google/owlvit-large-patch14")
....
inputs = processor(text=text_queries, images=image_rgb, return_tensors="pt").to(device)
# Set model in evaluation mode
model = model.to(device)
model.eval()
# Get predictions
with torch.no_grad():
outputs = model(**inputs)
....
```
At the moment, it takes 4 hours to process 31.000 input images. Could you suggest how to change the above code in order to run on more Gpus? The multigpu guide section on Huggingface is under construction. I’m using a supercomputing machine, having 4 GPUs per node. I would like to run also on multi node if possible.
Thanks in advance.
---
**Author:**  [@IdoAmit198](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIdoAmit198)%5C
**Post date:** [December 21, 2022, 8:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-on-multi-gpu-multinode%2F28217%2F2 "2022-12-21T20:08:30Z")
You can try to utilize [accelerate](https://huggingface.co/docs/transformers/accelerate).
In this link you can see how to modify a code similar to yours in order to integrate the accelerate library, which can take care of the distributed setup for you.
I didn’t work with it directly for long so I might forget the specific details. Like whether you need to pass it your nodes/GPUs or not, if you do how to, but I’m sure you can easily find all those details 🙂
By the way, I just came across this [recent post](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-does-multi-gpu-do-under-the-hood-and-how-to-use-it%2F27999)%2C which might also come handy to your needs
---
**Author:**  [@gfatigati](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgfatigati)%5C
**Post date:** [December 22, 2022, 7:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-on-multi-gpu-multinode%2F28217%2F4 "2022-12-22T07:01:16Z")
Thanks,
from what I understand, accelerate is used to distribute training. Does it works also with inference?
---
**Author:**  [@IdoAmit198](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIdoAmit198)%5C
**Post date:** [December 22, 2022, 7:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-on-multi-gpu-multinode%2F28217%2F5 "2022-12-22T07:35:59Z")
As far as I know, it does.
It should work all the same, but without the need to initialize an optimizer, scheduler, etc, using the accelerator, and only init the device, eval\_dataloader, model with the accelerator.
In case it won’t work for for some reason there are more other wrappers to run distributed inference with (which also give a speed up), such as [Optimum](https://huggingface.co/docs/optimum/quicktour) (made to accelerate inference).
In addition it’s worth to mention you can always do it the “hard” way and implement stuff with [torch.nn.DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) or with [torch.nn.parallel.DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel).
Then you can run your code via the [torchrun console script](https://pytorch.org/docs/stable/elastic/run.html). But again, I personally find this method harder than a wrapper like accelerate or Optimum.
---
**Author:**  [@gfatigati](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgfatigati)%5C
**Post date:** [January 12, 2023, 9:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-on-multi-gpu-multinode%2F28217%2F6 "2023-01-12T09:50:27Z")
Hi,
thanks for your reply. I’m reading Optimum User Guide, in particular: [Quick tour](https://huggingface.co/docs/optimum/quicktour) Accelerated Inference section.
In such section, is uses a tokenizer and and ort\_model. My actual code is quite different and I don’t understand how to integrate Optimum in Owl-Vit:
```auto
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-large-patch14")
processor = OwlViTProcessor.from_pretrained("google/owlvit-large-patch14")
```
Any help or some examples reference? I didn’t find anything. Thanks