# Recommended Approach for Distributed Inference

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-approach-for-distributed-inference%2F21001%3E%5C
**Category:** 🤗Optimum\
**Created:** [July 29, 2022, 2:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-approach-for-distributed-inference%2F21001 "2022-07-29T14:01:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cambonator](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@cambonator](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcambonator)%5C
**Post date:** [July 29, 2022, 2:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-approach-for-distributed-inference%2F21001%2F1 "2022-07-29T14:01:55Z")

</div>

I am looking to run inference with optimum in a distributed setting with PyTorch (multi-node, multi-CPU/GPU). Is there a recommended approach to do this? My data is coming from a HF Datasets object.

I tried using [this solution](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-trainer-at-inference-time%2F9378%2F7) with the HF Trainer, but it gives me an error when I run it with an optimum model (the optimum model does not have an eval() function).

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [July 30, 2022, 7:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-approach-for-distributed-inference%2F21001%2F2 "2022-07-30T07:12:22Z")

</div>

You could have multiple `ORTModelForXXX` classes and each one could use a different device and then iterate over your dataset either sync or async with a queue

---

<div class="post-metadata">

**Author:** ![cambonator](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@cambonator](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcambonator)%5C
**Post date:** [August 1, 2022, 7:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-approach-for-distributed-inference%2F21001%2F3 "2022-08-01T19:55:04Z")

</div>

So in this solution would I be using something like torch.distributed to manage this process and aggregate the prediction results? Are there any code examples you could point me to?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [August 1, 2022, 8:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-approach-for-distributed-inference%2F21001%2F4 "2022-08-01T20:34:52Z")

</div>

Or just python threads. There is no need for `torch.distributed`. No we don’t have any code example for that.