# Running inference on tuned models across multiple gpus

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-inference-on-tuned-models-across-multiple-gpus%2F101406%3E%5C
**Category:** Models\
**Created:** [August 8, 2024, 1:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-inference-on-tuned-models-across-multiple-gpus%2F101406 "2024-08-08T01:52:35Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ericp](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@ericp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fericp)%5C
**Post date:** [August 8, 2024, 1:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-inference-on-tuned-models-across-multiple-gpus%2F101406%2F1 "2024-08-08T01:52:35Z")

</div>

Im having a tough time running my tuned model across multiple gpus

I have various pt files that i tuned with torchtune. I can inference with their generate function on lora but not full precision as one of my cards cant hold the whole model. Wondering the right approach to do this I have tried various methods but am struggling\>

- hf\_model\_0001\_2.pt
- hf\_model\_0002\_2.pt
- hf\_model\_0003\_2.pt
- hf\_model\_0004\_2.pt