# How to run inference on multigpus

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-run-inference-on-multigpus%2F128452%3E%5C
**Category:** 🤗Accelerate\
**Created:** [November 29, 2024, 2:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-run-inference-on-multigpus%2F128452 "2024-11-29T14:19:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![LearnToGrow](https://avatars.discourse-cdn.com/v4/letter/l/5fc32e/32.png) [@LearnToGrow](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLearnToGrow)%5C
**Post date:** [November 29, 2024, 2:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-run-inference-on-multigpus%2F128452%2F1 "2024-11-29T14:19:27Z")

</div>

I am using LLama3.1 70B for inference. I have 4 gpus nvidia L4 (24GB) each. Here is my code:

```auto
nf4_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_use_double_quant=True,
  bnb_4bit_quant_type="nf4",
  bnb_4bit_compute_dtype=torch.bfloat16)
  llm_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-70B-Instruct",
                                                        quantization_config=nf4_config,
                                                        device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Llama-3.1-70B-Instruct", use_fast=True)
#Add padding in case we need to use batch_size > 1
self.tokenizer.padding_side = "left"
self.tokenizer.pad_token = self.tokenizer.eos_token

def run_llm(llm_model, tokenizer, prompt_messages: list[str],
                temperature: float = 0.001, batch_size, tokenizer_config, generation_config) -> list[dict]:
        """
        """
        data_loader = torch.utils.data.DataLoader(
            prompt_messages, batch_size=batch_size)
        tqdm_iterator = tqdm(data_loader, desc="Inference LLM model")

        outputs = []
        # Make a copy of the current generation config
        with torch.no_grad():
            for batch in tqdm_iterator:
                inputs_model = tokenizer(
                    batch, return_tensors="pt", **tokenizer_config)
                inputs_model.to(llm_model.device)

                model_input_length = len(inputs_model[0])
                output_encode = llm_model.generate( **inputs_model,** generation_config, pad_token_id=self.tokenizer.eos_token_id,
                                                        temperature=temperature)

                output_encode = output_encode[:, model_input_length:]
                output = self.tokenizer.batch_decode(
                    output_encode, skip_special_tokens=True)
                outputs.extend(output)
        return outputs

```

I remark that the model is split on all 4 gpus but the inference is running only on 1 GPU as depicted below:  
How Can I optimize the code to run the inference on 4 multiple gpus?

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/8/8/88b68ed0ffd24f212a3e809930216874d7ef25cc.png)