# I have a question about multi-GPU inference

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-have-a-question-about-multi-gpu-inference%2F33384%3E%5C
**Category:** DeepSpeed\
**Created:** [March 9, 2023, 1:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-have-a-question-about-multi-gpu-inference%2F33384 "2023-03-09T01:51:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![gaoxt1983](https://avatars.discourse-cdn.com/v4/letter/g/b4bc9f/32.png) [@gaoxt1983](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgaoxt1983)%5C
**Post date:** [March 9, 2023, 1:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-have-a-question-about-multi-gpu-inference%2F33384%2F1 "2023-03-09T01:51:24Z")

</div>

In the inference tutorial: [Getting Started with DeepSpeed for Inferencing Transformer based Models - DeepSpeed](https://www.deepspeed.ai/tutorials/inference-tutorial/) , for this example:

```auto
# Filename: gpt-neo-2.7b-generation.py
import os
import deepspeed
import torch
from transformers import pipeline

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))
generator = pipeline('text-generation', model='EleutherAI/gpt-neo-2.7B',
                     device=local_rank)

generator.model = deepspeed.init_inference(generator.model,
                                           mp_size=world_size,
                                           dtype=torch.float,
                                           replace_with_kernel_inject=True)

string = generator("DeepSpeed is", do_sample=True, min_length=50)
if not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0:
    print(string)

```

I want to know:

1. if I’m using deepspeed --num\_gpus 2 gpt-neo-2.7b-generation.py, the “generator” statement runs once or twice?
2. should I do something necessary for different rank of machine?