# Qwen-VL Parallel GPU run not able to solve

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqwen-vl-parallel-gpu-run-not-able-to-solve%2F55218%3E%5C
**Category:** Models\
**Created:** [September 16, 2023, 7:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqwen-vl-parallel-gpu-run-not-able-to-solve%2F55218 "2023-09-16T07:43:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Velan](https://avatars.discourse-cdn.com/v4/letter/v/49beb7/32.png) [@Velan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVelan)%5C
**Post date:** [September 16, 2023, 7:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqwen-vl-parallel-gpu-run-not-able-to-solve%2F55218%2F1 "2023-09-16T07:43:28Z")

</div>

Im not able to pass url after setting device map to Auto.

```auto
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig
import torch
torch.manual_seed(1234)

# Note: The default behavior now has injection attack prevention off.
tokenizer = AutoTokenizer.from_pretrained("bibimbap/Qwen-VL-Chat", trust_remote_code=True)

# use bf16
# model = AutoModelForCausalLM.from_pretrained("bibimbap/Qwen-VL-Chat", device_map="auto", trust_remote_code=True, bf16=True).eval()
# use fp16
model = AutoModelForCausalLM.from_pretrained("bibimbap/Qwen-VL-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()
# use cpu only
# model = AutoModelForCausalLM.from_pretrained("bibimbap/Qwen-VL-Chat", device_map="cpu", trust_remote_code=True).eval()
# use cuda device
#model = AutoModelForCausalLM.from_pretrained("bibimbap/Qwen-VL-Chat", device_map="cuda", trust_remote_code=True).eval()

# Specify hyperparameters for generation
model.generation_config = GenerationConfig.from_pretrained("bibimbap/Qwen-VL-Chat", trust_remote_code=True)

# 1st dialogue turn
query = tokenizer.from_list_format([
    {'image': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'}, # Either a local path or an url
    {'text': 'What is this?'},
])
response, history = model.chat(tokenizer, query=query, history=None)
print(response)
# 图中是一名女子在沙滩上和狗玩耍，旁边是一只拉布拉多犬，它们处于沙滩上。

# 2nd dialogue turn
response, history = model.chat(tokenizer, 'Hello!', history=history)
print(response)
# <ref>击掌</ref><box>(536,509),(588,602)</box>
image = tokenizer.draw_bbox_on_latest_picture(response, history)
if image:
  image.save('1.jpg')
else:
  print("no box")

```

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:3!

i keep getting this error.  
1.On running the model on fp32, everything works but it is very slow. So this is not an option i want to use.  
2.On loading all tensors to one gpu i am getting cuda out of memory error.  
3.I am also getting errors RuntimeError: “addmm\_impl\_cpu\_” not implemented for ‘Half’ and slow\_conv2d\_cpu not implemented for ‘half’ on running parallelly.  
GPU server used:  
we have azure server Standard\_NC64as\_T4\_v3, we have gpu with GPU memeory of 64 GIB ram and it has . It looks like it’s taking 16 gb ram.It has 64 vcpu, memroy of 440 GiB. It has 4 NVIDIA T4 GPUs with 16 GB of memory each, up to 64 non-multithreaded AMD EPYC 7V12 (Rome) processor cores and 448 GiB of system memory.

Please prove a solution for this problem

---

<div class="post-metadata">

**Author:** ![Nishgop](https://avatars.discourse-cdn.com/v4/letter/n/f05b48/32.png) [@Nishgop](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNishgop)%5C
**Post date:** [February 20, 2024, 4:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqwen-vl-parallel-gpu-run-not-able-to-solve%2F55218%2F2 "2024-02-20T16:50:32Z")

</div>

Did you get this issue resolved?