# Cannot launch multi-gpu training?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-launch-multi-gpu-training%2F54987%3E%5C
**Category:** 🤗Transformers\
**Created:** [September 14, 2023, 10:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-launch-multi-gpu-training%2F54987 "2023-09-14T10:36:10Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Data-drone](https://avatars.discourse-cdn.com/v4/letter/d/7ea924/32.png) [@Data-drone](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FData-drone)%5C
**Post date:** [September 14, 2023, 10:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-launch-multi-gpu-training%2F54987%2F1 "2023-09-14T10:36:10Z")

</div>

I am trying to run training on multiple GPUs following this script here:

> <https://github.com/databricks/databricks-ml-examples/blob/master/llm-models/llamav2/llamav2-7b/06_fine_tune_qlora.py>

single gpu is fine. When I switch to multi-gpu I got:

```auto
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! 

```

So I tried wrapping the dataloader in accelerate.prepare.  
then I get:

```auto
AttributeError: 'DataLoaderShard' object has no attribute 'map'

```

What is the correct path forward?