# Why am I out of GPU memory despite using device\_map="auto"?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-am-i-out-of-gpu-memory-despite-using-device-map-auto%2F77933%3E%5C
**Category:** 🤗Accelerate\
**Created:** [March 18, 2024, 1:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-am-i-out-of-gpu-memory-despite-using-device-map-auto%2F77933 "2024-03-18T13:07:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![u7122029](https://avatars.discourse-cdn.com/v4/letter/u/49beb7/32.png) [@u7122029](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fu7122029)%5C
**Post date:** [March 18, 2024, 1:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-am-i-out-of-gpu-memory-despite-using-device-map-auto%2F77933%2F1 "2024-03-18T13:07:14Z")

</div>

I have a NVIDIA RTX 2080Ti and I am trying to load the zephyr model

```auto
model = AutoModelForCausalLM.from_pretrained("HuggingFaceH4/zephyr-7b-beta",
                                             torch_dtype=torch.float16,
                                             device_map="auto")

```

This tutorial [Handling big models for inference (huggingface.co)](https://huggingface.co/docs/accelerate/en/usage_guides/big_modeling#complete-example) says that using `device_map="auto"` will split the large model into smaller chunks, store them in the CPU, and then put them sequentially into the GPU for each input as it passes through each stage of the model. This is especially emphasised by the linked youtube video

[![](https://us1.discourse-cdn.com/hellohellohello/original/3X/a/e/aee47b6cce4c661ec846e65fe711cb5651ca4f3d.jpeg "Accelerate Big Model Inference: How Does it Work?") ](https://www.youtube.com/watch?v=MWCSGj9jEAo)

Therefore I should not easily run out of gpu memory. So why am I getting this warning:

```auto
WARNING:root:Some parameters are on the meta device device because they were 
offloaded to the cpu.

```

If I am not wrong, this is the program essentially telling me “I tried to put the entire model on the GPU, but oh no it doesn’t fit so I will leave some parts in the CPU”  
It is only natural I’d get this error

```auto
WARNING:accelerate.big_modeling:You shouldn't move a model when it is dispatched on multiple devices.
Traceback (most recent call last):
  File "/home/user/main/self-contained/diffusion.py", line 39, in <module>
    model.to(distributed_state)
  File "/home/user/miniconda3/envs/311/lib/python3.11/site-packages/accelerate/big_modeling.py", line 447, in wrapper
    raise RuntimeError("You can't move a model that has some modules offloaded to cpu or disk.")
RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.

```

Could this be a bug? I am thinking this is the case because this does not resemble the behaviour shown in the youtube video at all.

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [March 18, 2024, 1:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-am-i-out-of-gpu-memory-despite-using-device-map-auto%2F77933%2F2 "2024-03-18T13:28:03Z")

</div>

> [@u7122029](#):
>
> `model.to(distributed_state)`

The issue would be here. It’s trying to load the entire model on the GPU

What’s the code you’re using?

---

<div class="post-metadata">

**Author:** ![u7122029](https://avatars.discourse-cdn.com/v4/letter/u/49beb7/32.png) [@u7122029](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fu7122029)%5C
**Post date:** [March 18, 2024, 5:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-am-i-out-of-gpu-memory-despite-using-device-map-auto%2F77933%2F3 "2024-03-18T17:17:45Z")

</div>

The idea my that my machine has 4 2080tis. I originally wanted to give each GPU their own process so that the dataset is split into 4 and inference is done quicker in parallel. To test that the model isn’t automatically using more than one GPU to fit the model, I ran the program with `accelerate` configured to only use one GPU. `distributed_state` is an `Accelerator`. I have also tried setting it to a `PartialState` with no success.

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [March 18, 2024, 7:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-am-i-out-of-gpu-memory-despite-using-device-map-auto%2F77933%2F4 "2024-03-18T19:34:27Z")

</div>

> [@u7122029](#):
>
> I originally wanted to give each GPU their own process so that the dataset is split into 4 and inference is done quicker in parallel.

You should use the pipeline parallelism inference API. What you’re currently doing is loading the entire model on each GPU. [Distributed Inference with 🤗 Accelerate](https://huggingface.co/docs/accelerate/usage_guides/distributed_inference#memory-efficient-pipeline-parallelism-experimental)