# Can't load huge model onto multiple GPU's
**URL:**
**Author:**  [@fffiend](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffffiend)%5C
**Post date:** [June 9, 2023, 7:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-huge-model-onto-multiple-gpus%2F42698%2F1 "2023-06-09T19:05:23Z")
Hey there, so I have a model that takes 45+ GB to just LOAD, and I’m trying to load the model onto 4 A100 GPU’s, with 40 GB VRAM each and I do the following
model = model.from\_pretrained(…,device\_map=“auto”) and it still just gives me this error:
OutOfMemoryError: CUDA out of memory. Tried to allocate 160.00 MiB (GPU 0; 39.45 GiB total capacity; 38.59 GiB already allocated; 42.25
MiB free; 38.59 GiB reserved in total by PyTorch) If reserved memory is \>\> allocated memory try setting max\_split\_size\_mb to avoid
fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF
I legit have no idea because this is documented on HuggingFace too as a solution that is supposed to work, any pointers?
Thank you.
---
**Author:**  [@drwootton](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrwootton)%5C
**Post date:** [June 14, 2023, 8:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-huge-model-onto-multiple-gpus%2F42698%2F2 "2023-06-14T20:13:35Z")
I had trouble splitting models across 2 GPUs until I also set the max\_memory parameter by creating a Python dict object such as the following
```
limits = {}
for n in range(len(self._maxGPUMemory)):
limits[n] = f'{self._maxGPUMemory[str(n)]}MiB'
params['max_memory'] = limits
```
where printing limits shows me limits {0: ‘10000MiB’, 1: ‘11000MiB’}
I set the memory limit for each GPU a bit less than the GPU memory and then it seems like the loading process worked
---
**Author:**  [@fffiend](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffffiend)%5C
**Post date:** [June 14, 2023, 11:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-huge-model-onto-multiple-gpus%2F42698%2F3 "2023-06-14T23:01:55Z")
Hmm, how does that fix it though? Right now the model is nearly loaded on the first GPU but it straight up doesn’t recognize the other GPUs, does that mean that the memory limit on the others is 0?
---
**Author:**  [@drwootton](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrwootton)%5C
**Post date:** [June 15, 2023, 1:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-huge-model-onto-multiple-gpus%2F42698%2F5 "2023-06-15T01:06:21Z")
I’m not sure how memory allocations on GPUs is done. I’m just learning how this works myself, and using the max\_memory parameter seemed like it helped in my case. I’m guessing that without it, the allocation process tries to load to just one GPU and runs out of memory while setting max\_memory value for each of the GPUs, 0, 1, 2, 3 to a lower limit solves this.
This page [Handling big models for inference](https://huggingface.co/docs/accelerate/main/en/usage_guides/big_modeling#limits-and-further-development) also mentions using infer\_auto\_device\_map which attempts to map model layers to GPUs,instead of using an ‘auto’ map.
---
**Author:**  [@fffiend](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffffiend)%5C
**Post date:** [June 15, 2023, 3:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-huge-model-onto-multiple-gpus%2F42698%2F6 "2023-06-15T15:15:34Z")
Hmm alright. Could you show me a snippet of how you set the device memory limit ?
---
**Author:**  [@drwootton](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrwootton)%5C
**Post date:** [June 15, 2023, 11:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-huge-model-onto-multiple-gpus%2F42698%2F7 "2023-06-15T23:19:07Z")
I have a Python dict object named ‘params’ that I set all the parameters I need to load the model. I have the memory limit values for both my GPUs in another Python dict object loaded from a JSON profile file.
I set up ‘limits’ to contain my memory limit settings as
```auto
limits = {}
for n in range(len(self._maxGPUMemory)):
limits[n] = f'{self._maxGPUMemory[str(n)]}MiB'
if (self._useCPU):
limits['cpu'] = f'{self._maxCPUMemory}MiB'
params['max_memory'] = limits
```
After I set up the remaining params parameter values, I load the model as
```auto
model = AutoModelForCausalLM.from_pretrained(self._modelPath, **params)
```
In the case where I want to build a device map instead of using ‘auto’ as my device\_map I call accelerators.infer\_auto\_device\_map as
```auto
params['device_map'] = infer_auto_device_map(model, max_memory=params['max_memory'], no_split_module_classes=model._no_split_modules, dtype=torchDType)
```