# Loading model directly to GPU omitting RAM
**URL:**
**Author:**  [@bikekowal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbikekowal)%5C
**Post date:** [March 26, 2025, 12:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-model-directly-to-gpu-omitting-ram%2F147501%2F1 "2025-03-26T12:52:03Z")
Hi,
I have a problem when loading a LLM model.
While the model is being loaded it is first allocated in RAM memory, then in the GPU. However I have currently only 128 GB of RAM available but two NVIDIA A100 80GB. I would like to omit loading the model first to RAM and instead loading it directly to GPU.
This is the code:
```auto
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
device_map="auto" # automatically places model layers on available devices
)
prompt = "xxxxx."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=50,
do_sample=True,
top_k=50,
top_p=0.9,
temperature=0.7
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
```
I tried with device\_map set as “auto” or “cuda”, but still the model is first loaded to RAM.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 26, 2025, 2:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-model-directly-to-gpu-omitting-ram%2F147501%2F2 "2025-03-26T14:44:39Z")
With device\_map=, accelerate manages memory, so I think it is more reliable to specify it with device for your use.
```py
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
#device_map="auto" # automatically places model layers on available devices
#device_map=0 # maybe cuda:0 only
device="cuda", # "cuda:0" is also ok
).to("cuda") # if necessary
```
---
**Author:**  [@bikekowal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbikekowal)%5C
**Post date:** [March 26, 2025, 3:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-model-directly-to-gpu-omitting-ram%2F147501%2F3 "2025-03-26T15:57:33Z")
Thanks, but these options have already been tried. The same effect, still loading to RAM at the first place.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 26, 2025, 4:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-model-directly-to-gpu-omitting-ram%2F147501%2F4 "2025-03-26T16:14:25Z")
I see, so it’s the Transoformers’ specifications that are using up the RAM…
I thought about offloading it to disk as a last resort, but RAM is better than that…
>
>
> \### Feature request
>
> I would like the ability to lazy load models to the GPU u…sing \`AutoModelForCausalLM.from\_pretrained\`.
>
> At the moment, it is possible to reduce the RAM usage using the \`low\_cpu\_mem\_usage=True\` option, but on systems with more VRAM than RAM (like Google Colab with 12GB RAM and 16GB VRAM), it is not possible to load certain models due to a RAM bottleneck.
>
> \### Motivation
>
> See above
>
> \### Your contribution
>
> \--
>
>
---
**Author:**  [@bikekowal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbikekowal)%5C
**Post date:** [March 26, 2025, 6:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-model-directly-to-gpu-omitting-ram%2F147501%2F5 "2025-03-26T18:37:13Z")
Thank you, the second link seems to have some solution. I will try and let know.
---
**Author:**  [@bikekowal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbikekowal)%5C
**Post date:** [March 28, 2025, 7:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-model-directly-to-gpu-omitting-ram%2F147501%2F6 "2025-03-28T07:09:01Z")
This is strange.
I added “low\_cpu\_mem\_usage=True”, but still received:
```auto
Loading checkpoint shards: 40%|████ | 12/30 [02:35<03:00, 10.05s/it]
Process finished with exit code 137 (interrupted by signal 9:SIGKILL)
```
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 28, 2025, 9:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-model-directly-to-gpu-omitting-ram%2F147501%2F7 "2025-03-28T09:36:25Z")
RAM is consumed, but I don’t think it consumes that much…🤔
To begin with, it’s not that there is a lot of VRAM and not enough RAM.
As a rare case, for example, with old versions of Ubuntu, if the swap settings were wrong, SIGKILL would occur even if there was enough RAM.
>
>