# Inference with CPU offload

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-with-cpu-offload%2F50305%3E%5C
**Category:** 🤗Accelerate\
**Created:** [August 10, 2023, 3:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-with-cpu-offload%2F50305 "2023-08-10T15:43:09Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![volsk](https://avatars.discourse-cdn.com/v4/letter/v/fbc32d/32.png) [@volsk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvolsk)%5C
**Post date:** [August 10, 2023, 3:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-with-cpu-offload%2F50305%2F1 "2023-08-10T15:43:09Z")

</div>

Hi,  
I want to infer Falcon40b model on GPU with CPU offload.  
I use `device_map="auto"` parameter in `AutoModelForCausalLM.from_pretrained()` method.  
I expect that all maximum space available on GPU will be used and then model will be offloaded to CPU.  
But I checked memory consumption and it turns out that only 414Mb out of 40Gb VRAM (1 A100) and almost 100% of RAM are used. So it seems that model is almost completely offloaded to CPU.

How to set GPU to be a primary device and offload to CPU after there is not available space on GPU?