# How to perform training on CPU +GPU offloading?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-perform-training-on-cpu-gpu-offloading%2F66288%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 19, 2023, 6:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-perform-training-on-cpu-gpu-offloading%2F66288 "2023-12-19T06:27:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saugatkafley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saugatkafley/32/26738_2.png) [@Saugatkafley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSaugatkafley)%5C
**Post date:** [December 19, 2023, 6:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-perform-training-on-cpu-gpu-offloading%2F66288%2F1 "2023-12-19T06:27:03Z")

</div>

I wanted to fine-tune `bloom-7b1` using qlora with 1x3090(24 GB) and Nvidia Titan(12 GB) .  
Is there way to offload weights to CPU ?

The [peft github](https://github.com/huggingface/peft?tab=readme-ov-file#getting-started) has shown offloading results . How to perform oflloading in qlora?

---

<div class="post-metadata">

**Author:** ![enochlev](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/enochlev/32/22267_2.png) [@enochlev](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fenochlev)%5C
**Post date:** [December 19, 2023, 7:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-perform-training-on-cpu-gpu-offloading%2F66288%2F2 "2023-12-19T19:09:03Z")

</div>

mess around with the device\_map function

model = AutoModelForCausalLM.from\_pretrained(  
base\_model,  
quantization\_config=quant\_config,#load\_in\_8bit=True,  
device\_map={“”: Accelerator().local\_process\_index},

```
)

```

check out this

> **[Quantization](https://huggingface.co/docs/transformers/main/en/quantization#offloading)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

notice how they offload layers to diffrent gpus/cpus (in there case 0 is gpu\_0 and 1 will be your other gpu)