# qloRA with cpu offload

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqlora-with-cpu-offload%2F51543%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 21, 2023, 3:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqlora-with-cpu-offload%2F51543 "2023-08-21T15:52:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![cerisara](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cerisara/32/5907_2.png) [@cerisara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcerisara)%5C
**Post date:** [August 21, 2023, 3:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqlora-with-cpu-offload%2F51543%2F1 "2023-08-21T15:52:39Z")

</div>

Hi,

I’m facing an error: “ValueError: You can’t train a model that has been loaded in 8-bit precision with CPU or disk offload.” when I tried to train LoRA parameters (all of them are on the GPU) with all transformer parameters frozen and in 4bits (bnb) on the GPU. The trick is that to save VRAM, I’m offloading the (unquantized) embeddings onto the cpu (with a device\_map). This shouldn’t be a problem to train LoRA parameters, but apparently it’s not supported?  
Any work around existing?

Thanks!

---

<div class="post-metadata">

**Author:** ![saireddy](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@saireddy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaireddy)%5C
**Post date:** [February 22, 2024, 3:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqlora-with-cpu-offload%2F51543%2F2 "2024-02-22T03:08:14Z")

</div>

Hey @cerisara ,  
were you able to find solution for this?