# Memory consumption qlora with gradient checkpointing

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmemory-consumption-qlora-with-gradient-checkpointing%2F70948%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 28, 2024, 5:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmemory-consumption-qlora-with-gradient-checkpointing%2F70948 "2024-01-28T17:14:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![cerisara](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cerisara/32/5907_2.png) [@cerisara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcerisara)%5C
**Post date:** [January 28, 2024, 5:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmemory-consumption-qlora-with-gradient-checkpointing%2F70948%2F1 "2024-01-28T17:14:18Z")

</div>

Hi !  
I’ve used qlora with gradient checkpointing on llama-2-7b and I’m surprised by the huge quantity of VRAM it’s taking when calling forward on a 2577 tokens: before the forward pass, it was using only 4.8GB, and during forward, crashed with OOM on a A100 80GB !

I used gradient checkpointing with:

```auto
model = AutoModelForCausalLM.from_pretrained(wd, load_in_4bit = True)
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing = True)
model.gradient_checkpointing_enable()
peft_config = LoraConfig(r = 8, task_type = TaskType.CAUSAL_LM)
model = get_peft_model(model, peft_config)

```

It shouldn’t use that much VRAM with gradient checkpointing, should it?

Thanks,  
Christophe