# GPU memory usage of optimizer's states when using LoRA
**URL:**
**Author:**  [@YalunHu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYalunHu)%5C
**Post date:** [July 2, 2024, 4:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-memory-usage-of-optimizers-states-when-using-lora%2F95063%2F1 "2024-07-02T04:10:33Z")
Hi, recently I am using LoRA(by using peft) + transformers Trainer + DeepSpeed(ZeRO3) to finetune my model (around 7B params). Before this, I tried full-param finetune as well.
The wierd thing is that I found LoRA seems not help to reduce the GPU memory usage compared with the full-param finetune method.
My question is : when using LoRA, I checkecd that a lot of paramerters’s `requires_grad` is set to `False`, will this help to reduce the Adam optimizer’s GPU memory usage? Since I think setting `requires_grad` to `False` must help reduce the memory usage of gradients tesnor, but the GPU mem usage did not reduce a lot, so I was wondering if it is because the GPU-mem taken by the optimizer’s states is still as the same as before? Will the optimizer record the states of model weights which does not need the gradient?
---
**Author:**  [@terjenf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fterjenf)%5C
**Post date:** [July 5, 2024, 12:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-memory-usage-of-optimizers-states-when-using-lora%2F95063%2F2 "2024-07-05T00:56:41Z")
Hi,
Sorry for diverging from your question, but I don’t find so much info about this online. How do you merge the adapters resulting from the Lora+Zero3 finetuning back to the base model?
---
**Author:**  [@YalunHu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYalunHu)%5C
**Post date:** [July 5, 2024, 1:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-memory-usage-of-optimizers-states-when-using-lora%2F95063%2F3 "2024-07-05T01:12:10Z")
Solved! I finally find out that it is because I didn’t set the `gradient_checkpointing=True` during my LoRA training, which takes a lot of GPU memory!
---
**Author:**  [@YalunHu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYalunHu)%5C
**Post date:** [July 5, 2024, 1:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-memory-usage-of-optimizers-states-when-using-lora%2F95063%2F4 "2024-07-05T01:13:35Z")
Ah I just call the PeftModel’s `.merge_and_unload` method. It will change the base model’s weights in-place.
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [July 5, 2024, 1:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-memory-usage-of-optimizers-states-when-using-lora%2F95063%2F5 "2024-07-05T13:14:09Z")
Closed on July 5, 2024, 1:14pm UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.