# Cuda out of memory error
**URL:**
**Author:**  [@prashanth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprashanth)%5C
**Post date:** [May 14, 2022, 6:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F1 "2022-05-14T18:37:40Z")
I encounter the below error when I finetune my dataset on mbart
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 10.76 GiB total capacity; 9.57 GiB already allocated; 16.25 MiB free; 9.70 GiB reserved in total by PyTorch) If reserved memory is \>\> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CON
my train data contains only 5000 sentences. Could anyone of you help me in sorting this out?
Prashanth
---
**Author:**  [@prashanth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprashanth)%5C
**Post date:** [May 14, 2022, 6:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F2 "2022-05-14T18:38:24Z")
@sgugger can you please help me in resolving this error?
---
**Author:**  [@kkumari06](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkkumari06)%5C
**Post date:** [May 16, 2022, 5:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F3 "2022-05-16T05:03:11Z")
Hello @prashanth , you can try out reducing the batch\_size or enable [gradient-checkpointing](https://huggingface.co/docs/transformers/main/en/performance#gradient-checkpointing) or can do training in [fp16](https://huggingface.co/docs/transformers/main/en/performance#fp16-training) to save memory.
---
**Author:**  [@JLinton](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJLinton)%5C
**Post date:** [May 17, 2022, 5:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F4 "2022-05-17T17:53:40Z")
As @kkumari06 says, reduce batch size. I recommend restarting the kernel any time you get this error, to make sure you have a clean GPU memory; then cut the batch size in half. Repeat until it fits in GPU memory or until you hit batch size of 1… in which case, you’ll need to switch to a smaller pretrained model. (If training a model from scratch, you can instead reduce the size of your model, for example by reducing maximum input size or reducing number of layers.) Finally, you may want to bump up the gradient accumulation if your batch size is very small. For example, if you have a batch size of 4, gradient accumulation of 8 would give you an “effective” batch size of 32, which some research suggests is ideal… however, YMMV.
---
**Author:**  [@Raisa06](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRaisa06)%5C
**Post date:** [December 10, 2023, 1:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F5 "2023-12-10T13:33:54Z")
I’m hyper tuning bert-multilingual-uncased model for NER use case. I’m using AWS EC2 instance g4dn.metal which has 8 GPU. My training sample contact 110k samples. I tried the model training instance having 4 GPU I got CUDA out of memory error.
Tried reducing batch size, clear cache, setting up max\_split\_size (pytorch memory management) didn’t fix the error. So I started the model in bigger instance with 8 GPU still facing the same error.
Can someone please help me out in this?
@kkumari06 , @prashanth
---
**Author:**  [@OxxoCodes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOxxoCodes)%5C
**Post date:** [December 21, 2023, 11:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F6 "2023-12-21T23:13:14Z")
@Raisa06 Number of GPUs doesn’t matter unless you’re being very particular to ensure the model is being split among those GPUs. Instead of 1/8th the model being on each GPU, you likely have 8 full copies of the model. Aim for GPUs with more memory per card. Also, if you have already decreased batch size and gradient accumulation steps as much as possible, try using distilbert-base-multilingual-cased, which is a smaller version of the model you’re training. Also, with HF’s accelerate library you can enable deepspeed which is a free memory decrease, and you can also use it to train with mixed precision to further decrease memory usage.
---
**Author:**  [@Pekka10](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPekka10)%5C
**Post date:** [February 26, 2024, 10:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F7 "2024-02-26T10:25:36Z")
Hi @Raisa06
Getting the same issue were you able to resolve it or not and how you clear the cache for the instance as I am using sagemaker for fine tuning.
---
**Author:**  [@ppallavalli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fppallavalli)%5C
**Post date:** [March 16, 2024, 11:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F8 "2024-03-16T23:43:15Z")
Is it possible to clear the GPU storage by any chance? Even when requests are not being made, the gpu storage does not get cleared
---
**Author:**  [@akshat-kumar-akight](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fakshat-kumar-akight)%5C
**Post date:** [April 4, 2024, 1:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F9 "2024-04-04T13:45:51Z")
Remember to delete unneeded tensors and variables. You should also move the tensors and variables that you want to store to cpu and then call torch.cuda.empty\_cache() to free up GPU space, after you’re done with each batch
---
**Author:**  [@codewithRiz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FcodewithRiz)%5C
**Post date:** [July 17, 2024, 2:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F10 "2024-07-17T14:57:13Z")
by reducing the batch size my error got solved im using kaggle notebook with dual gpu.
note training proces will slow
---
**Author:**  [@devcnn5](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdevcnn5)%5C
**Post date:** [December 20, 2024, 7:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F11 "2024-12-20T07:10:24Z")
Try to set max\_length=1024 or 512 in your tokenize function which you’re using to tokenize the dataset.
For example in my tokenize function i set it like so:
example[‘input\_ids’] = tokenizer(prompt, padding=“max\_length”, truncation=True, return\_tensors=“pt”, max\_length=1024).input\_ids
example[‘labels’] = tokenizer(example[‘answer’], padding=“max\_length”, truncation=True, return\_tensors=“pt”, max\_length=1024).input\_ids
---
**Author:**  [@vinaysan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvinaysan)%5C
**Post date:** [January 27, 2025, 10:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-error%2F17959%2F12 "2025-01-27T10:00:56Z")
Thank you. This is especially important for models like deberta, which have no max\_length.