# Run\_mlm.py cuda error memory after resuming a training
**URL:**
**Author:**  [@Shiro](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FShiro)%5C
**Post date:** [April 16, 2021, 9:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549%2F1 "2021-04-16T09:54:11Z")
Hello,
I am trying to pretrain an XLM Roberta model, but I got some issue. I have trained the model for few thousand steps and get the checkpoint. Then I wanted to continue the pre-training from the checkpoint, but got a memory error âCUDA error memoryâ after few steps. I wonder if there is a leak of something like that ? The memory used during the 1st pretraining used around 15.X GB / 16.2 GB so I quite donât understand whatâs going on.
---
**Author:**  [@IamAdiSri](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIamAdiSri)%5C
**Post date:** [April 17, 2021, 4:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549%2F2 "2021-04-17T04:09:58Z")
Iâm running into the same issue but with the mBART model. For some reason, running training from scratch with the `Seq2SeqTrainer` works just fine, but resuming from checkpoint exceeds the memory limit, and produces a CUDA âout of memoryâ error.
I think it might be related to [this](https://github.com/huggingface/transformers/issues/3730) issue on the GitHub repository.
@sshleifer I think this is another issue with training large models, as we discussed [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpruning-a-model-embedding-matrix-for-memory-efficiency%2F5502) although this just seems to be a bug in the trainer.
---
**Author:**  [@IamAdiSri](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIamAdiSri)%5C
**Post date:** [April 20, 2021, 12:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549%2F3 "2021-04-20T12:09:48Z")
This fix by @sgugger in a [recent commit](https://github.com/huggingface/transformers/pull/11318) resolved the issue for me. The fix is not in the official release yet, but you can install the bleeding edge version of the Transformers library from source using `pip install git+https://github.com/huggingface/transformers` which pulls it from GitHub.
---
**Author:**  [@xusky](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fxusky)%5C
**Post date:** [April 21, 2021, 2:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549%2F4 "2021-04-21T14:34:21Z")
having the same issue here with GPT-2 large, gonna try what @IamAdiSri suggested
---
**Author:**  [@xusky](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fxusky)%5C
**Post date:** [April 21, 2021, 3:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549%2F5 "2021-04-21T15:10:48Z")
Can confirm that installing the bleeding edge version of transformers as suggested by @IamAdiSri fixed the issue.