# Run\_mlm.py cuda error memory after resuming a training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 16, 2021, 9:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549 "2021-04-16T09:54:11Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![IamAdiSri](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iamadisri/32/20434_2.png) [@IamAdiSri](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIamAdiSri)%5C
**Post date:** [April 17, 2021, 4:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549%2F2 "2021-04-17T04:09:58Z")

</div>

I’m running into the same issue but with the mBART model. For some reason, running training from scratch with the `Seq2SeqTrainer` works just fine, but resuming from checkpoint exceeds the memory limit, and produces a CUDA ‘out of memory’ error.

I think it might be related to [this](https://github.com/huggingface/transformers/issues/3730) issue on the GitHub repository.

@sshleifer I think this is another issue with training large models, as we discussed [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpruning-a-model-embedding-matrix-for-memory-efficiency%2F5502) although this just seems to be a bug in the trainer.

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-cuda-error-memory-after-resuming-a-training%2F5549)._