# Training a LM from scratch on large corpus

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-lm-from-scratch-on-large-corpus%2F657%3E%5C
**Category:** Beginners\
**Created:** [August 10, 2020, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-lm-from-scratch-on-large-corpus%2F657 "2020-08-10T13:54:25Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrm8488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrm8488/32/5836_2.png) [@mrm8488](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmrm8488)%5C
**Post date:** [August 10, 2020, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-lm-from-scratch-on-large-corpus%2F657%2F1 "2020-08-10T13:54:25Z")

</div>

Hi guys, I want to train a RoBERTa like model for Spanish on a corpus of 200 GB approx. I have already trained one following the original RoBERTa implementation availiable on github but only on 20GB.  
Now, I want to use HF libraries/tooling and ofc I would like to use HF/nlp. I have been testing on local with different configurations. I got one that “works” but after some tranining steps I always get and error (CUBLAS ERROR).  
I leave you here the colab in order to reproduce everything:  
[https://colab.research.google.com/drive/15m7F30TIdAw0wtyVM9\_MSXmGgt6L7W0M?usp=sharing](https://colab.research.google.com/drive/15m7F30TIdAw0wtyVM9_MSXmGgt6L7W0M?usp=sharing)

Thanks in advance!