# Train Roberta from scratch for custom dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-roberta-from-scratch-for-custom-dataset%2F4331%3E%5C
**Category:** Intermediate\
**Created:** [March 11, 2021, 11:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-roberta-from-scratch-for-custom-dataset%2F4331 "2021-03-11T23:30:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![clementine](https://avatars.discourse-cdn.com/v4/letter/c/8dc957/32.png) [@clementine](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclementine)%5C
**Post date:** [March 11, 2021, 11:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-roberta-from-scratch-for-custom-dataset%2F4331%2F1 "2021-03-11T23:30:56Z")

</div>

Hey there, I am training Roberta from scratch for protein sequences. To this end, I build a tokenizer for protein sequences, which is very much like a character-level tokenizer. After that, I stored my tokenizer and used it in the Roberta model followed this tutorial: [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train).

The code runs fine on CPU but failed on GPU, and the error message is as follows:  
RuntimeError: CUDA error: CUBLAS\_STATUS\_NOT\_INITIALIZED when calling `cublasCreate(handle)`

Thanks in advance for any thoughts on this issue!

---

<div class="post-metadata">

**Author:** ![LidorPrototype](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@LidorPrototype](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLidorPrototype)%5C
**Post date:** [May 2, 2023, 4:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-roberta-from-scratch-for-custom-dataset%2F4331%2F2 "2023-05-02T04:08:08Z")

</div>

Hello, have you found any solution for that?