# RobIt : PreTrain RoBERTa-base from scratch in Italian
**URL:**
**Author:**  [@Sheyz-max](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSheyz-max)%5C
**Post date:** [June 28, 2021, 7:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frobit-pretrain-roberta-base-from-scratch-in-italian%2F7564%2F1 "2021-06-28T19:45:23Z")
# 1. RobIt
The idea is to use the Italian portion of mC4 (which is about 590 GB) and OSCAR dataset(which is about 137 GB) to pre-train a RoBERTa-base model.
# 2. Language
The model will be trained in Italian
# 3. Model used
ROBERTa-base
# 4. Datasets
OSCAR-Italian, Italian Portion of [multilingual C4 dataset](https://github.com/allenai/allennlp/discussions/5265)
# 5.Training Scripts:
There are already Flax scripts to pre-train RoBERTa that we can easily use:
[https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
[Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/masked_language_modeling_flax.ipynb#scrollTo=DO1SwHdi55en)
# 6. Challenges:
- It is too much data. We need a way to reduce the amount of data to finish on time so we will use random sampling.
- Achieving SOTA.
# 7. Desired Possible Outcome:
- A Italian monolingual well performing model. We could optionally test the validity of our results at a later stage by fine-tuning the model on some downstream tasks.
# 8. Reads
- [https://arxiv.org/pdf/1907.11692.pdf](https://arxiv.org/pdf/1907.11692.pdf)
- We found a model trained on [RoBERTa](https://github.com/idb-ita/GilBERTo) but we plan on increasing the dataset with a different tokenisation.
---
**Author:**  [@prateekagrawal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprateekagrawal)%5C
**Post date:** [June 28, 2021, 7:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frobit-pretrain-roberta-base-from-scratch-in-italian%2F7564%2F2 "2021-06-28T19:51:45Z")
Count me in !!!
---
**Author:**  [@ruchi798](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fruchi798)%5C
**Post date:** [June 28, 2021, 7:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frobit-pretrain-roberta-base-from-scratch-in-italian%2F7564%2F3 "2021-06-28T19:59:08Z")
Count me in 😄
---
**Author:**  [@yotanay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyotanay)%5C
**Post date:** [June 29, 2021, 4:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frobit-pretrain-roberta-base-from-scratch-in-italian%2F7564%2F4 "2021-06-29T04:35:54Z")
Would love to be a part of this, count me in too!
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 1:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frobit-pretrain-roberta-base-from-scratch-in-italian%2F7564%2F5 "2021-06-29T13:51:01Z")
Great! Finalizing this project!