# PreTrain RoBERTa from scratch in Portuguese
**URL:**
**Author:**  [@diiogo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiiogo)%5C
**Post date:** [June 24, 2021, 3:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F1 "2021-06-24T15:32:27Z")
## RoBERTa for Portuguese
Currently, there is only a very limited amount of RoBERTa or RoBERTa Large model that was trained from scratch for Portuguese on the hub: [Hugging Face – The AI community building the future.](https://huggingface.co/models?filter=roberta,pt). For this project, the goal is to create a RoBERTa model for just the Portuguese language.
## Model
A randomly initialized RoBERTa model
## Available training scripts
A masked language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py). It can be used pretty much without any required code changes.
## Dataset
We can use dataset from BrWaC (Brazilian Web as Corpus) a large Portuguese corpus or Wikidata in Portuguese.
## Challenges
- Data pre-processing
- Sample preparation for LM
## (Optional) Desired project outcome
The desired project output is a strong RoBERTa model in Portuguese.
---
**Author:**  [@airtonchagas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fairtonchagas)%5C
**Post date:** [June 27, 2021, 1:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F3 "2021-06-27T13:46:48Z")
What a cool project! You can count on me.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 28, 2021, 4:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F7 "2021-06-28T16:54:06Z")
Sounds great! Finalizing this project as well 🙂 One thing that you should think about is also how to evaluate the pretrained model on a downstream task. [paws-x · Datasets at Hugging Face](https://huggingface.co/datasets/paws-x) or [xtreme · Datasets at Hugging Face](https://huggingface.co/datasets/xtreme) would make sense I think 🙂
Putting everybody in one group for now. You can check it out [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit#gid=0).
---
**Author:**  [@ronebrandao](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fronebrandao)%5C
**Post date:** [June 28, 2021, 5:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F8 "2021-06-28T17:23:04Z")
That would be great. Wanna be part of it!!
---
**Author:**  [@WillFerreiraSantos](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FWillFerreiraSantos)%5C
**Post date:** [June 28, 2021, 8:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F9 "2021-06-28T20:16:36Z")
Wow, how cool this project is!
Quite innovative idea.
I’ve already signed up to help with the project, you can count on me!
Congratulations to those involved!
😀 👏
---
**Author:**  [@diiogo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiiogo)%5C
**Post date:** [June 28, 2021, 8:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F10 "2021-06-28T20:27:12Z")
Thanks for all, at this moment we think to evaluate the pretrained model on [HAREM](https://huggingface.co/datasets/harem) or [squad](https://github.com/nunorc/squad-v1.1-pt) or [LeNER](https://cic.unb.br/~teodecampos/LeNER-Br/).
---
**Author:**  [@mSancle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FmSancle)%5C
**Post date:** [June 29, 2021, 5:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F12 "2021-06-29T17:58:44Z")
Very interesting project! It’ll be great to participate!
---
**Author:**  [@pierreguillou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpierreguillou)%5C
**Post date:** [July 1, 2021, 6:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F14 "2021-07-01T18:58:20Z")
> [@diiogo](#):
>
> Thanks for all, at this moment we think to evaluate the pretrained model on [HAREM](https://huggingface.co/datasets/harem) or [squad](https://github.com/nunorc/squad-v1.1-pt) or [LeNER](https://cic.unb.br/~teodecampos/LeNER-Br/).
Good afternoon @diiogo.
I tried to put my username in the [spreadsheet](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?pli=1#gid=0) but it is now closed.
As I’m interested in the subject, I will follow your work, in particular regarding to the evaluation through downstreams tasks of your Portuguese RoBERTa model.
- **Question Answering (QA) with SQuAD 1.1 pt** : I published 2 models BERT finetuned with this dataset. That will be interesting to compare your f1 e exact-match ([link to the SQuAD 1.1 pt BERT models in the HF model hub](https://huggingface.co/pierreguillou)).
- **Token Classification with LeNER-br** : the [dataset is in the datasets hub of HF](https://huggingface.co/datasets/lener_br). I will publish soon a BERT model finetuned with it, too.
- **Text Classification with ASSIN and ASSIN2 (RTE: Recognizing Textual Entailment)**: the ASSIN and ASSIN2 (Avaliação de Similaridade Semântica e INferência textual) corpus are corpus annotated with pairs of sentences written in Portuguese that is suitable for the exploration of textual entailment and paraphrasing classifiers. The datasets are in the datasets hub of HF ([ASSIN](https://huggingface.co/datasets/assin), [ASSIN2](https://huggingface.co/datasets/assin2)). I will publish soon a BERT model finetuned with them, too.
It will be great to test you model with other datasets and/or tasks. What do you think?
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 2, 2021, 12:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F15 "2021-07-02T00:25:36Z")
This group has a lot of participation 🙂
Giving you guys directly to TPUs tomorrow! Split the team randomly into two in the official google sheet, but this shouldn’t change anything - just that you have access two 2 TPU v3-8s 🙂
Might make organization a bit easier to split work on two VMs!
---
**Author:**  [@diiogo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiiogo)%5C
**Post date:** [July 2, 2021, 12:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F16 "2021-07-02T12:53:09Z")
Good Morning @pierreguillou.
Its a nice idea to include your results in evaluate of this task (squad).
I think too, we will evaluate the model in other tasks as well.
---
**Author:**  [@diiogo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiiogo)%5C
**Post date:** [July 2, 2021, 12:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F17 "2021-07-02T12:53:31Z")
thank you so much
---
**Author:**  [@pierreguillou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpierreguillou)%5C
**Post date:** [July 5, 2021, 1:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F18 "2021-07-05T13:35:49Z")
Hello @diiogo.
I edited my [post](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F14) with datasets ASSIN and ASSIN2 for Text Classification task.
---
**Author:**  [@pierreguillou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpierreguillou)%5C
**Post date:** [July 5, 2021, 2:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F19 "2021-07-05T14:11:48Z")
> [@diiogo](#):
>
> ## Dataset
>
> We can use dataset from BrWaC (Brazilian Web as Corpus) a large Portuguese corpus or Wikidata in Portuguese.
The [BrWaC](https://huggingface.co/datasets/brwac) is in the dataset hub of Hugging Face and [OSCAR](https://huggingface.co/datasets/oscar), too.
It might be interesting to mix the 2 together for training.
About the embeddings and weights initialization, starting from those of a trained RoBERTa model could help, too
1. Download the model from the HF Model Hub to get the weights.
2. Train a RoBERTa tokenizer on the training dataset to get the vocabulary.
3. Keep the ambeddings of the common tokens between the downloaded model vocabulary and the new one.
4. For the new tokens, initialize them.
---
**Author:**  [@Piero](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPiero)%5C
**Post date:** [September 27, 2021, 2:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F20 "2021-09-27T14:30:36Z")
Hi! What’s the current status of this initiative?
---
**Author:**  [@diiogo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiiogo)%5C
**Post date:** [September 27, 2021, 6:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F21 "2021-09-27T18:51:09Z")
Hi, now we are doing the training with the fairseq.
---
**Author:**  [@Piero](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPiero)%5C
**Post date:** [September 28, 2021, 6:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F22 "2021-09-28T18:16:23Z")
Thanks @diiogo,
Any particular reason for this change? (Fairseq instead of HF Trainer)
---
**Author:**  [@diiogo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiiogo)%5C
**Post date:** [October 4, 2021, 5:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-portuguese%2F7272%2F23 "2021-10-04T17:22:32Z")
Hi @Piero,
Some features was unstable or missing in Flax/JAX
Kind regards,
Diogo