”])
> tokenizer.save\_model(“mymodel”)’
and use it to preprocess my data, FYI I have 700,000 sentences stored in a txt file, one sentence per line.
> from transformers import LineByLineTextDataset
> training\_dataset = LineByLineTextDataset(tokenizer=tokenizer, file\_path=“data/training.txt”, block\_size=128,)
> evalutation\_dataset = LineByLineTextDataset(tokenizer=tokenizer, file\_path=“data/testing.txt”, block\_size=128,)
1. Get the roberta config
> from transformers import RobertaConfig, RobertaForMaskedLM
> config = RobertaConfig( vocab\_size=52\_000, max\_position\_embeddings=514,
> num\_attention\_heads=12, num\_hidden\_layers=12, type\_vocab\_size=1,)
> config.save\_model(“mymodel”)
> model = RobertaForMaskedLM(config=config)
Or should I instead use:
> from transformers import RobertaForMaskedLM
> model= RobertaForMaskedLM.from\_pretrained(‘roberta-base’)
1. Get data Collaor
> from transformers import DataCollatorForLanguageModeling
> data\_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm\_probability=0.15)
1. Set up training arguments and train
> from transformers import Trainer, TrainingArguments, EvalPrediction
What I currently have:
> training\_args = TrainingArguments(
> output\_dir=“./mymodel”,
> evaluation\_strategy=“steps”,
> prediction\_loss\_only=True,
> per\_device\_train\_batch\_size= 32,
> per\_device\_eval\_batch\_size=32,
> eval\_accumulation\_steps = 200,
> weight\_decay=0.01,
> adam\_epsilon=1e-6,
> max\_steps=200000,
> warmup\_steps=1,
> save\_steps=200,
> save\_total\_limit=5,
> eval\_steps= 100,
> )
> trainer = Trainer(
> model=model,
> args=training\_args,
> data\_collator=data\_collator,
> train\_dataset=training\_dataset,
> eval\_dataset=evalutation\_dataset,
> )
> trainer.train()
Should I instead use:
> python run\_mlm.py
> –model\_name\_or\_path roberta-base
> –train\_file path\_to\_train\_file
> –validation\_file path\_to\_validation\_file
> –do\_train
> –do\_eval
> –output\_dir /tmp/test-mlm
Thanks in advance for any help you can provide!
---
**Author:**  [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [November 29, 2020, 5:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-domain-specific-roberta-from-roberta-base%2F2324%2F2 "2020-11-29T17:15:31Z")
Hi aberquand,
I don’t think you can use the pre-trained weights with your domain-specific vocabulary.
[I am not an expert, and I’ve only used BERT not RoBERTA, and I didn’t use the Trainer, so I could be wrong.]
If I understand it correctly, the way the weights learn is dependent on the particular vocabulary.
I suggest you use the pre-trained vocabulary as well as the pre-trained weights.
How different is your vocabulary from the original RoBERTA vocabulary? I would expect 700000 sentences would be enough to do fine-tuning, but probably not enough to train from scratch.
Did you know, if you use Google Colaboratory you can get a limited amount of GPU-time for free. It generally maxes out after about 7 hours each day. Colab uses Jupyter notebooks.
For more information on tokenizer vocabulary, I recommend Chris McCormick’s blogs, eg [https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/)
---
**Author:**  [@aberquand](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faberquand)%5C
**Post date:** [November 30, 2020, 9:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-domain-specific-roberta-from-roberta-base%2F2324%2F3 "2020-11-30T09:25:11Z")
Hi @rgwatwormhill, thanks for the answer!
Actually indeed that would make sense to need the roberta vocabulary with the pre-trained weights, I was trying to do something similar to BioBERT ([https://arxiv.org/ftp/arxiv/papers/1901/1901.08746.pdf](https://arxiv.org/ftp/arxiv/papers/1901/1901.08746.pdf)) which seem to have used BERT weight initialization and a biomedical domain corpora…
I guess I’ll try both:
- pre-train my domain specific with my tokenizer + from scratch
- initialize training with roberta and continue training from there with my domain specific corpus processed with roberta tokenizer.
Still wondering how to initialize the weights to roberta’s though 😕
ps: yes using GCP credits and a paid Nvidia V100 for the simulation now, colab free GPU was not enough 😅
---
**Author:**  [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [November 30, 2020, 9:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-domain-specific-roberta-from-roberta-base%2F2324%2F4 "2020-11-30T09:30:21Z")
The term that you are looking for is fine-tuning, (or in this case perhaps further pretraining because it is the same task objective). You can easily do that with the example scripts that are available in the Github [repository](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling).
As @rgwatwormhill notes, you cannot create a new vocabulary and use that with a pretrained model: that model is trained with a specific vocabulary, and swapping that out for another vocab will lead to unexpected results. If I were you I’d first try fine-tuning the existing model with the script above, and if the performance is not good enough for you, you can still train from scratch. Note that you need a lot of data to get good results (that’s why typically finetuning is preferred).
---
**Author:**  [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [November 30, 2020, 3:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-domain-specific-roberta-from-roberta-base%2F2324%2F5 "2020-11-30T15:23:42Z")
hello again,
I’ve had a look at the BioBERT article. Fig 1 does look as though they (pre-) train with medical texts using BERT base weights, but Section 3.3 clarifies that actually they used the standard BERT base vocabulary and that the training of BioBERT effectively started with the standard Wiki+Books data and then was followed by the medical texts. So, the BioBERT article is all about fine-tuning a pre-trained BERT model.
There can be as many training steps as you care to imagine when doing NLP with BERT. So far as I know, people only use two terms for it: pre-training and fine-tuning. We could really do with a term for the intermediate steps. I call it intermediate-training.
Pre-training: that which somebody else has already done.
Intermediate training: further modification, for example using language modelling with domain-specific texts.
Fine-tuning: training to the task of interest.
The Intermediate stages can also include other steps such as Distilling or Pruning, and might include Freezing some or all of the layers.
Note that the BioBERT article says “pre-trained on biomedical corpora”, but I would classify that as Intermediate training. I suppose, if you pick up their finished model, you would then consider both the Wiki+Books and the medical corpus as pre-training.
If anyone knows a standard classification of what counts as “pre-training” I’d like to know. @BramVanroy maybe?
---
**Author:**  [@aberquand](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faberquand)%5C
**Post date:** [November 30, 2020, 3:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-domain-specific-roberta-from-roberta-base%2F2324%2F6 "2020-11-30T15:48:11Z")
Agreed, BioBERT uses in fact the BERT-base vocabulary.
Also interested by a better classification of what “pre-training” encompasses, perhaps could use "post-training’ or ‘further training’ as suggested by @BramVanroy.
Another relevant work for anyone interested to compare the effect of using BERT-base vocab or your own vocab: SCIBERT ([https://arxiv.org/pdf/1903.10676.pdf](https://arxiv.org/pdf/1903.10676.pdf)) further trained a BERT Base model with Scientific data, either using BERT’s vocab or their own SciVocab.
But based on your previous comments and the size of my training set, I’ll start by building on top of roberta-base. Thanks a lot for your help @rgwatwormhill and @BramVanroy, finally moving forward! 🥳
---
**Author:**  [@Max](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMax)%5C
**Post date:** [February 2, 2021, 12:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-domain-specific-roberta-from-roberta-base%2F2324%2F7 "2021-02-02T00:43:31Z")
Hi, I am trying something similar to do “further training” as mentioned above by running the run\_mlm.py script in google colab from a checkpoint.
cmd = ‘’‘python run\_mlm.py
–output\_dir {0}
–model\_name\_or\_path roberta-base
–mlm\_probability 0.15 \
–train\_file {1}
–validation\_file {2}
–config\_name /content/models/smallBERTa
–tokenizer\_name /content/models/smallBERTa
–do\_train
–line\_by\_line
–overwrite\_output\_dir
–do\_eval
–learning\_rate 1e-4 \
–num\_train\_epochs 5
–save\_total\_limit 2
–save\_steps 2000
–logging\_steps 500
–per\_device\_eval\_batch\_size 32
–per\_device\_train\_batch\_size 32
–seed 42
‘’’.format(weights\_dir,train\_path,eval\_path)
Run:
!{cmd}
Problem is, I have the error:
Traceback (most recent call last):
File “run\_mlm.py”, line 386, in
main()
File “run\_mlm.py”, line 154, in main
model\_args, data\_args, training\_args = parser.parse\_args\_into\_dataclasses()
File “/usr/local/lib/python3.6/dist-packages/transformers/hf\_argparser.py”, line 135, in parse\_args\_into\_dataclasses
obj = dtype(\*\*inputs)
File “”, line 12, in **init**
File “run\_mlm.py”, line 133, in **post\_init**
raise ValueError(“Need either a dataset name or a training/validation file.”)
ValueError: Need either a dataset name or a training/validation file.
/bin/bash: line 1: --train\_file: command not found
/bin/bash: line 2: --num\_train\_epochs: command not found
I have no idea if the error is because of some bash configuration i did not set on google colab, or if the python script was written wrongly. Appreciate if anyone encountered the same issue and know what is going on!
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [February 2, 2021, 8:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-domain-specific-roberta-from-roberta-base%2F2324%2F8 "2021-02-02T08:04:06Z")
Hi @Max, I think the source of your error is that you’re missing some line-break characters `\` in your `cmd` string. In other words, put a backslash at the end of each line in the script command:
```auto
python run_mlm.py \
--output_dir {0} \
--model_name_or_path roberta-base \
...
```
There might also be something quirky happening with the string formatting, so I’d check that it makes sense by printing out the `cmd` string