# How to input Scibert in run\_mlm? (Is it possible?)

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-input-scibert-in-run-mlm-is-it-possible%2F5206%3E%5C
**Category:** Beginners\
**Created:** [March 30, 2021, 10:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-input-scibert-in-run-mlm-is-it-possible%2F5206 "2021-03-30T10:39:44Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![aberquand](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@aberquand](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faberquand)%5C
**Post date:** [March 30, 2021, 10:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-input-scibert-in-run-mlm-is-it-possible%2F5206%2F1 "2021-03-30T10:39:44Z")

</div>

Hi there,

I’m trying to further train from the scibert\_scivocab\_uncased model, using the run\_mlm script. I’ve had no issues further training from BERT\_base and RoBERTa but I’m a bit stuck with sciBERT.

SciBERT is not one of the basic models you can directly call from run\_mlm.py  
So I downloaded the model from [allenai/scibert\_scivocab\_uncased at main](https://huggingface.co/allenai/scibert_scivocab_uncased/tree/main)

to run:  
`"python myrun_mlm.py "`  
`"--model_name_or_path=scibert_scivocab_uncased "`

But the tokenizer files (tokenizer.json, tokenizer\_config.json,…) are missing so it’s not working. I can’t find the tokenizers files in the allenAI scibert git repo either.

What am I missing there?  
Thanks for the help! 🤗

---

<div class="post-metadata">

**Author:** ![aberquand](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@aberquand](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faberquand)%5C
**Post date:** [March 31, 2021, 2:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-input-scibert-in-run-mlm-is-it-possible%2F5206%2F2 "2021-03-31T14:23:59Z")

</div>

Alternative solution: not using the run-mlm.py script, but a code inspired from: [lordtt13/COVID-SciBERT · Hugging Face](https://huggingface.co/lordtt13/COVID-SciBERT) / [word-embeddings/COVID-SciBERT.ipynb at master · lordtt13/word-embeddings · GitHub](https://github.com/lordtt13/word-embeddings/blob/master/COVID-19%20Research%20Data/COVID-SciBERT.ipynb) Thanks @lordtt13 !!

with:  
` tokenizer = transformers.AutoTokenizer.from_pretrained('allenai/scibert_scivocab_uncased')`  
`model = transformers.AutoModelWithLMHead.from_pretrained('allenai/scibert_scivocab_uncased').to('cuda')`

Still would be interesting to know if the integration of scibert as a pre-trained model is planned for run\_mlm.py