# Reproduce BERT and RoBERTa

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproduce-bert-and-roberta%2F7182%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 23, 2021, 10:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproduce-bert-and-roberta%2F7182 "2021-06-23T22:54:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![wyu1](https://avatars.discourse-cdn.com/v4/letter/w/f1d935/32.png) [@wyu1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwyu1)%5C
**Post date:** [June 23, 2021, 10:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproduce-bert-and-roberta%2F7182%2F1 "2021-06-23T22:54:24Z")

</div>

Hi to all,

I tried to use the running `run mlm.py` to reproduce the result of the bert-base-uncased version. However, I found my reproduced results are always lower than the one reported in this [website](https://github.com/huggingface/transformers/tree/master/examples/pytorch/text-classification) provided by the Huggingface team.

| Task | Metric | Reported by Huggingface | Our reproduced result |
| --- | --- | --- | --- |
| CoLA | Matthew’s corr | 56.53 | 47.92 |
| SST-2 | Accuracy | 92.32 | 87.56 |
| MRPC | F1/Accuracy | 88.85/84.07 | 82.03/80.97 |
| STS-B | Person/Spearman corr. | 88.64/88.48 | 82.45/82.76 |
| QQP | Accuracy/F1 | 90.71/87.49 | 88.23/86.12 |
| MNLI | Matched acc./Mismatched acc. | 83.91/84.10 | 82.34/83.01 |
| QNLI | Accuracy | 90.66 | 85.45 |
| RTE | Accuracy | 65.70 | 56.95 |

I think there must be some problems with my experiment. I ran my experiment by using:

(1) I used the code in this [file](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py) without any change.

(2) I loaded the datasets of bookcorpus and wiki directly from `dataset` library; the text is chunked into 512 tokens.

(3) I set the batch size as 256 and ran 1M steps; and batch size as 8K and ran 50K steps. Both results are worse than the reported numbers.

I really apprecitate if you could provide me a script that I can use to reproduce BERT or RoBERTa. Thank you very much!

---

<div class="post-metadata">

**Author:** ![JackBAI](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jackbai/32/17827_2.png) [@JackBAI](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJackBAI)%5C
**Post date:** [July 24, 2023, 10:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproduce-bert-and-roberta%2F7182%2F2 "2023-07-24T10:20:41Z")

</div>

It’s very usual to have lower scores than the official team, probably due to randomness. But I’m still interested in why this is the case. Do you have any updates on this? Looking forward to it! @wyu1