# I'm making ROBERTA dumber, and I don't know why

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fim-making-roberta-dumber-and-i-dont-know-why%2F3283%3E%5C
**Category:** Beginners\
**Created:** [January 19, 2021, 12:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fim-making-roberta-dumber-and-i-dont-know-why%2F3283 "2021-01-19T12:06:37Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![aberquand](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@aberquand](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faberquand)%5C
**Post date:** [January 19, 2021, 12:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fim-making-roberta-dumber-and-i-dont-know-why%2F3283%2F1 "2021-01-19T12:06:37Z")

</div>

Hi there,

I’m further training from roberta-base using my domain-specific corpus (parsed text related to space systems) and the run-mlm.py script.

Here is my code:

> output=os.system("python run\_mlm.py "  
> "–model\_name\_or\_path=roberta-base "  
> "–overwrite\_output\_dir "  
> "–train\_file=‘data/training.txt’ "  
> "–validation\_file=‘data/testing.txt’ "  
> "–per\_device\_train\_batch\_size=8 "  
> "–per\_device\_eval\_batch\_size=8 "  
> "–do\_train "  
> "–do\_eval "  
> "–line\_by\_line "  
> "–save\_steps=53769 "  
> "–num\_train\_epochs=40 "  
> "–output\_dir=‘./spaceROBERTA/’ "  
> “–logging\_steps=4481”)

The training loss is decreasing (from around 2 to 1), the perplexity over the evaluation set is a bit high but also decreasing (starts at 10 and finishes around 7). So I thought all lights were green for the training, yeah!

But when I fine-tune it over our labeled dataset for a Concept Recognition task, the performance is slightly worse than roberta-base, and getting significantly worse and worse as the number of training epochs increases 😱

I’m basically making roberta-base dumber and dumber and I don’t know why…  
I appreciate if anyone can point to a solution, thanks 🤗

---

<div class="post-metadata">

**Author:** ![aberquand](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@aberquand](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faberquand)%5C
**Post date:** [March 8, 2021, 10:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fim-making-roberta-dumber-and-i-dont-know-why%2F3283%2F2 "2021-03-08T10:32:12Z")

</div>

Update: Increasing the batch size to 256, thanks to gradient-accumulation, improved the performance 🙂

```
"--per_device_train_batch_size=16 "
"--per_device_eval_batch_size=16 "
"--gradient_accumulation_steps=16 "
```