# BERT and RoBERTA giving same outputs
**URL:**
**Author:**  [@theguywithblacktie](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftheguywithblacktie)%5C
**Post date:** [September 22, 2021, 10:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-and-roberta-giving-same-outputs%2F10214%2F1 "2021-09-22T10:48:55Z")
Hi All.
I tried using Roberta model in two different models. In both these models, I’ve faced same problem of getting same output for different test input during evaluation process.
Earlier, I thought it might be due to some implementation problem and hence I took a small dataset to overfit the dataset and predict the outputs for the same. I still got the same problem. Roberta was still giving out same output for different records.
I replaced Roberta with Bert and still got same issue.
Is there any bug in latest transformer version i.e. 4.10.2 (which I’m surely believe is very unlikely) or do have any other suggestion that I can try? I’ve used 4.2.1 version of transformer earlier and didn’t face this problem.
Also, I keep getting this warning while training and evaluation:
```auto
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```
I checked online and I suspect this is not a issue but I am still not sure what it actually means. Could this be any issue?
---
**Author:**  [@Chetan06](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FChetan06)%5C
**Post date:** [January 4, 2022, 4:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-and-roberta-giving-same-outputs%2F10214%2F2 "2022-01-04T04:23:43Z")
@theguywithblacktie have you figured out what was wrong with your code as I am facing the same issue while using roberta from transformer library.
---
**Author:**  [@Atharva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAtharva)%5C
**Post date:** [January 29, 2022, 5:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-and-roberta-giving-same-outputs%2F10214%2F3 "2022-01-29T05:32:16Z")
Any solutions?
---
**Author:**  [@veronica320](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fveronica320)%5C
**Post date:** [July 5, 2022, 11:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-and-roberta-giving-same-outputs%2F10214%2F4 "2022-07-05T23:02:03Z")
Having the same issue +1
---
**Author:**  [@cspartalis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcspartalis)%5C
**Post date:** [July 31, 2022, 12:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-and-roberta-giving-same-outputs%2F10214%2F5 "2022-07-31T12:44:41Z")
If I understood the problem correctly, the fine-tuned model always outputs the same value (e.g. the same class for a classification task). I had the same issue and tried to overcome it by tuning hyperparameters. The only hyperparameter that worked was **the number of epochs**.
I hope it helps,
Christoforos Spartalis
---
**Author:**  [@felixckyeung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffelixckyeung)%5C
**Post date:** [April 4, 2024, 8:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-and-roberta-giving-same-outputs%2F10214%2F6 "2024-04-04T20:20:16Z")
Anyone fix this?
---
**Author:**  [@swtb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fswtb)%5C
**Post date:** [April 9, 2024, 9:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-and-roberta-giving-same-outputs%2F10214%2F7 "2024-04-09T09:27:43Z")
Its difficult to diagnose the issue without seeing your training script.