# How to correctly evaluate a Masked Language Model?
**URL:**
**Author:**  [@denocris](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdenocris)%5C
**Post date:** [August 31, 2021, 1:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-correctly-evaluate-a-masked-language-model%2F9634%2F1 "2021-08-31T13:47:06Z")
Hi All,
my question is very simple. Starting from a pre-trained (Italian) model, I fine-tuned it on a specific domain of interest, say X, using masked language model (MLM) training. I then computed perplexity on a test text on domain X and checked that the final model performs better than the pre-trained one.
Is it sufficient?
People who trained this language model [Umberto](https://huggingface.co/Musixmatch/umberto-wikipedia-uncased-v1) using MLM training, performed several tests on downstream tasks such as NER and POS.
How did they do?
My understanding is that starting from Umberto, they fine-tune it on NER and test it on WikiNER-ITA for instance. Is there a simple procedure to do it?
Thanks!
---
**Author:**  [@jonathanalis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjonathanalis)%5C
**Post date:** [February 13, 2022, 2:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-correctly-evaluate-a-masked-language-model%2F9634%2F2 "2022-02-13T02:10:58Z")
In RoBerta they use accuracy and f1 scores of the language model. O got this code that I think performs the accuracy:
> import sklearn
> from datasets import load\_metric
> import numpy as np
> metric = load\_metric(âaccuracyâ)
>
> def compute\_metrics(eval\_pred):
> logits, labels = eval\_pred
> predictions = np.argmax(logits, axis=-1)
>
> ```
> indices = [[i for i, x in enumerate(labels[row]) if x != -100] for row in range(len(labels))]
>
> labels = [labels[row][indices[row]] for row in range(len(labels))]
> labels = [item for sublist in labels for item in sublist]
>
> predictions = [predictions[row][indices[row]] for row in range(len(predictions))]
> predictions = [item for sublist in predictions for item in sublist]
>
> results = metric.compute(predictions=predictions, references=labels)
> results["eval_accuracy"] = results["accuracy"]
> print(results)
> results.pop("accuracy")
>
> return results
>
> ```
Then create a trainer and use this as the compute\_metric parameter
> from transformers import Trainer
> trainer = Trainer(
> model=model,
> âŚ
> compute\_metrics=compute\_metrics,
> )
Then
> results = trainer.evaluate()
> accuracy = results[âeval\_resultsâ]
---
**Author:**  [@NHendrickson9616](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNHendrickson9616)%5C
**Post date:** [July 3, 2023, 4:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-correctly-evaluate-a-masked-language-model%2F9634%2F3 "2023-07-03T16:55:25Z")
Very much late to the party here, but I have been struggling to find perplexity with my BERT model. How exactly did you determine the perplexity of yours?
---
**Author:**  [@lkurlandski](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flkurlandski)%5C
**Post date:** [August 11, 2023, 2:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-correctly-evaluate-a-masked-language-model%2F9634%2F4 "2023-08-11T14:05:08Z")
@NHendrickson9616 [Perplexity](https://huggingface.co/docs/transformers/perplexity) is really only well-suited for causal/autoregressive language models, e.g., GPT. Since it only uses information from the tokens on the left-hand-side, it wouldnât really provide valuable feedback for a model like BERT, which leverages bidirectional information.